FreeToken Lets a Single RTX 5090 Run 284B MoE Models Locally
Running frontier-scale language models locally has traditionally been constrained by one fundamental problem: the model may be open-weight, but the hardware required to serve it is not.
A new open-source inference system called FreeToken attempts to close that gap by redesigning how Mixture-of-Experts (MoE) models use consumer CPUs, system memory, PCIe bandwidth, and GPUs.
According to its authors, FreeToken can run models such as Qwen 3.6-35B on a laptop RTX 4060 and DeepSeek-V4-Flash 284B on a single RTX 5090, without requiring the entire model to reside in GPU VRAM.
The key is not simply more aggressive quantization. FreeToken uses a full-stack co-design built around bandwidth-aware scheduling, double-buffered data movement, state reuse, and dynamically adjustable GPU residency.
For MoE models, that changes the assumption that a model must fit entirely into VRAM before it can deliver practical local inference.
🚀 FreeToken Targets the Real Bottleneck in Local LLMs #
FreeToken was developed by researchers associated with institutions including UC Berkeley and MIT, with contributors including Shuo Yang, Xiaoze Fan, Ion Stoica, Matei Zaharia, Kurt Keutzer, and Song Han.
The project focuses specifically on edge-native MoE serving, where the GPU is only one component of a heterogeneous computing system.
Instead of treating GPU VRAM as the only usable memory resource, FreeToken coordinates:
- GPU compute
- CPU compute
- System RAM
- GPU VRAM
- PCIe bandwidth
- Expert caching
- Model execution state
The result is an inference architecture designed around the actual hardware topology available in consumer PCs.
Open weights do not guarantee accessible inference #
The rapid expansion of open-weight models has dramatically reduced the barrier to accessing advanced AI capabilities.
However, another barrier remains: inference cost.
A model can be freely downloaded while still requiring expensive multi-GPU servers to run at useful speeds.
FreeToken addresses this second layer of accessibility by attempting to make large sparse models practical on hardware already owned by consumers.
The project is available as open-source software, with desktop applications for Windows and Linux as well as a command-line installation path.
💻 Consumer GPUs Can Run Much Larger MoE Models #
The most striking demonstrations involve hardware that would normally be considered too small for the corresponding models.
A laptop RTX 4060 can reportedly run Qwen 3.6-35B at approximately 39.3 tokens/s under the evaluated configuration.
A desktop RTX 5090 can reportedly serve DeepSeek-V4-Flash 284B locally.
The significance is not that a 284B-parameter model has somehow been compressed into 32GB of VRAM.
It has not.
Instead, FreeToken exploits the sparse execution characteristics of MoE architectures while using system memory to hold the much larger expert pool.
Model size and active parameters are different #
DeepSeek-V4-Flash is described as having approximately:
- 284B total parameters
- 43 layers
- 256 routed experts per layer
- 6 routed experts activated per layer
- Approximately 13B active parameters per token
Only a fraction of the total model participates in generating any individual token.
This dramatically reduces compute requirements.
However, it does not reduce the total memory required to store the model’s expert weights.
That distinction is fundamental to understanding why conventional local inference approaches struggle.
🧠 MoE Sparsity Does Not Eliminate the Memory Problem #
MoE architectures reduce computation by activating only a subset of experts for each token.
But during prompt processing, the situation becomes more complicated.
A long prompt contains many tokens, and different tokens can route to different experts. Over the course of an entire prefill operation, the union of selected experts can cover a large portion of the model.
Consequently, a system that fetches every expert strictly on demand can end up streaming a huge amount of weight data through PCIe.
Prefill can destroy the practical benefit of sparsity #
Consider DeepSeek-V4-Flash.
Although only approximately 13B parameters are active for an individual token, the entire expert pool can still be enormous.
Under the described FP4 deployment configuration, approximately 140GB of expert weights may need to be transferred during the relevant prefill workload.
On an RTX 5090, that transfer can introduce roughly two seconds of additional latency.
If the inference engine waits for each expert to arrive before continuing computation, the GPU spends substantial time idle.
The challenge therefore becomes a data-movement scheduling problem, not simply a compute problem.
⚡ Double-Buffered Prefill Hides PCIe Latency #
FreeToken addresses the problem through double-buffered execution across model layers.
While the GPU computes layer l, the runtime simultaneously begins transferring the expert weights needed for layer l+1.
This creates an overlapping pipeline:
$$
GPU computation: [Layer l] [Layer l+1] [Layer l+2]
PCIe prefetch: [Layer l+1] [Layer l+2] [Layer l+3]
$$
Instead of waiting for data after each layer, computation and transfer proceed concurrently.
The objective is to turn PCIe from a blocking dependency into a continuously utilized data path.
Why PCIe topology matters #
The effectiveness of this strategy depends heavily on the actual hardware configuration.
FreeToken reportedly evaluates systems using different PCIe configurations, including:
- PCIe 3.0 x8
- PCIe 4.0 x16
- PCIe 5.0 x16
It also considers different CPU configurations, including Intel Core, AMD Ryzen, and Threadripper platforms.
The runtime adapts to the available bandwidth rather than assuming a fixed transfer rate.
📡 Bandwidth-Adaptive Execution Balances CPU and GPU Work #
One of FreeToken’s central ideas is that GPU offloading should not be treated as a binary decision.
A conventional approach might attempt to send every required expert to the GPU whenever possible.
FreeToken instead continuously estimates the system’s current capabilities and determines how much work should remain on the GPU versus the CPU.
Dynamic offloading ratio #
The runtime probes:
- Available PCIe bandwidth
- Instantaneous CPU compute capability
- GPU availability
- GPU cache residency
- Current transfer conditions
It then calculates an appropriate offloading ratio.
Frequently accessed experts can remain in the GPU’s LRU cache.
When an expert is not cached, the runtime can decide whether it is more efficient to:
- Transfer the expert to the GPU and execute it there, or
- Execute the computation directly on the CPU.
This prevents the PCIe bus from becoming a mandatory bottleneck for every expert.
Background I/O changes the scheduling decision #
The adaptive model is particularly useful on consumer PCs.
A user may simultaneously run:
- A game
- A 3D rendering workload
- Large file transfers
- Development tools
- Browser workloads
- Other GPU applications
If another application consumes PCIe bandwidth, FreeToken can shift more computation toward the CPU.
When the bus becomes available again, the runtime can increase GPU utilization.
The result is an inference engine that adapts to the actual machine rather than assuming ideal benchmark conditions.
⏱️ TTFT Improvements Matter for Agent Workloads #
Token generation speed is only one part of interactive LLM performance.
For coding agents and tool-calling systems, Time-to-First-Token (TTFT) can be equally important.
An agent may repeatedly:
- Read context
- Generate a response
- Call a tool
- Receive tool output
- Append the new information
- Generate another response
Each iteration can require additional prompt processing.
FreeToken reportedly reduces TTFT by approximately 42–58% for long prompts under its evaluated configurations.
The more interesting optimization, however, targets repeated interactions.
🔄 Agentic State Reuse Reduces Repeated Prefill #
Modern agent workloads frequently modify context incrementally.
A coding agent may append tool output, add reasoning traces, or modify portions of its conversation history.
Naively recomputing the entire context after every change can become extremely expensive.
FreeToken introduces lightweight checkpoints at selected token boundaries.
When the context changes, the runtime can restore the nearest valid checkpoint and recompute only the affected portion.
Checkpointing avoids unnecessary recomputation #
Without state reuse, a small context modification near the end of a long prompt can invalidate subsequent state and force the system to process thousands of tokens again.
FreeToken’s checkpoint mechanism limits the amount of work that must be repeated.
The paper reports 65–80% reductions in TTFT for subsequent multi-turn interactions under its evaluated agent workloads.
This optimization is especially relevant to coding agents because their workloads naturally produce repeated, incrementally changing contexts.
🧮 Recurrent Architectures Make State Reuse More Important #
Some newer language models use recurrent-style components, including architectures such as gated DeltaNet or KDA.
These architectures can compress a long prefix into an evolving state rather than relying exclusively on conventional transformer KV-cache behavior.
That can create a different reuse problem.
A context modification may invalidate the state following the edited point, forcing the system to roll back to a valid checkpoint and recompute the affected sequence.
Because checkpoints themselves consume memory, an engine cannot simply store every possible state.
FreeToken’s lightweight checkpointing approach attempts to find a practical balance between memory consumption and recomputation cost.
🧊 Elastic VRAM Prevents OOM Failures #
Consumer GPUs introduce another problem that data-center inference systems generally avoid: VRAM is shared with everything else running on the machine.
A user might start an LLM server and then launch a game or GPU-intensive application.
Available VRAM can suddenly fall by several gigabytes.
Traditional inference systems may respond with a CUDA out-of-memory error and terminate the workload.
FreeToken instead supports dynamic hot resizing of the GPU-resident expert cache.
When VRAM availability decreases, the system can shrink its LRU cache without restarting the serving process.
More cache misses are subsequently handled through CPU execution or additional data transfers.
Graceful degradation is more useful than maximum residency #
This design changes the failure mode.
Instead of:
VRAM pressure
↓
CUDA OOM
↓
Inference stops
FreeToken aims for:
VRAM pressure
↓
Reduce GPU expert cache
↓
Increase CPU/offloaded execution
↓
Inference continues at lower efficiency
For personal AI infrastructure, this can be considerably more useful than maximizing benchmark throughput under ideal conditions.
🖥️ System RAM Becomes Part of the AI Accelerator #
The FreeToken approach effectively turns the entire PC into a heterogeneous inference platform.
GPU VRAM provides high-bandwidth execution and caching.
System RAM provides substantially more capacity.
PCIe acts as the data transport layer.
The CPU provides additional compute capacity when transferring an expert to the GPU is not worthwhile.
For example, the described DeepSeek-V4-Flash configuration uses approximately 32GB of RTX 5090 VRAM alongside roughly 192GB of system RAM as a practical capacity target.
The important architectural shift is that the GPU no longer needs to contain the entire model.
It only needs enough capacity to hold the currently useful working set.
📊 Why FreeToken Changes the Local LLM Equation #
The conventional local LLM model is straightforward:
If the model does not fit in VRAM, inference becomes impractical.
FreeToken replaces that assumption with a different model:
If the hardware can move and compute the required data efficiently enough, the model does not need to fit entirely in VRAM.
That is a much more flexible proposition for sparse architectures.
The system can exploit the fact that only a subset of experts is required at any particular point while using RAM and PCIe to provide additional capacity.
The architecture is hardware-aware by design #
FreeToken does not assume that every consumer machine has identical characteristics.
A system with PCIe 3.0, a fast Threadripper CPU, and an RTX GPU has a different optimal execution strategy from a PCIe 5.0 system with a slower CPU.
The runtime therefore adapts its scheduling decisions to the actual hardware topology.
This is closer to an operating-system-style resource scheduler than a traditional GPU inference engine.
🏠 Edge AI Becomes More Practical #
The broader implication is significant for local AI.
Consumer PCs already contain considerable compute resources:
- Large DDR5 memory pools
- High-performance CPUs
- High-bandwidth PCIe interfaces
- GPUs with tens of gigabytes of VRAM
- Increasingly capable NPUs
The problem is that these resources are normally treated as separate components.
FreeToken attempts to unify them into a single inference system.
For sparse models, that can substantially expand the range of models that can be served locally.
🌍 Open Models Need Open Inference Infrastructure #
Open model weights have solved only part of the accessibility problem.
A developer can download a frontier-scale model without paying an API provider, yet still be unable to run it efficiently because the required GPU cluster is prohibitively expensive.
Projects such as FreeToken attack the second half of the problem: making open models computationally accessible.
This distinction becomes increasingly important as MoE models grow larger.
The total parameter count can continue increasing while the number of active parameters per token remains comparatively manageable.
If inference software can efficiently exploit that sparsity, increasingly capable models can become viable on consumer hardware.
🔮 Token Freedom Could Become a Realistic Goal #
FreeToken demonstrates a different direction for local LLM inference.
The critical breakthrough is not simply running a large model with aggressive quantization. It is treating VRAM, system RAM, CPU compute, GPU compute, and PCIe bandwidth as a coordinated resource pool.
Its core techniques—double-buffered prefetching, bandwidth-adaptive execution, agentic state reuse, and elastic VRAM resizing—address several of the most important bottlenecks in consumer-scale MoE serving.
If these techniques continue to mature, the practical definition of “AI-capable hardware” could change significantly.
The question may no longer be whether a model fits entirely inside a GPU.
Instead, it may be whether the available hardware can coordinate its memory hierarchy and compute resources efficiently enough to keep tokens flowing.
That is a much more interesting path toward local AI.
And if a single consumer GPU can eventually deliver genuinely interactive inference for models with hundreds of billions of parameters, the long-standing assumption that frontier-scale AI requires a data-center cluster could become considerably less relevant.