NVIDIA Boosts Local AI Inference by Up to 1.9x on RTX and DGX

NVIDIA Boosts Local AI Inference by Up to 1.9x on RTX and DGX

NVIDIA is expanding its local AI ecosystem across RTX and DGX platforms with improvements aimed at two major bottlenecks: inference performance and deployment complexity.

The latest updates combine low-level inference optimizations developed with the llama.cpp and vLLM open-source communities with simplified deployment support for AI agents on NVIDIA GPUs equipped with at least 24GB of VRAM.

According to NVIDIA’s SpeedBench-Coding 8K Throughput benchmark results from AIPerf, the combined optimizations can deliver up to a 1.9x performance improvement in specific local-agent workloads. Actual gains depend on the GPU, model, backend, quantization, workload, and system configuration.

The broader goal is straightforward: make local AI inference faster while reducing the amount of manual configuration required to run capable AI agents locally.

🚀 NVIDIA Targets the Local AI Inference Stack

#

NVIDIA’s latest local AI push can be divided into two layers:

1. Inference-engine optimization through llama.cpp and vLLM.
2. Application-level deployment automation for local AI agents.

The first layer focuses on improving tokens-per-second and end-to-end inference latency.

The second focuses on making local AI applications easier to install, configure, and operate without requiring users to manually select models, download weights, tune parameters, or configure inference backends.

This distinction is important because local AI performance is no longer determined solely by GPU compute capability.

The software stack—including model quantization, attention kernels, speculative decoding, memory management, prefill efficiency, and backend scheduling—can have a significant impact on real-world throughput.

⚡ llama.cpp Optimizations Push RTX 5090 Throughput Higher

#

NVIDIA’s collaboration with the llama.cpp community targets efficient local inference on consumer and professional RTX GPUs.

According to NVIDIA’s benchmark results, the GeForce RTX 5090 sees substantial token-throughput improvements when running the specified Qwen models.

The larger improvement on the 35B model illustrates an important point: inference optimization does not necessarily scale uniformly across model sizes.

A backend optimization can have different effects depending on:

• Model architecture
• Parameter count
• Quantization format
• Attention workload
• KV-cache requirements
• GPU memory bandwidth
• Kernel utilization
• Prompt and generation characteristics

For local users, the practical result is higher token throughput without changing the underlying GPU.

🔧 vLLM Extends Optimization Across RTX and DGX

#

NVIDIA’s vLLM work covers both workstation-class RTX hardware and DGX Spark systems.

Reported benchmark results include:

The differences between these results again highlight how strongly inference performance depends on the combination of hardware, model, backend, and workload.

For enterprise and workstation deployments, vLLM is particularly relevant because it provides a production-oriented inference serving stack rather than simply acting as a local command-line runtime.

XQA Kernels Improve Attention Efficiency

#

One of the key components of NVIDIA’s vLLM optimization work is the introduction of XQA attention kernels from FlashInfer, combined with backend-level improvements.

Attention is one of the most computationally and memory-intensive portions of transformer inference, particularly as context lengths increase.

Efficient attention kernels can improve GPU utilization while reducing the overhead associated with processing queries, keys, and values.

The benefit becomes particularly relevant for agentic workloads because AI agents frequently maintain longer contexts and repeatedly process accumulated conversation, tool outputs, retrieved information, and task state.

Faster Prefill Improves Interactive Workloads

#

NVIDIA also attributes the performance gains to improvements in the prefill phase of inference.

LLM inference can broadly be divided into two phases:

• Prefill: processes the input prompt and builds the initial KV cache.
• Decode: generates output tokens sequentially.

Prefill is generally more compute-intensive, while decode can become increasingly constrained by memory bandwidth and KV-cache movement.

For local AI agents, faster prefill can significantly improve responsiveness when a task involves:

• Long prompts
• Large context windows
• Tool results
• Retrieved documents
• Multi-turn conversations
• Agent planning loops

Improving only decode throughput is therefore not sufficient for many real-world agent workloads.

🧠 Speculative Decoding Becomes More Important

#

Another component of NVIDIA’s optimization strategy is enhanced speculative decoding.

Instead of relying exclusively on a large model to generate every token sequentially, speculative decoding uses a smaller or faster model to propose candidate tokens. The larger model then verifies those candidates in batches.

When the proposed tokens are accepted at a high rate, the system can generate multiple output tokens with fewer expensive large-model decoding steps.

Conceptually:

Small / draft model
        │
        ▼
Generate candidate tokens
        │
        ▼
Large target model
        │
        ▼
Verify candidates
        │
   ┌────┴────┐
   │         │
Accept     Reject
   │         │
   ▼         ▼
Continue   Regenerate

The effectiveness of speculative decoding depends heavily on model pairing, acceptance rate, workload characteristics, and implementation efficiency.

That is why backend-level optimization can produce substantially different results across models.

💻 Local AI Deployment Expands to 24GB+ GPUs

#

Performance is only half of NVIDIA’s strategy.

The company is also simplifying local deployment for GPUs with 24GB or more of VRAM.

The initial rollout covers three AI agent applications:

• Perplexity Portable Computer
• Hermes Agent
• OpenClaw

The 24GB threshold is significant because local agent workloads often require substantially more memory than a basic chatbot.

A local agent may need to simultaneously accommodate:

• Model weights
• KV cache
• Runtime overhead
• Tool execution state
• Context history
• Embeddings or retrieval data
• Multiple processes

The exact memory requirement depends heavily on model size, quantization, context length, and application architecture, so 24GB should be viewed as a deployment target rather than a universal minimum for every model.

🔍 Perplexity Portable Computer Brings Research Workloads Local

#

Perplexity Portable Computer was previously introduced with support for DGX Spark and Linux systems equipped with RTX GPUs.

The application combines models, orchestration tools, and supporting components into a unified local AI workflow.

The planned September expansion brings the application to Windows and Linux systems using RTX GPUs with at least 24GB of VRAM.

Users can execute complete workflows locally without consuming paid API credits.

When a task requires stronger research or reasoning capabilities, the system can selectively access more than 15 cloud-based frontier models.

Importantly, users are asked for permission before content is sent to the cloud.

This creates a hybrid architecture:

                  ┌──────────────────┐
                  │   Local RTX GPU  │
                  │                  │
                  │ Local model      │
                  │ Agent runtime     │
                  │ Local data       │
                  └────────┬─────────┘
                           │
                    User permission
                           │
                           ▼
                  ┌──────────────────┐
                  │ Cloud Frontier   │
                  │ Models           │
                  └──────────────────┘

The approach allows users to keep routine or sensitive workloads local while selectively using cloud models for tasks that benefit from additional reasoning or research capabilities.

🤖 Hermes Agent Automates Local Configuration

#

Developed by Nous Research, Hermes Agent is a general-purpose AI agent designed to operate across models and providers.

The system is intended to run continuously on local hardware, including:

• RTX PCs
• RTX PRO workstations
• DGX Spark

Its upcoming one-click configuration feature is designed to remove much of the manual setup traditionally associated with local AI agents.

On supported Windows and Linux systems, Hermes can automatically detect the NVIDIA GPU and select an appropriate model and configuration.

The application then launches through a llama.cpp backend incorporating NVIDIA’s inference optimizations.

This eliminates several common setup steps:

• Manually identifying GPU capabilities
• Selecting compatible model variants
• Downloading model files
• Choosing runtime parameters
• Configuring inference backends
• Tuning deployment settings

For experienced developers, these options can still matter for optimization. For general users, however, automated configuration significantly reduces the barrier to entry.

Agent Memory and Tool Use

#

Hermes is designed to go beyond basic text generation.

It can:

• Invoke external tools
• Maintain context across tasks
• Retain information between sessions
• Generate reusable skills
• Incrementally improve its task capabilities

This makes local inference more interesting than simply running a chatbot.

The GPU becomes the execution layer for an autonomous software workflow rather than merely a device for generating text.

🦞 OpenClaw Lowers the Barrier to Local Agent Deployment

#

OpenClaw is another major project in NVIDIA’s local-agent strategy.

The open-source AI agent project has accumulated more than 380,000 GitHub stars and has developed an ecosystem of tools and skills covering areas such as research, engineering, project management, and productivity.

NVIDIA, Microsoft, and the OpenClaw community are collaborating to simplify deployment on Windows PCs.

The planned Windows application will configure optimized local models for RTX GPUs that satisfy the 24GB VRAM requirement.

This approach shifts local AI deployment from a developer-centric workflow toward a more application-oriented experience.

Instead of starting with a runtime and manually assembling the software stack, users can increasingly start with the application they want to run.

🏗️ From GPU Hardware to a Complete Local AI Stack

#

NVIDIA’s strategy is increasingly focused on the entire local AI stack rather than GPU hardware alone.

The stack can be viewed as several layers:

This architecture is important because improvements at one layer can amplify the value of improvements elsewhere.

A faster GPU does not automatically produce faster local AI if the inference runtime is inefficient.

Likewise, an optimized inference backend provides limited practical value if deployment is too complicated for users to configure correctly.

NVIDIA is therefore addressing both sides of the problem simultaneously.

📊 What the 1.9x Performance Claim Actually Means

#

The headline 1.9x performance improvement should be interpreted carefully.

It does not mean that every RTX or DGX workload becomes 1.9x faster.

The reported number comes from specific SpeedBench-Coding 8K Throughput benchmarks and represents the maximum improvement observed under the tested configurations.

Actual performance depends on variables such as:

• GPU model
• Model architecture
• Model size
• Quantization
• Context length
• Batch size
• Prompt-to-output ratio
• Backend implementation
• Kernel selection
• Memory bandwidth
• Speculative decoding acceptance rate

For developers evaluating local inference, the most meaningful metric is therefore not a single headline multiplier but the performance of the exact GPU + model + runtime + workload combination being deployed.

🔐 Local AI Gains a Privacy and Cost Advantage

#

The deployment improvements also reinforce two of the strongest arguments for local inference: data control and operating cost.

Running workloads locally can eliminate per-request API charges for routine inference.

It can also keep sensitive prompts, documents, source code, and internal information on the user’s machine when cloud access is unnecessary.

However, local inference is not automatically private or isolated.

Applications can still communicate with external services, download models, invoke APIs, or use cloud-based capabilities.

The hybrid approach used by Perplexity Portable Computer illustrates a more practical model:

Run locally by default, use the cloud selectively, and obtain permission before sending data externally.

For enterprise and developer workloads, this model can provide a useful compromise between local data control and access to larger frontier models.

🔮 NVIDIA Is Turning RTX Into a Local AI Platform

#

The significance of these updates extends beyond individual benchmark improvements.

NVIDIA is attempting to transform RTX and DGX systems from general-purpose GPU platforms into increasingly complete local AI execution environments.

The strategy combines:

• Faster inference kernels
• Improved vLLM and llama.cpp backends
• Speculative decoding
• Faster prefill
• Optimized attention
• Automated model selection
• One-click agent deployment
• Local-first execution
• Optional cloud model access

This addresses a fundamental weakness of local AI: the hardware has become capable enough to run increasingly sophisticated models, but the software and deployment experience can still be complicated.

The next stage of local AI adoption will therefore depend not only on GPU performance, but also on how efficiently the surrounding software stack converts that hardware into usable inference throughput.

NVIDIA’s latest updates target precisely that gap.

The long-term direction is increasingly clear: local AI is moving from manually configured model runtimes toward integrated agent platforms where the GPU, inference engine, model, tools, and deployment workflow operate as a unified stack.

For users with 24GB or more of GPU memory, that shift could make capable local AI agents substantially easier to deploy—and for developers, the continuing optimization of llama.cpp, vLLM, FlashInfer, and NVIDIA’s GPU software stack could deliver meaningful performance gains without requiring a hardware upgrade.