OpenAI Jalapeño ASIC: Architecture, Performance & Trade-Offs
OpenAI, in collaboration with Broadcom, has unveiled its first custom AI inference ASIC, Jalapeño, with the chip reportedly reaching tape-out in roughly 16 months. Rather than optimizing for peak FLOPS alone, Jalapeño is designed around a more production-oriented metric: Tokens per Megawatt (tok/s/MW), or equivalently token throughput per unit of energy.
That design target drives nearly every major architectural decision. Jalapeño reduces dependence on conventional GPU abstractions, tightly couples compute and memory, uses smaller systolic arrays to handle irregular workloads more efficiently, and keeps prefill and decode workloads within a unified hardware pool.
The result is an accelerator optimized primarily for high-throughput inference under strict data-center power constraints rather than a general-purpose training architecture.
🚀 Key Hardware Specifications #
| Metric | OpenAI Jalapeño (A0/B0) | NVIDIA GB200 (Blackwell) | NVIDIA Vera Rubin (VR200) |
|---|---|---|---|
| Primary Workload | General AI Inference | Training & Inference | Training & Inference |
| Process Node | TSMC N3P | TSMC N4P | TSMC N3P |
| TDP | 700 W | 1,000–1,200 W | 900–1,150 W per die |
| MXFP4 / NVFP4 Compute | 13.4 PFLOPS | ~10 PFLOPS | 17.5 PFLOPS |
| FP8 Compute | 3.4 PFLOPS | ~5 PFLOPS | ~8.7 PFLOPS |
| FP16 | Unsupported | 1,250 TFLOPS | Supported |
| Memory | 216 GiB HBM4 | 192 GB HBM3e | HBM4 |
| Memory Bandwidth | 15.4 TB/s | 8.0 TB/s | ~13–15 TB/s |
| HBM Bandwidth per Watt | ~22 GB/s/W | Lower | Moderate |
The specification profile makes Jalapeño’s design objective clear. It does not attempt to dominate every compute metric. Instead, the accelerator allocates substantial silicon and power budget to memory bandwidth and inference efficiency, where data movement frequently becomes the limiting factor for production LLM serving.
Its 700 W power envelope is particularly significant when deployed at rack scale. At the same time, 216 GiB of HBM4 and approximately 15.4 TB/s of bandwidth provide a large local memory subsystem for workloads that are often constrained by memory traffic rather than arithmetic throughput.
🧠 Four Architectural Trade-Offs #
Jalapeño departs from several conventional GPU design assumptions. The common theme is eliminating fixed overheads that become increasingly expensive as inference workloads move toward low batch sizes, variable sequence lengths, and highly concurrent serving.
Sliced Memory and Direct-Connected NoC #
The compute fabric and HBM are divided into 64 matched slices, creating a localized relationship between compute resources and memory.
Each compute slice has a low-latency view of its corresponding HBM slice, while a dedicated Collective Network handles synchronization and communication between slices.
This organization reduces the need to treat the accelerator as a single monolithic memory system. Instead, the architecture exposes locality directly in the hardware and network topology, allowing workloads to exploit predictable compute-to-memory paths.
Out-of-Order Cores and L1 Cache #
Jalapeño uses out-of-order (OoO) cores and L1 caches rather than relying primarily on software-managed scratchpads and explicit DMA engines.
The motivation is particularly relevant to inference. Software-managed movement can introduce synchronization barriers and expose latency that is difficult to hide when batch sizes are small. OoO execution provides additional scheduling flexibility, allowing the hardware to overlap operations and maintain utilization without requiring every dependency to be explicitly orchestrated by software.
The objective is to preserve roofline efficiency across less regular production workloads rather than only maximizing throughput under idealized high-batch conditions.
Smaller-Dimension Systolic Arrays #
Jalapeño also reduces the matrix dimensions of its weight-stationary systolic arrays.
Large fixed matrix engines can deliver excellent utilization when tensor dimensions align with their preferred tile sizes. However, irregular dimensions can introduce padding, inefficient tiling, and abrupt utilization losses.
Smaller systolic arrays reduce these performance cliffs. The trade-off is that the accelerator gives up some peak matrix throughput in exchange for more consistent utilization across the shapes encountered by real inference workloads.
Unified Pool Instead of Prefill-Decode Separation #
Jalapeño does not adopt dedicated hardware pools for prefill and decode, commonly referred to as Prefill-Decode Disaggregation (PDD).
Instead, it uses a unified accelerator pool that can dynamically absorb changing production traffic. This is important because inference workloads rarely maintain a stable ratio between prefill and decode demand. Context lengths, concurrency, and request composition can change continuously.
A unified pool avoids statically allocating hardware to either phase and eliminates the need to migrate KV caches across network boundaries. That can reduce both communication overhead and energy consumption, although it also places greater demands on the scheduler and runtime.
💻 Co-Designed Software Stack: Gluon and Codex #
Hardware efficiency alone is insufficient for a custom accelerator. Jalapeño is therefore paired with a software stack designed around the architecture rather than treating the ASIC as a drop-in GPU replacement.
Gluon Domain-Specific Language #
Gluon is a domain-specific language built on Triton. Its key abstraction is Linear Layouts, which use binary matrix algebra over (\mathbb{F}_2) to formally represent mappings between tensors, registers, and memory.
This approach is intended to make tensor layout transformations more tractable. Rather than relying on extensive combinations of manually specified layouts, the compiler can reason about transformations algebraically, reducing the combinatorial complexity associated with mapping tensors onto the underlying hardware.
For a custom accelerator, that level of control is important because conventional compiler abstractions may not expose enough information to exploit specialized memory and compute layouts efficiently.
Codex-Driven Kernel Generation #
OpenAI also uses an internal Codex-derived system to generate optimized kernels and prefetch strategies.
Rather than depending exclusively on manually written kernels or a conventional general-purpose compiler pipeline, the system uses detailed trace harnesses to evaluate generated implementations against actual execution behavior.
This creates a tighter feedback loop between software generation and hardware performance. Kernel implementations can be optimized according to observed bottlenecks instead of relying solely on static compiler heuristics.
Rapid Performance Iteration #
The software-hardware co-design approach reportedly enabled substantial performance improvements during development. The engineering team was able to double throughput within two weeks and expand tensor parallelism from TP8 to rack-level TP32 in approximately eight days.
For a custom inference ASIC, this iteration speed is strategically important. Hardware is effectively fixed after tape-out, so much of the remaining performance headroom must come from compiler, kernel, scheduling, and networking improvements.
🏗️ Rack System and Network Topology #
Jalapeño is designed as part of a complete rack-scale system rather than as an isolated accelerator.
Rack Composition #
The system consists of three major tray types:
- Katsu Host Tray: Dual AMD EPYC Turin CPUs with 1.5 TB of DRAM.
- Vindaloo ASIC Tray: A 1U chassis containing eight Jalapeño ASICs, contributing to a total of 128 XPUs per cabinet.
- Chana Switch Tray: Built around Broadcom Tomahawk 6 switches with 102.4 Tb/s switching capacity.
This architecture treats compute, host processing, and networking as a single co-designed system.
Two-Tier Scale-Up Fabric #
The scale-up network is divided into local and global domains.
The local domain connects 128 XPUs through an all-to-all topology using a 6,144-differential-pair passive copper backplane. Each XPU has up to 4.8 Tb/s of unidirectional bandwidth within the local domain.
The global domain extends connectivity through a hybrid copper-optical rail network incorporating Optical Circuit Switches (OCS). This enables scale-up to as many as 16 racks, or 2,048 XPUs.
The topology reflects the same principle found elsewhere in Jalapeño: keep high-bandwidth communication close to the compute fabric where possible, while using optical switching to extend scale without forcing every accelerator into an expensive full-mesh physical interconnect.
⚠️ Key Caveats and Open Questions #
Despite the attractive efficiency profile, Jalapeño’s published results should be interpreted within their benchmark context.
Limited Initial Workload Coverage #
The initial benchmarks were recorded using single-turn 8k1k workloads. That provides useful information about inference throughput, but it does not establish how the architecture will behave under substantially different production patterns.
Long-context, multi-turn, and agentic workloads can produce different memory-access patterns, KV-cache behavior, concurrency profiles, and synchronization requirements. Performance on workloads such as AgentX therefore remains an important open question.
The Relevant Competitive Baseline Is Vera Rubin #
Jalapeño may compare favorably with current-generation Blackwell hardware, but its planned 2027 volume ramp changes the appropriate competitive baseline.
By the time Jalapeño reaches significant production volume, NVIDIA’s Vera Rubin platform is expected to be the more relevant comparison. The key question is therefore not whether Jalapeño can outperform an earlier Blackwell configuration, but whether its inference efficiency and system-level economics remain competitive against the next generation of GPU infrastructure.
Software Headroom Remains #
Another important caveat is that the reported benchmark results were obtained using Single-Token Prediction (STP) without Multi-Token Prediction (MTP) or speculative decoding.
That leaves potential optimization headroom in the software stack. If MTP and speculative decoding can be implemented efficiently on Jalapeño, effective token throughput could increase beyond the initial benchmark results.
However, these techniques also introduce additional scheduling, memory, and acceptance-rate considerations. Their real-world benefit will depend on workload characteristics rather than simply adding their theoretical compute advantage to the published numbers.
🔍 What Jalapeño’s Design Signals #
Jalapeño is best understood not as an attempt to build a general-purpose GPU replacement, but as an example of inference-first accelerator design.
Its architecture makes several deliberate compromises: lower emphasis on peak FLOPS, no FP16 support, smaller systolic arrays, localized HBM, OoO execution, unified prefill/decode resources, and a software stack built specifically around the accelerator.
Those choices are coherent with a production metric centered on tokens per unit of energy. For large-scale AI inference, the economic bottleneck is increasingly determined by the cost of moving weights and KV caches, maintaining memory bandwidth, serving variable workloads, and operating thousands of accelerators within a fixed power envelope.
The more important test for Jalapeño will therefore be system-level economics rather than a single peak-performance number. Its eventual competitiveness will depend on how effectively the ASIC, HBM subsystem, compiler, kernel-generation stack, networking fabric, and serving software operate as one integrated platform.