OpenAI has unveiled Jalapeño, a custom-designed AI inference chip built from a blank slate in partnership with Broadcom. Revealed at Hot Chips, the ASIC went from initial team hiring in mid-2024 to manufacturing tape-out in roughly 16 months—an exceptionally fast development cycle. According to benchmarking verified by SemiAnalysis using their InferenceX suite, Jalapeño currently beats Nvidia’s Blackwell architecture on performance per watt across nearly all tested scenarios.
The early performance numbers are highly competitive. Running the open-weight DeepSeek R1 model, Jalapeño hit over 700 tokens per second per user at concurrency 1. On other models, including Kimi-K2.5 and GPT-OSS, it pushed approximately 1,400 tokens per second per user. OpenAI achieved these metrics using single-token prediction (STP), bypassing speculative decoding and prefill-decode disaggregation entirely.
While the chip was built by OpenAI, it is not locked exclusively to their proprietary models. It operates as a generalized inference chip capable of running various workloads. To demonstrate this flexibility, OpenAI engineers ported Doom to the hardware using only Codex prompts.
The engineering focus of Jalapeño isn't just raw speed; it is tokens per megawatt. Data centers are currently constrained by power capacity, not just floor space or budget. As the report points out, grid interconnection delays routinely outpace hardware timelines, forcing operators to rely on behind-the-meter (BtM) power setups, like on-site gas turbines. Because power is a fixed, finite resource, the ability to convert a joule of energy into a token is the defining bottleneck of the AI industry. Packed with HBM4 memory, Jalapeño addresses this directly, even surpassing the early performance-per-megawatt figures of Nvidia’s upcoming Vera Rubin systems.
SemiAnalysis correctly notes that single-turn benchmarks do not fully replicate production workloads. Realistic, multi-turn workflows stress components like prefix caching and routers—which is why the firm prefers its AgentX benchmark suite to test long-context cache behavior. Still, the baseline throughput and the fact that Jalapeño matches Nvidia chips on GSM8k evaluations confirm it is a serious disruption to the hardware status quo.
What This Means for SMB Operations
Small and mid-sized businesses do not buy data center racks or negotiate behind-the-meter power contracts. However, the downstream effects of tokens-per-megawatt dictate exactly what kind of automation an SMB can afford to deploy.
Right now, inference cost and latency put a ceiling on operational AI. Building a reliable automation often requires agentic workflows—having the model run multiple steps, check its own output, and pull context from large documents before taking action. If you want an AI to cross-reference a 50-page PDF vendor contract against an incoming invoice, format the data, and update a CRM, that requires a massive context window and multiple inference calls. When you rely on current-generation compute, running that process across hundreds of invoices daily gets expensive, and the latency makes real-time applications sluggish.
When chips like Jalapeño drive down the energy required to produce a token, the cost of AI compute drops, and processing speed increases. This makes high-volume, low-margin administrative tasks viable for automation.
If 700 to 1,400 tokens per second becomes the new standard for AI infrastructure, businesses can build internal tools that feel instantaneous. Small teams will be able to deploy multi-step agents to read incoming client emails, categorize them by urgency, draft customized responses, and route them to the right department—without watching a loading screen or racking up a massive monthly API bill. Our team tracks these hardware developments because today's data center efficiency benchmarks become tomorrow's accessible, everyday business tools.