OpenAI’s ChatGPT performance team is facing two accelerations at once: a sharply rising user base and a much faster software delivery cycle driven by agentic coding. In a presentation recorded for QCon AI Boston 2026, Martin Spier argued that this combination is changing how performance engineering works, pushing teams to watch far more than GPUs and to automate the detection and repair of regressions across the full product path.
ChatGPT performance under sustained growth
Spier, who leads ChatGPT Performance at OpenAI, framed the problem as a scale story as much as a tooling story. He said ChatGPT began as a research preview in late 2022 and quickly became a product with strict latency, reliability and support requirements. He also said the service reached 1 million users in five days and later grew to 900 million weekly active users as of February this year.
That growth has not been smooth. According to Spier, launches can trigger major spikes in demand, including last year’s image-generation surge, when more than 700 million images were generated by over 130 million users in seven days. He said the global nature of ChatGPT adds architectural complexity as well, because teams must think about points of presence, application servers, GPUs, network connectivity and data replication across regions.
OpenAI says agentic coding is changing the pace of change
The second acceleration is inside the development process itself. Spier said agentic coding has increased the amount of code and logic moving through OpenAI’s systems, and that developers are now working on many tasks in parallel rather than one at a time. He described a shift from traditional, largely linear development to a more “multi-threaded” workflow where engineers may juggle seven, eight, nine or even 10 tasks concurrently.
He said the company’s shipping volume increased by 70% by late last year, based on figures from October and November, and that nearly every engineer at OpenAI now uses Codex weekly or daily. Spier also said every pull request is automatically reviewed by Codex, and that the tool is now used for more than coding, including production troubleshooting, metric analysis, data work, document drafting and even presentation preparation.
More code, more parallelism, more hidden cost
Spier’s central warning was that every new feature, refactor or configuration change carries a cost, even when the change seems small. Extra conditionals, requests, data structures and logs all consume from a shared budget of latency, CPU, memory and I/O. He said these small costs compound over time, and with the rate of change now much higher, the time between introducing a problem and noticing it is shrinking.
He argued that performance problems are not only about user experience. Slower systems also increase operating costs and reduce headroom for growth, which can then become a reliability issue when systems can no longer absorb spikes. In his view, performance engineering now has to keep pace with product and feature velocity rather than act as a slower, separate correction layer.
Why ChatGPT speed is more than inference speed
One of the presentation’s strongest themes was that AI teams often focus too narrowly on inference metrics. Spier said time to first token and tokens per second matter, but they are only part of the experience. For ChatGPT, he said teams must also measure what users feel at different points in the journey: the response to a user action, the moment the first useful output appears and the time until the task is truly complete.
He gave examples showing why those moments differ by use case. A simple factual question should feel quick, while an agentic coding workflow or image generation should be allowed more time. Spier also said teams need to break the total path into layers such as client work, networking, data fetching, serialization and tokenization, not just inference itself.
- User-facing timing: response to action, first visible value and task completion
- System layers: client, network, data retrieval, tokenization and streaming
- Workload differences: simple chat, reasoning workflows, agents and image generation
He said ChatGPT’s request path is broader than many people assume. Before a message reaches the model, the system may need to verify identity and plan state, check quotas, fetch long conversation histories, retrieve uploaded files and project context, and prepare everything for tokenization and context-window limits. After the model responds, the system still has to persist data and continue moving information through storage and processing layers.
Keeping pace with performance engineering
Spier’s answer to faster development is not to slow engineering down, but to accelerate performance work itself. He said OpenAI is exploring “always-on” AI agents that can run continuously in both reactive and proactive loops. In the reactive loop, agents can detect regressions, run or compare profiles, identify root causes, propose fixes and help validate the result. In the proactive loop, agents can keep looking for optimizations in hot paths, CPU-heavy methods, allocations and bundle size.
He said the ideal end state is a system where many perf-related tasks happen automatically, with humans setting direction and agents handling the repetitive work. But he stressed that this only works if the underlying engineering discipline is strong. Good test coverage, benchmarks, observability, clear contracts between components and safe rollout practices all become more important when agents are allowed to operate with less supervision.
Best practices that make agentic perf work
Among the practices Spier highlighted were full-loop automation, fast feedback and reliable context. Agents need a clear signal to know whether a change helped or hurt, and they need it quickly enough to avoid waiting on long manual cycles. He also said detailed written architecture information matters because agents do not have the same tribal knowledge humans rely on.
He added that many performance optimizations are essentially search problems. Agents can explore alternatives, benchmark them and compare outcomes faster than humans can, but only if teams define the right metrics and reproduce realistic workloads. If the measurement is wrong, he warned, the agent will optimize the wrong thing.
The broader lesson for AI teams
Spier’s presentation made a broader point than just ChatGPT internals. AI products are often discussed in terms of model quality and GPU capacity, but he argued that the broader system matters just as much. As agentic development raises the volume and speed of changes, the surrounding infrastructure, observability, developer tooling and performance workflows must evolve too.
For teams building AI products at scale, the message is straightforward: speed is now a system property, not just a model property. The companies that keep up will be the ones that treat performance engineering as a continuously automated discipline, not a periodic cleanup exercise.
Explore more: Blog Our Services Contact Us
Source: Original report
Was this helpful?
Last Modified: August 10, 2026 at 4:52 pm
2 views
