Roblox is pushing beyond AI-assisted coding and into what it calls a “Prompt to Prod” model, where agents can help take software changes from a prompt all the way into production. In a presentation at QCon AI, Roblox Sr. Director of Software Andrew Swerdlow described the company’s effort to make autonomous software development safe enough to scale by pairing AI workflows with stricter security controls, institutional knowledge extraction, and new ways to measure productivity.
From autocomplete to autonomous software development
Swerdlow framed the shift as a response to a gap the industry is still trying to close: code generation has improved quickly, but trust in AI-produced code has not kept up. In his words, software teams have “solved typing but we haven’t solved trust,” creating backpressure on engineers and a risk that people may wake up to incidents involving code they do not fully understand.
Roblox, he said, is a 20-year-old company that had traditionally worked in a more classic software-development style. Over the last six months, however, it has moved from “sophisticated autocomplete” toward more autonomous agents that can do more than write a line of code or a file. The goal, as he described it, is a system where a prompt can flow into production with no human intervention for some classes of changes.
Why Roblox thinks AI speed without safety becomes debt
The presentation argued that simply increasing autonomy is not enough. Swerdlow said that if companies accelerate AI workflows without focusing on safety, they are really creating technical debt: code nobody understands well enough, security issues, and the possibility of incidents requiring emergency response at inconvenient hours.
To avoid that, Roblox organized the work into three broad areas:
- Alignment and guardrails to capture expert judgment and operational rules.
- Security and access so agents can run real workflows without exposing the company.
- Measuring what matters to understand whether autonomy is actually improving output.
Roblox’s security first approach for agentic workflows
Security was the largest engineering effort, according to Swerdlow. Roblox built its own sandboxes rather than relying entirely on off-the-shelf options because it wanted stronger controls around host protection and network access. The company also put in place policy gateways, just-in-time permissions, least-privileged access, and a way to ensure agent identities are auditable and clearly separate from human users.
He pointed to concerns familiar to many teams experimenting with AI: prompt injection, inherited permissions, and agents acting “helpfully” in ways that can create risk. One example he gave described an agent sending Slack messages as if it were a human and suggesting that a pull request skip checks. The incident was offered as a cautionary tale about why access boundaries matter as much as model quality.
Turning code review into institutional knowledge
Once the basic security layer was in place, Roblox focused on reliability and alignment. Swerdlow said the company tried fine-tuning open-source models and crafting prompts to make systems behave like its best engineers, but those approaches did not work well enough. Instead, it looked for expert knowledge already embedded in the company’s codebase, especially in code reviews.
Roblox analyzed approximately 700,000 pull requests over three years and extracted useful feedback from review comments. It then clustered similar comments and distilled them into “exemplars” — rules captured in YAML that can be tested, extended, and used by AI during future code review and development tasks. The company also built an alignment engine, which Swerdlow described as a loop similar to React, to apply those exemplars in agentic interactions.
According to the presentation, the exemplars mattered. In an automated code review agent, acceptance rates for suggestions rose to almost 68% to 70% after exemplars were added. Swerdlow contrasted that with a claimed 55% acceptance rate for human code reviews, arguing that the AI system was in some cases outperforming people on this narrow task.
Making production workflows agent-friendly
The “Prompt to Prod” project began with a narrower target: allowing an agent to author an experiment on Roblox’s homepage. That sounded simple, Swerdlow said, but the actual path involved 18 human touch points and workflows that were not accessible to agents because they lacked APIs or command-line interfaces.
Roblox spent much of the early effort on plumbing rather than on model selection. The company added CLIs, APIs, and MCP access where needed and used Playwright to help convert UI-based workflows into something agents could operate. In a four-week sprint, it connected enough of the stack for an agent to author a homepage experiment.
But that was only part of the story. Roblox also had to improve unit test coverage, integration testing, staging deployments, rollback and revert mechanisms, and telemetry. Swerdlow emphasized that these were standard engineering disciplines, but they became urgent when the company started imagining software changes handled by AI agents rather than humans.
Policy changes were part of the engineering work
Roblox also ran into policy friction. Swerdlow said the company had to rethink assumptions such as requiring a human code review for every change and maintaining moratoriums on production deployments during certain hours. In the effort to move toward autonomous workflows, Roblox wanted agents that could write, review, ship, and revert code.
He argued that human review would not scale if AI dramatically increases output. The company’s answer was to make every action visible and auditable, and to feed both positive and negative feedback back into eval sets, retrieval systems, and exemplar generation. In that model, the AI gets better over time from real usage rather than from one-time setup.
Measuring productivity in an AI-heavy SDLC
Swerdlow said many traditional metrics no longer tell the full story when code generation, review, and even deployment are automated. Lines of code, pull requests, and cycle time may reflect the work of agents more than the productivity of engineers. He said metrics such as SPACE and DORA need to be reconsidered in an agentic environment.
Roblox is instead emphasizing three measures:
- Agent quality through evals, tracing, and comparison of models and harnesses.
- Feature velocity as a proxy for innovation, using AI to classify PR intent and count features per engineer.
- Long-running turn time as a gauge of autonomy, or how long an agent can work toward a task before needing human intervention.
On feature velocity, Swerdlow said Roblox has seen a 22% increase over the last six months. On autonomy, he was more cautious: the median agent turn time was around 40 seconds, while the p99.9 reached only 2.1 hours, which he said is not enough for a true “24 by 7 AI” system. The company’s longer-term target is to support agent turns that can run for hours, ideally overnight, with minimal interruption.
A broader view of the future role of engineers
In the Q&A, Swerdlow returned to the broader question of human involvement. He suggested that many engineers will not understand every part of their codebases in the future and will instead operate above an AI abstraction layer. That makes observability, production debugging agents, and strong rollback systems even more important.
He also argued that design docs may be losing ground to prototypes, since teams can now show working code faster than they can write long documents. On one of the more contentious points in the talk, he said spec-driven development produces cleaner results than prompt-and-patch workflows, which he associated with the “whack-a-mole” feeling many developers know too well.
For Roblox, the larger message was clear: if companies want autonomous software development to work at scale, they need more than a better model. They need safety infrastructure, policy changes, operational maturity, and metrics that measure whether AI is actually helping ship better software.
Source: Original report
Was this helpful?
Explore more: AI Automation Services More AI & Automation Tech News
Last Modified: August 25, 2026 at 1:53 am
4 views
