AI systems are reaching production faster than many organizations can adapt their engineering, security, and approval processes. In a recent InfoQ webinar, senior practitioners argued that the hardest problems are no longer just model quality or prompt design, but silent failures, unsafe automation, and governance gaps that traditional software controls were never built to catch.
What breaks first in AI in production
The panel, titled “AI in Production: What Breaks, What Works, and Who Approves It?”, was moderated by Hien Luu and featured Zichuan Xiong and Prem Chandrasekaran from ThoughtWorks, privacy specialist Katharine Jarmul, and financial AI practitioner Lan Chu. Luu framed the discussion around three pressures: AI can fail by producing confident but wrong answers rather than crashing; coding agents can produce production code faster than humans can review it; and sensitive data can leak through prompts, context windows, and retrieved documents.
Xiong described “semantic drift” in LLM pipelines, saying that text-to-SQL translation agents can flatten domain-specific meaning in ways that look correct on the surface but distort business logic underneath. He pointed to cases where a term such as “member” may be treated the same across sales and customer support, even when the word carries different operational meanings in each domain.
Those errors can be especially dangerous because they may pass ordinary tracing and monitoring. Instead of an outage, teams get subtly wrong analytics, incorrect joins, or decisions based on data that appears valid until it is checked against real business context.
Silent failures and embedding drift
Chu described a separate production problem: “embedding drift” in financial compliance systems. In one example, an agent used vector similarity thresholds to block prohibited investment advice, but an unannounced background update to the embedding model changed the vector distance distribution.
The result was severe spikes in both false positives and false negatives during live traffic, while all infrastructure health checks remained green. The example underscored a key point from the webinar: AI systems can degrade without the classic signals that operators rely on, making observability and evaluation much more important than simple uptime checks.
Chandrasekaran added that retrieval-augmented generation pipelines can accidentally surface sensitive documents, while Jarmul warned that models can memorize training data and be used to de-anonymize information or expose API keys. Chandrasekaran also referenced the GitSpawn vulnerability, which shows how malicious repository configurations can execute unauthorized code during routine commands.
Why code review alone no longer scales
As coding assistants generate more production code, the panel said the traditional model of line-by-line human review becomes a bottleneck. Chandrasekaran, Tech and Business Services Lead at ThoughtWorks, argued that manual review cannot keep up with the volume and speed of agent-generated changes.
His recommendation was to automate tactical checks such as style and syntax within CI/CD pipelines, while reserving human attention for architecture, business intent, and non-functional requirements. In other words, teams should use people where judgment matters most, not as the last defense against every minor code issue.
Xiong said teams should also “shift left” and assess intent at the harness level before pull requests are even created. Chandrasekaran highlighted mutation testing as a useful practice here, since deliberately altered code can reveal whether a test suite is truly validating logic or merely satisfying coverage metrics.
AI evals as living infrastructure
Chu emphasized that evaluations should not be treated as a one-time checklist before launch. Instead, evals need to operate as permanent infrastructure that evolves alongside the system in production.
That approach includes continuous online tracing at the span level, log filtering, and using LLMs to categorize failure patterns. The goal is to feed real-world edge cases back into offline benchmark suites so that production behavior helps improve future testing.
The webinar repeated this message in different forms: agentic systems are too dynamic for static approval gates alone. If the system learns, retrieves, or changes behavior over time, then the evaluation process has to move with it.
Governance for autonomous agents
On privacy and security, Jarmul cautioned against relying on prompt engineering as a primary defense. She instead argued for systematic red teaming, ongoing threat modeling, strict functional scope for autonomous agents, and observability that can handle higher event volumes.
Chandrasekaran said organizations should treat each AI agent like a new engineering hire. That means giving it a distinct non-human identity, applying tiered access controls that separate read-only actions from reversible write actions, versioning third-party Model Context Protocol (MCP) dependencies, and keeping API secrets outside the agent runtime environment in sandboxes.
The panel also drew a line between technical validation and final business approval. Engineering, security, and privacy teams can determine whether the guardrails are adequate, but the decision to deploy remains a strategic one for the business.
Key takeaways for teams deploying AI systems
- AI failures may be silent, producing plausible output rather than obvious crashes.
- Semantic drift and embedding drift can damage production systems without breaking infrastructure health checks.
- Human code review alone cannot absorb the volume of agent-generated code.
- CI/CD automation, mutation testing, and harness-level checks are becoming more important.
- Evals should be continuous, production-informed, and treated as living infrastructure.
- Autonomous agents need strict identity, access, and dependency controls.
The broader message from the webinar was pragmatic rather than speculative: production AI needs operational discipline, not just model capability. As organizations move from pilots to real deployments, the work shifts from asking whether a model can produce a result to asking how often it fails, how those failures are detected, and who is accountable when it does.
Source: Original report
Was this helpful?
Explore more: Software Testing Services More AI & Automation Tech News
Last Modified: October 18, 2026 at 10:32 pm
0 views
