May 29th, 2025 by Adam Sandman
Artificial intelligence is revolutionizing software development. But with great power comes great... hallucinations. During a recent executive briefing with our partners at AWS and Cloud303, we shared some thoughts on this growing challenge—and how the structure of software development helps mitigate, but not eliminate, the risk.
From Programmer to AI Evangelist
As someone who started coding over 25 years ago and now leads a company at the forefront of AI-enabled DevOps, I've seen firsthand how fast the landscape is shifting. AI isn't just augmenting code—it’s creating it. But we must stay grounded in how we assess its output.
Built-in Checks: Why Code Is Special
In the software world, we benefit from certain natural constraints. Code has to compile. It has to pass unit tests. These built-in safeguards provide real-time feedback that helps detect and reduce the impact of AI hallucinations. That’s not something writers, designers, or analysts enjoy in the same way.
But while code compilation and testing catch some errors, they don’t catch all.
When AI Fills in the Blanks—Wrongly
I gave an example in the session that reflects this. AI sometimes “hallucinates” fictitious API endpoints when it encounters poor documentation. Instead of responding with “no data available,” it extrapolates patterns based on known endpoints and invents something that sounds plausible—but doesn't exist.
Imagine integrating with an API where the AI confidently returns a /getCustomerProfile endpoint because it saw /getUserProfile and /getOrderDetails. It might compile. It might even pass initial tests. But when the integration fails in production, you're chasing a ghost.
The False Sense of Security
This is the illusion of functionality. Just because code runs doesn't mean it delivers value. In fact, one of the challenges we discussed is that AI-generated code can increase testing bottlenecks by flooding systems with seemingly valid but ultimately unhelpful logic.
One of our partners even observed a 300% increase in code quantity paired with a 400% decrease in code quality. AI can produce more, faster—but not necessarily better.
Why It Matters for Enterprises
In the executive session, we also talked about how the industry is evolving through phases:
- Phase 1: Low-hanging fruit like AI-generated documentation, faster code drafting, and knowledge base expansion.
- Phase 2: Workflow automation and IT modernization, like Amazon’s own AI-assisted migration that reportedly saved 4,500 years of development time.
- Phase 3: Deeper personalization and data-driven intelligence—like detecting customer churn risk from support transcripts before a human could spot it.
These phases make one thing clear: hallucination isn't just a bug; it's a business risk. And enterprises must treat AI like a junior developer—review everything with scrutiny.
Inflectra’s Approach
At Inflectra, we’ve integrated AI into our Spira platform, working with Amazon Bedrock. Our Phase 1 strategy allowed customers to bring their own LLMs. Now, in Phase 2, with Inflectra.ai, we're embedding native support using Amazon Bedrock and Amazon Nova LLMs to improve performance and cost efficiency (up to 75% savings).
We're also using guardrails for data security and prompt management to ensure consistency. In the near future, we're building retrieval-augmented generation (RAG) capabilities that mine our rich project data for real-time insights.
Hallucinations, Handled the Right Way
So where does that leave us? AI in development is not a hands-off solution. It’s a powerful co-pilot that demands thoughtful oversight. As I said in the briefing, we’re entering a phase where autonomous testing and intelligent orchestration will reshape software quality.
But we won’t get there without discipline. Validate outputs. Classify your data. Use AI to enhance human intelligence, not bypass it.
AI may write the code—but humans must still write the rules.