From AI Experiments to AI Engineering: My Takeaways from the Gartner Application Innovation & Business Solutions Summit

AI Value Still Starts With Humans

The opening Gartner keynote, “Unlock the Human-AI Multiplier to Go From Pilot to Payoff,” was presented by Gartner analysts Jason Wong and Brent Stewart. The premise was particularly timely: organizations have already conducted extensive experimentation with AI, but the challenge now is moving beyond interesting pilots toward measurable and repeatable business value.

One concept that particularly resonated with me was Gartner’s AI value formula, which emphasized that several foundational elements need to be in place before AI initiatives can generate meaningful results. Human ingenuity was presented as a multiplier within that equation, and I think that distinction is important. There is sometimes a tendency to frame AI primarily as a replacement for people, whereas the greater opportunity is often to use AI to augment people who already understand the business, its customers, its processes, its constraints, and its objectives.

As models themselves become increasingly commoditized, simply having access to an LLM is unlikely to remain a meaningful competitive advantage. Differentiation will increasingly come from the context and knowledge an organization can provide to that model, the systems it connects to, and the ingenuity of the people determining how it should be used. This strongly reinforces something we have been saying at Inflectra: context and knowledge are becoming the new oil of the AI economy.

A New Architecture for the AI Enterprise

Another interesting part of the opening keynote was Gartner’s discussion of an emerging architecture for AI-enabled applications. What struck me was that some of the concepts felt surprisingly familiar. Anyone who has worked in enterprise software for long enough has seen architectural models evolve from client/server systems through n-tier architectures, service-oriented architectures, microservices, cloud-native systems, and now agentic architectures. The technologies change, but the need for clear separation of concerns remains.

In an AI-enabled enterprise, I increasingly think there is an important relationship between systems of record, systems of action, and systems of intelligence: Systems of record contain trusted business information, systems of intelligence reason across that information using LLMs and agents, and systems of action provide the tools and APIs through which those agents can actually accomplish work. This distinction is highly relevant to what we are building at Inflectra. Spira has traditionally served as a system of record for requirements, tests, risks, releases, defects, and other lifecycle information, while APIs, MCP, Inflectra.ai, and our broader agentic capabilities increasingly allow intelligent systems to reason over that information and take action.

For me, this was one of the strongest confirmations that enterprise AI cannot simply be treated as an LLM bolted onto an existing application. The underlying systems, contextual information, interfaces, governance, and available actions all need to work together if organizations are going to build reliable and useful AI-enabled applications.

Testing With AI and Testing AI Are Two Different Challenges

Another session particularly close to my heart was led by Gartner VP Analyst Joachim Herschmann, who discussed the strategic trends changing software engineering. One of the most important distinctions is between two related but fundamentally different concepts: using AI to test software and testing software that contains AI.

The first is already transforming software quality. AI can help generate test cases, translate requirements into tests, maintain automation, analyze failures, identify coverage gaps, and help testers work more efficiently. At Inflectra, we are already investing heavily in this area across Spira, Rapise, Inflectra.ai, and our agentic integrations. However, testing AI itself introduces a very different problem because traditional software testing usually assumes deterministic behavior. Give a conventional application the same input under the same conditions, and you generally expect the same output. Generative AI and autonomous agents break that assumption.

An LLM may produce several different answers that are all acceptable, while an agent may select different tools or take a different sequence of actions and still achieve the correct outcome. Testing therefore becomes increasingly probabilistic rather than purely deterministic. Instead of simply asking whether the expected result was returned, teams also have to consider how frequently the system succeeds, whether the response is semantically correct, whether the appropriate tools were selected, whether the overall workflow completed successfully, and whether the behavior remains reliable over time.

Techniques such as LLM-as-a-Judge become useful alongside conventional testing approaches because correctness increasingly needs to be evaluated semantically rather than simply through exact matching. For the testing industry, this represents a significant shift in both methodology and mindset.

The Hard Part Isn't Necessarily the Coding Agent

Another observation from Joachim’s discussion that I found particularly interesting was the growing importance of context management and harness engineering. Much of the current excitement around AI in software engineering focuses on coding agents. Tools such as Claude Code, Codex, Cursor, and others can generate substantial amounts of working code extraordinarily quickly, but generating code is increasingly not the hardest part of the problem.

The more difficult challenge is ensuring that an agent understands why something is being built, the requirements it should satisfy, architectural constraints, existing decisions, coding standards, test results, risks, dependencies, and business objectives. Without that context, organizations may simply generate bad software faster. This is why I think the relationship between AI coding tools and systems such as Spira is so important. The AI agent is extremely powerful, but its usefulness increases dramatically when it has access to the structured context surrounding the software development lifecycle.

Requirements, tests, defects, architectural decisions, risk information, and release plans all become fuel for the agent. In that sense, the competitive advantage shifts away from simply having the latest model and toward having the best context architecture around that model.

The Battle of the AI APIs

On the second day, I attended a particularly interesting session from Gartner Distinguished VP Analyst Mark O’Neill titled “Battle of the APIs 2026: What’s Up With MCP, A2A, UTCP, REST... Did Someone Say SOAP?” It provided an excellent overview of how the API ecosystem is evolving in response to AI and agentic applications.

For many years, REST has been the dominant approach for connecting applications. AI agents are now introducing additional protocols and approaches, including MCP, A2A, and others, each designed to solve somewhat different problems. One of my main takeaways was that organizations should be cautious about assuming that every new protocol automatically replaces what came before it. REST is not suddenly obsolete simply because AI agents now exist.

At the same time, MCP has emerged as an extremely useful mechanism for exposing tools and context to AI systems in a standardized way. That was reassuring for us at Inflectra because we have designed our own agentic capabilities around a combination of REST and MCP, rather than attempting to force every interaction through a single protocol. I expect these standards to continue evolving quickly, and the key for software vendors will be maintaining architectures flexible enough to participate in this changing ecosystem without tying customers to one particular model, agent framework, or protocol.

Governance Should Help Organizations Say Yes

One of my favorite sessions was from Gartner Sr Director Analyst Max Goss, who discussed governance for generative and agentic AI. Governance sometimes has a bad reputation in technology organizations because it can be perceived primarily as the function responsible for saying “No” — no, you cannot use that model; no, you cannot connect that data; no, you cannot deploy that agent.

Max made an important point that effective governance should instead create the processes and technical guardrails that enable organizations to say yes safely. If organizations simply prohibit the use of AI, employees will often find their own alternatives, creating the risk of shadow AI where unsanctioned systems are used without appropriate security, data controls, auditing, or governance.

That may ultimately create greater risk than providing an approved environment with clear boundaries, controls, and oversight. For me, the key takeaway is that governance needs to become an enabler of responsible experimentation rather than simply a barrier to adoption. The organizations that manage this balance well will likely be better positioned to innovate without losing control of security, compliance, or operational risk.

AI Agents Need Observability

The governance discussion also highlighted an area that I believe many organizations are underestimating: what happens after an AI agent goes into production. Deploying an agent is not the end of the lifecycle, because organizations still need visibility into how that agent is behaving over time.

Traditional applications are monitored for errors, performance degradation, security incidents, and availability, but autonomous AI systems introduce additional challenges. An agent’s behavior may drift, its underlying model may change, the systems it interacts with may evolve, and its available context may change. A workflow that performed reliably last month may begin producing different results without the sort of obvious failure that conventional monitoring systems are designed to detect.

Organizations therefore need mechanisms to continually evaluate what their agents are doing and identify when their behavior starts moving outside acceptable boundaries. This creates an important convergence between testing and observability, because testing can no longer stop when an agent reaches production. In many cases, AI assurance needs to continue throughout the complete operational lifecycle.

Individual AI Productivity Does Not Automatically Create Team Productivity

Another fascinating session was Gartner VP Analyst Tigran Egiazarov’s presentation, “Why Your Developers Are Productive With AI But Your Teams Are Not.” This gets to the heart of one of the biggest contradictions in the current AI discussion.

An individual developer can become dramatically more productive with an AI coding agent. They can generate code faster, understand unfamiliar codebases more quickly, create tests, write documentation, and automate repetitive work. However, that does not automatically mean the engineering organization becomes proportionally more productive, because software development remains fundamentally a team activity.

If developers generate code three times faster but requirements remain unclear, testing becomes a bottleneck, architecture fragments, documentation falls behind, security reviews cannot keep pace, or integration becomes more difficult, then the organization has simply moved the constraint elsewhere. There is also a risk that enormous amounts of independently generated AI code increase technical debt rather than reduce it.

The lesson I took away is that organizations need to stop thinking purely in terms of developer productivity and start thinking about the productivity of the software delivery system as a whole. Optimizing one part of the lifecycle is useful, but it does not necessarily improve the throughput, quality, or predictability of the overall system.

From Individual Agents to the Software Factory

That leads into something we have been discussing extensively with customers at Inflectra. The future is not simply one developer paired with one AI coding assistant. The more interesting model is an AI-enabled software factory, in which humans and specialized agents collaborate across the complete software lifecycle.

One agent might help refine requirements, another may generate architecture, another may write code, while others create tests, review changes, analyze security, update documentation, or prepare deployments. However, those agents cannot operate effectively as isolated individuals because they all need access to shared context. If the requirements agent understands one version of the business requirement while the coding agent has another and the testing agent has no visibility into either, we have simply recreated the same silos that organizations have spent decades trying to eliminate.

A shared system of context and record therefore becomes increasingly important as more of the SDLC becomes agentic. To me, this was one of the strongest themes running through the entire conference: the real opportunity is not simply deploying more agents, but creating an environment in which those agents can operate against trusted, consistent, and connected lifecycle information.

My Five Biggest Takeaways

After two days of sessions, discussions, and conversations with analysts and attendees, I came away with five broad conclusions. First, AI is clearly moving from experimentation into engineering. The question is increasingly not whether organizations will use AI, but how they will build reliable architectures, processes, and governance around it.

Second, context is becoming more valuable than the model itself. Models will continue to improve and commoditize, while the proprietary requirements, decisions, processes, test results, risks, and business knowledge surrounding those models will increasingly differentiate organizations.

Third, testing has to evolve for non-deterministic systems. Traditional deterministic testing remains essential, but semantic evaluation, probabilistic testing, AI assurance, and production monitoring will become increasingly important as organizations deploy systems whose outputs and actions can vary from one execution to another.

Fourth, governance needs to enable AI rather than simply restrict it. Organizations need practical technical guardrails that allow responsible adoption while avoiding the much greater risks associated with uncontrolled shadow AI.

Finally, individual AI productivity is not enough. Organizations need to think about the complete software engineering system — the people, agents, context, processes, testing, architecture, and governance that allow entire teams to become more effective rather than merely accelerating isolated activities.

The People Still Matter

One final takeaway had very little to do with AI. One of the reasons I continue to value in-person events is the people you unexpectedly meet along the way. While in London, I had the pleasure of running into Jide Modele, who was previously an important Inflectra partner. It was wonderful to reconnect and hear that he is doing well. I also had several impromptu conversations with Shiva Varma with whom I realized we had several common friends in the Oxford-Washington, DC tech community.

That kind of interaction is difficult to replicate virtually. Conferences are valuable for the presentations and research, but they are equally valuable because they create opportunities to compare notes with analysts, customers, partners, peers, and old friends.

I left London with plenty to think about.

The transition to agentic software development is happening quickly, but the biggest lesson for me from Gartner’s Application Innovation & Business Solutions Summit was that success will not come simply from deploying more AI. It will come from combining AI with trusted context, sound engineering, effective governance, continuous assurance, and human ingenuity. That is where I believe the next phase of the software industry is going to get very interesting.