← Back to Insights

Why Our AI Pilot Stalled Before Production

Most AI pilots never reach production, and the model is rarely why. A post-mortem on our own stalled platform, read against a Stanford AI maturity model.

Almost every conversation we have with a new client starts the same way. Someone says, "We're already doing AI." Then we ask what that means, and the answer is usually some version of: a few people on the team use ChatGPT, marketing tried a tool for a while, and there's a pilot somewhere in operations that nobody has checked on in two months.

I am in no position to be smug about that. Our own AI platform spent most of this year stuck in roughly the same place. What finally moved it had nothing to do with the models, and that turns out to be the whole point.

The four stages of AI maturity

I picked up a useful frame for this from Stanford's AI-Driven Leadership program, where Professor Melissa Valentine and her team lay out a four-stage maturity model for generative AI use cases. What makes it worth borrowing is that each stage describes what your organization has to be able to do, not what the technology can do. The framework below is theirs. Everything after it is ours.

The short version:

  • Stage 1: Ad hoc. Individuals tinkering. No strategy, no shared infrastructure, no governance. Evaluation is "that looked good to me."
  • Stage 2: Projects. A team builds something real and useful. Usually one person, usually one department, usually no orchestration behind it.
  • Stage 3: Programmatic. AI is in multiple workflows, somebody owns it, and performance is measured rather than asserted.
  • Stage 4: Enterprise ecosystems. Networks of agents coordinating with each other and with humans, under real governance and continuous evaluation.

Cutting across all four is a second dimension: whether a use case is agentic or non-agentic. Both can be transformative. They fail differently, and they need different governance.

That is the map. Here is what it looked like from inside a company actually trying to climb it.

Stage 2 is where we were, and the wall was not the model

Our platform, sego.pm, does something that sounds like it should be a Stage 3 capability. A client describes what they need in plain English. The platform shapes that into structured development tasks. Those tasks get pulled down to local coding agents running Claude Code over MCP, which do the actual work, and a developer refines from there. Prose in one end, working software out the other.

The shaping worked. The models were fine. The platform still behaved like a Stage 2 pilot, and here is the honest reason why.

Task shaping ran in an external monitor process, bridged to the web app over Ably. That bridge dropped. Not catastrophically, not every time. Just often enough. The visible symptom was a client sitting in a chat window typing "are you done yet?" and getting nothing back. The AI had done its job. The message never made it home.

That is what Stage 2 actually feels like when you are living in it. Not a model that hallucinates. A demo that works beautifully for the person who built it and degrades the moment it meets a real user on a real connection with real patience limits. My own assessment of the platform through this stretch was "going OK, not great, but it has potential," which is a generous way of saying it had not earned anyone's trust yet.

The fix removed infrastructure rather than adding it. We killed the Ably bridge and moved shaping into the web app, streaming over SSE from the same request. Faster, smoother, and one fewer moving part between the client and the answer. We retired the monitor process too. It was barely used, because execution runs through MCP and local agents rather than an autonomous server-side loop.

That is the least glamorous possible answer to "how do we advance our AI maturity," and it was the correct one.

The most expensive mistake is buying Stage 4 for Stage 1 data

The same week, we seriously considered indexing every client codebase into a vector store, full embedding-based RAG over the source, the way a few well-funded AI dev tools do it. It is the impressive-sounding option. It would have made a great slide.

We rejected it, and the reasoning has held up: a good map is enough. Turning a client's request into well-scoped tasks does not require live grep and embeddings across an entire repository. It requires knowing the shape of the project. Instead we are building a per-project knowledge graph covering requirements, decisions, architecture and conventions. Shaping retrieves from it and writes back to it. It exists primarily to make task shaping better. Living documentation is the secondary benefit, not the pitch.

This is the trap I see most often in client conversations, and it generalizes well past our situation. Buying an agentic platform does not skip you past the stages. It means you now have a Stage 4 tool sitting on Stage 1 data. The tool is not the constraint. It never was.

Stage 3 is the destination, not a waypoint

Here is where I would diverge from how maturity models usually get read. Most of them are drawn as a staircase, with the implication that you should be climbing.

For most mid-market companies, Stage 3 is the destination. A company with three or four integrated, measured, governed AI workflows will out-earn a company chasing an agent ecosystem it has no data foundation to support. Getting to Stage 3 in your two highest-volume processes is a better use of the next eighteen months than a roadmap to Stage 4.

The widely cited MIT NANDA report found that roughly 95% of generative AI pilots produced no measurable impact on the P&L. The methodology is narrower than the headline suggests and it has been argued over fairly. But the direction matches what we see: pilots that never survive contact with a second team, a second data source, or the person who built it leaving. Johnson & Johnson hit a version of this at scale. They accumulated around 900 generative AI use cases, found that roughly 15% of them produced 80% of the value, and responded by concentrating on the high-value cases under central governance rather than by doing less AI.

The lesson is not to lock experimentation down. It is that a pilot without an owner and a metric is a demo.

Where the agents actually belong

One deliberate choice in our architecture is worth pulling out, because it connects to something I wrote about earlier this year: deciding where deterministic and non-deterministic processes each belong.

Task shaping is non-deterministic. A person writes prose, the model interprets it, a human reviews the result. That is the right job for a model. Task execution is not autonomous. It runs through local coding agents under a developer's supervision, with the work landing in a real repository and a real review. Publishing is fully deterministic: an edit goes to a preview environment, a human looks at it, and one click promotes it to production.

We did not arrive at that split because it was philosophically tidy. We arrived at it because the alternative failed in front of clients.

Meanwhile, the non-agentic value is enormous and badly underexploited. A sales team whose call summaries are generated automatically and land in the CRM correctly, every time, is a Stage 3 capability with no agent anywhere in it. Most of the ROI available to a mid-market company in the next two years is sitting in non-agentic use cases executed well.

Where are you, actually?

Three questions will place you more honestly than any assessment framework:

  1. If the person who built your best AI workflow left tomorrow, would it survive? If no, you are at Stage 2 regardless of how good it is.
  2. Can you state a number for how well it works? Not "people like it." A review rate, an error rate, hours saved. If not, you are not at Stage 3 yet.
  3. Do you know where your company's data goes when your team uses AI? If not, start there before anything else.

We have been building custom software for small and medium businesses for over twenty years: ecommerce and BI platforms for a spirits portfolio, a product filter engine for a 70-year-old government contractor, scheduling systems replacing legacy software. The AI work is newer. What has not changed is that the hard part was never the technology. It is whether your data is clean enough, your processes are defined enough, and your governance is real enough to support the level of ambition you are claiming.

Our platform got unstuck by deleting a message bus. Yours will probably be something equally unglamorous. That is the actual work.

Working through where your organization actually sits? Get in touch. We help companies find the two or three AI use cases that are worth doing, and then actually get them into production.


Sources and credit

  • The four-stage maturity model is from Professor Melissa Valentine and her research team at Stanford, taught in the AI-Driven Leadership program. The framework is theirs; the interpretation, the platform history, and the opinions are ours.
  • MIT NANDA, The State of AI in Business 2025. Coverage via Forbes.
  • Johnson & Johnson's AI strategy pivot, via The Batch, DeepLearning.AI.

Ready to Put Ideas Into Practice?

Let's talk about how custom software can transform the way your business operates.

Start a Conversation