The AI Echo Chamber: Why Vendors Are Fluent in a Language Most Buyers Don’t Speak

Scott King: I wanted to talk about the breadth of everyone’s knowledge. On one end you have researchers testing the latest models, figuring out how to use them. On the other end you have people asking what AI terms mean. Where does everyone fit in the middle? Then I wanted to get to the enterprise brain — and the question about whether the brain would prefer a competitor. That’s how I thought we’d frame it, but I’ll listen to your story.

John Michelsen: You can weave this in — it would naturally follow. Preconceived notions about how AI should work is one of those issues. Just like how there’s a wide variety of understanding around AI, there’s a vast disconnect in grounding people’s understanding of what generative AI is really capable of doing and how it actually works.

The piece I can add to that story: an actual customer bought with the expectation that, because they had given Krista all their content, Krista would learn what the content didn’t have. They asked: “Why is your software continuously bothering our people to answer questions?” I told them: Krista needs to learn from your people. If the content is ambiguous or missing, she has to go somewhere to get it. She’s been configured to ask them. The customer said: “Krista is supposed to learn. If she was going to ask me anyway, why would I even have this?” I said: “Where do you want her to get the answer if not from the content you gave or the people you have on staff?” They said they didn’t know, but they were told she was going to learn. I said: “You learned how? Either by reading or by being told. Krista works the same way. You either give Krista the information or you tell Krista the information. There’s no mystical ‘I’ll just make it up for you.’ You can’t have a situation where someone asks what the policy is on something, Krista doesn’t have that information, and she makes up the policy.” They said: “I didn’t want her to make it up. I wanted her to know it and say it. But how would Krista know this?”

We went around like this for thirty minutes. This was a real customer call. What it comes down to: certain words mean one thing to us and a completely different thing to a lay person. When we say “learn,” we mean providing vast amounts of content or literal training material. If you say that to a lay person, they think: here comes the warm, glowing, fuzzy AI that will just learn. When we said during the sale “Krista will learn from your people,” they didn’t think that meant “make your people answer questions.” By the way, this wasn’t a belligerent call. This was not an incompetent person in most walks of life. This is just someone who had no idea they would need to be part of their own success, and that the holes and misunderstandings related to knowledge management would need to be resolved somehow — not through Ouija boards or whatever else we were supposed to use to come up with answers to policy questions.

Scott King: That’s interesting, because when people started using ChatGPT, all of a sudden it just knew stuff. So they probably weren’t aware of the learning aspect and assumed this thing already knows everything. Maybe they assumed that if it learns, it goes off and learns by itself. And that’s not what you want — you want it to know your business, your business rules, your context. How did that turn around? Did you explain learning from a data science perspective or from a school perspective?

John Michelsen: I stuck with school. As soon as we go data science on them, we lose them completely. And frankly they have the right to say: “That’s your game, not mine. Your game is to make this work, please.” What we’ve had to tell plenty of customers is that they are significantly contributing to the success of their own project. If not, it’s quite possible they could land in the ninety percent of projects that fail. We have a great track record, but it requires customer involvement. That involvement is almost the single biggest indicator of project success — and it is because of things exactly like this.

Getting practical: when we help them understand they are repairing holes and ambiguities in their content once and for all, that lands. In knowledge management we don’t need thirty or sixty samples of labeled data to get confidence up. One definitive answer — “the policy is X” — is enough. We’ve been telling customers for years that we deploy at a lower level of accuracy that we’ll reach fairly quickly, and they will be highly involved in closing that accuracy divide. Whether it’s machine learning based or knowledge management, both have the same kind of rapid growth from sixty, seventy, eighty percent accuracy up to eighty-five, ninety, ninety-five percent or better. But that is very much a customer burden. It’s not one we can take away from them. We’re filling in all the holes and ironing out all the wrinkles in their knowledge base. If they don’t go through that process, they don’t come out the other end with the value AI can really deliver.

These customers were a couple of weeks in — probably as bad as it was ever going to be — and I totally understood. “You were supposed to deflect all of these very important, very timely questions. We were supposed to be running this at machine speed. Isn’t that what you guys say?” Yes. And some of them already are. But the ones not yet well represented in the knowledge management system have to go to people, or you need another escalation path. Maybe there’s another repository of information you can make available.

That’s where we say: give it another couple of weeks. If you follow the path most customers do, the large initial volume of unknowns trends down fairly quickly. Within a few weeks you realize it’s much lower. Then after another few weeks, you don’t remember getting an interruption from Krista for days. At that point you know you’re in pretty good shape.

There’s never one hundred percent. The reason is most businesses are too complex for there to ever be one point in time where everything gets known. And as soon as that were the case, the world would change and content would no longer be accurate. We’ve got to keep that customer involved.

This is a good example of people thinking they understood. The phrases our sales team used were correct, but the understanding was completely different. When they said “Krista will learn from your people,” that didn’t translate. The customer thought: “No, that would be normal. This is AI. It’ll go the AI way.” But what would that actually mean? That’s where the misunderstanding lives. I’m not trying to offend anyone. The opposite. Those of us who live on a podcast like this totally understand and spend enough time with it to recognize there’s no magic. An LLM is making a prediction about the next syllable. A token is essentially a syllable. We’ve built massive capabilities with it, but what it’s doing is a very practical, predictable, mathematical thing. It didn’t come up with a new science. The questions we get, when you put them against the science, are understandable. Maybe that’s where we have to go.

Scott King: The aha moment from an expert’s perspective is probably just as big as the confusion on the other end. You have to meet in the middle somewhere. I’ve explained this to someone and when they tried to explain it back to me, I had no idea where they were. “Okay, let me back up a little bit.”

It’s really hard, especially when AI pundits are talking about which model creates better images. Most of the market is nowhere near that. They’re closer to what you experienced — you were in a meeting with executives and they stopped you to ask what an AI term meant. What was that story?

John Michelsen: There are a hundred of those. Someone will ask what I mean by “prediction,” or “generation,” or what LLM stands for. My favorite — and I don’t mean to offend anyone, I’m doing the opposite, I’m really sympathetic here: a super intelligent person with a very successful career, someone who knows things I have no clue about and whose career complexity would spin my head around, stopped about halfway through a meeting and said: “Look, I know this will sound silly, but you have other customers and Krista is already doing good things for them. When we come along, how do we know Krista won’t prefer those existing customers and do a better job for them than for us?”

At that moment it really dawned on me: we perpetuate this. Everyone in this business puts very human-sounding voices on their AI. We are absolutely crushing the notion of humanizing AI. And he went right along with it. He was thinking there would be personal bias — not even through training data per se, but through experience — from a singular Krista working for many companies simultaneously. I don’t know if I kept my composure perfectly, but I appreciate that he felt comfortable asking. How do you get on your toes and help someone understand they’re not even in the same galaxy as what’s actually happening?

Where we landed was pretty good. I said: think of Krista as a robot that gets cloned. We clone her for each customer. We call them tenants — in a SaaS application each tenant is its own environment. We get audited to confirm no tenant’s data can see another tenant’s data. Your Krista won’t even know there’s another company that Krista works for, or that she operates in a competitive field with you. There’s no opportunity for preference.

But it goes a little further. Krista’s only input is what you give. If you actually told Krista to prefer your competitor — let’s say we’re talking about pricing — you’d get poor pricing advice. You’d never do that. You’d do the exact opposite. You’d say: “We’re the best in our industry, we deserve a great price,” and have her interrogate your systems for all the places where you can act on that. The notion of cloning — that the capability comes with Krista but you fill her with whatever knowledge is needed — that made it through.

What I didn’t feel good about was walking away from that meeting thinking: we are just blowing right past eighty percent of the population. Most people are barely using an LLM as a surrogate for web search, which is a very rudimentary use case. Meanwhile the people we all hang out with are talking about solving humanity’s last exam. That divide is getting wider faster.

Scott King: Maybe those two guys should get together — the one who thought Krista was learning on a competitor’s business, and the one worried she’d prefer the competition. Hopefully they’re in the same industry, selling airplane parts or whatever.

John Michelsen: We’d make the news in a way I do not want to be associated with if Krista started mouthing off at one of our customers because she preferred the other one. But it would be funny to make a fictitious setup for that just to see what it would be like.

Scott King: I don’t remember the Joaquin Phoenix movie “Her” very well — that was a long time ago — but he was always arguing with her or she was offering great advice. Anyway, it’s interesting to understand the divide in understanding. People not in the industry only know ChatGPT and think it knows everything, when it’s really just a surrogate for web search instead of Google.

Where do you think enterprise buyers are? Let’s take it past the awareness level. They understand public LLMs. Hopefully they understand you shouldn’t put company information in there, since no one reads the terms and conditions anyway. On one end you have companies like Walmart with massive data science teams running for decades — buying patterns, weather, shopping patterns for certain weather. My favorite example is strawberry Pop-Tarts at Walmart: a storm comes, they sell more strawberry Pop-Tarts, and they know this and stock accordingly. But the enterprise buyer — on a zero to one hundred scale, where do you think the average and maybe the median sit?

John Michelsen: I’d love to be generous, but let’s be grounded. Ten or twenty on a scale to one hundred. A lot of it is because, like any relatively new technology — AI’s been around forever, but LLMs have matured at an incredible pace. We’ve rewritten Moore’s curve on generative AI model progression. If you go back to 2020 or 2021, the earliest models are now in their fifth, sixth, seventh generation. That’s an incredible rate of acceleration.

People are listening too much to the sci-fi aspect. They’re filling in the blanks with sci-fi movies. They’re listening to vendors who say things that aren’t well grounded. I won’t name the company, but there are people who get on TV with famous actors and say: “If so-and-so agent were doing that for you, then blah blah blah.” That’s gibberish. We take an under-educated population — even in enterprise procurement teams — and jam overly inflated marketing babble at them, moving so fast that it actually sounds like it could be true. When a prominent voice says their model is smarter than any person who will ever live, what does that tell people? It tells them: it’s all-knowing. I should just warm up to it and hope it’s nice to me. Which is exactly where that executive landed.

Scott King: Right — hopefully I’m nicer to it than the other guy. So you stroke it a little bit.

John Michelsen: Right. And hey, I’ll be candid. Those earliest times when I was prompting a model for the first time — many years ago — it was: if I prompt it this way, it goes down a path I don’t want it going down. And it’s not remembering this now, but we’re going to wire this up to a database at some point and it’s going to know. So I wasn’t sure I wanted to bring that up with an LLM. Then I caught myself: “John, stop. You’re doing exactly what you know you need to be teaching people not to worry about. And if you want to erase memory, you just erase memory.” It’s easy to succumb to this because we’re personalizing it on purpose.

Scott King: We play with it every day. I remember the first time I was using ChatGPT. We had an episode about the nine-hundred-thousand-dollar prompting engineer — I think it was Airbnb or some big California company advertising for someone like that. Prompting was such a big deal back then, stuffing it full of context. I remember someone watching me write a prompt. I was asking the AI questions and telling it to check back with me because I didn’t want it to drift and hallucinate in a certain area. “Check in with me — am I making sense?” They said: “Why are you doing all of this? This is not how you do it.” I said: “You can do it the other way, but I learned that doesn’t work because it drifts so fast and so far.” They’ve gotten much better. I prompt much differently now than I did three years ago.

And now with the local client and everything it’s become easier. But it still is not all-knowing. I have source documents for the things we create, but clients don’t reach across projects or folders. The all-knowing idea is so far-fetched that I don’t know how people get there. I’m the opposite of the person you’re describing — I don’t even understand how you got there, and you don’t understand why I’m doing what I’m doing. We have to meet in the middle. The business aspect of it is: we have to figure out how this hits the balance sheet. How do I make a process run better that doesn’t need the check-in? It did learn. And now it runs by itself and I’m making dollars instead of pennies because I’ve reduced the bottlenecks. That’s the real goal, but people don’t seem to be thinking that way.

John Michelsen: There doesn’t seem to be enough of that happening, and that might be the issue. We produce a business case for every transaction we do, even when the customer says “this is so obvious.” We’ve decided that obvious is not enough. We want to see it for ourselves. We need to make sure we’re taking a customer down the best path and getting them on their journey in the right direction. Many are either doing a certain amount well and continuing, or they’re seeking a second or third attempt because they’ve been burning cash and not achieving much.

Even we message this whole notion of enterprise understanding and getting Krista to understand the entirety of your business. I hope that doesn’t translate into “Krista is all-knowing.” No. We want Krista to be incredibly knowledgeable about every aspect of your business that’s possible, because that makes her incredibly effective at the things you want her to do. Without that knowledge, you don’t get the right outcomes, and you end up with people in the loop all the time.

There was a great example just a few minutes before we started this podcast. Geeky thing — I apologize if I’m about to lose the audience — but MCP has no type safety on the response of a tool call. When you’re using tools, which is basically how you go agentic with generative AI, you give a model a task and say use tools to go read your files, go into your system, go create that PowerPoint. Those tools have a fairly well-defined input. But most MCP servers do not provide any kind of structured output. That is a terrible mess. It literally makes MCP inappropriate for enterprise-grade, trustworthy, high-throughput use. And don’t think TPS means tokens per second — to a real business person, TPS means transactions per second. When you’re thinking about that kind of throughput, you’ve got to have structured, typed information. If you don’t, you always have to have a human in the loop: Did it get this right? Did it parse this right? Did this response come back the way I needed it to?

Without type safety, there are a hundred little examples of why people who rush into agentic without understanding — without recognizing these are very immature tools — are failing. This reminds me of the old RPA movement. We shifted people off the swivel chair, and now we’ve got one person looking at a really big screen trying to keep it running. All we did was shift it.

Scott King: He’s looking at rows in a spreadsheet to make sure the bot didn’t incorrectly type it in there. You didn’t reduce any work, you just moved it over.

John Michelsen: Shifted it from one job skill to another, not necessarily at a reduction in labor cost. We’re still playing those games out — rushing in and doing things not necessarily right. The data shows it. We’re trying to fix that. Let’s get people grounded. Let’s make sure that when we say a phrase, we’re not implying something from a sci-fi movie — we’re implying something with a practical meaning. Let’s make sure there is a real business case so customers get a win and have house money to spend on the continuation of that journey.

We believe there are going to be a handful of companies in each industry that actually nail the ability to transform with AI. The rest are in genuine jeopardy. Nowhere is that more obvious than in the software business. We’re playing this out ourselves. We’re probably just first — and ironically because we’re the simpler one to tackle. Most businesses work in a much bigger scope of automation. Software companies have a fairly tight scope: the construction, support, and evolution of software. That is totally rewriting the economics of software companies right now, and it is rewriting ours. No business will sidestep that process. You have to embrace it and be good at it.

With stakes that high, it becomes very important to help people. Getting back to why we started this conversation: we’ve got to get people well grounded. When we stay at a superficially high level and babble at them with things that sound too good to be true, they make the leap without understanding, their expectations get completely destroyed, and it seems like the technology failed. I hope we can settle it down. Not slow down — settle down the rhetoric. Stay practical. Capabilities speak. Bold claims don’t.

Scott King: We’ve talked about throwing out a future vision, talking to companies about the enterprise brain, enterprise understanding, having context to help with decisions and processes. We told a brain story in the last podcast. What’s been the feedback when you throw the brain concept out there? Do people understand it? Do they desire it?

John Michelsen: Brain is cutting both ways. I was concerned about this and it’s founded, because it makes things a little more challenging for those trying to get a handle on these concepts. “Where is it? What is it? Can I touch it?” It’s a concept, an approach. It constructs something, but it’s more of a fabric of integrated capabilities and data — not an individual thing you can put your finger on. The abstraction is a challenge.

But when you explain that it must progress — starting with understanding, then reasoning, policy, procedure, compliance, and ultimately outcomes that can be orchestrated based on that understanding and reasoning — that makes sense. It demystifies the process of getting from where most organizations are today to a really elevated level of sophistication. We’ve said since our founding that people are your integration strategy, and that is your biggest fundamental problem. When you think about how to change that, there’s got to be a brain beyond the one in your people. That brain has to have the capacity to understand, reason, and act — and it has to happen in that order, because if you reverse it, you get mayhem.

Scott King: From a physical example, it’s like really thinking before you take each step. People as the integration strategy made sense because there are too many applications in these businesses — they were built to solve really small problems and nothing works across them. Then you put AI on top and those apps just get taller. Technical requirements become more complex and nothing works together. But a fabric like your nervous system makes sense. Hopefully the enterprise brain takes less time to learn than ours does. What is ours — twenty-five years before it stops growing? So we’re probably looking at two and a half months, hopefully, pretty soon.

John Michelsen: This is the type of brain that has the opportunity to genuinely evolve. Ours will take eons, but this one gets reconstructed every eighteen months or less. The capabilities woven together by an enterprise brain are growing at an unbelievable pace. If you’re a business positioned to take advantage of the changing pace of technology — and you now have technology that embraces it for you — these concepts make a ton of sense.

The concept: you can’t buy an enterprise brain. You need to build it. And that build is not a five-year software engineering project. It’s starting with a single use case and spreading thoughtfully. That’s practical. But describing it as a brain makes sense only once someone is ready to embrace the reason we describe it that way. For an outsider still trying to figure out whether Krista likes them more than a competitor, telling them to “build an enterprise brain” is just piling on. We’re talking way over our audience. We’re assuming they understand phrases that they have a completely different frame of reference for.

A great example: I stump people by asking what TPS means. “Tokens per second.” No. To most business people, TPS means transactions per second. If you throw TPS around meaning tokens, you are out of phase with most of the people you’re trying to reach — whether they’re buying software, hiring you as a consultant, or anything else. We’re the minority. We have to carry the frame of reference of the majority in order to help their understanding become part of the majority. We’re not doing that when TV commercials run complete nonsense and celebrities make ridiculous claims about AGI-level capabilities.

Scott King: I looked at the price of the OpenAI models yesterday — I was putting it into a presentation. One model was fifteen dollars per million tokens. That meter would go pretty fast. The next tier was around twenty or thirty bucks. You’d have to have a high limit on your credit card. The faster they go, the more they charge.

John Michelsen: That theme has shown up in a big way. A lot of companies started with: “It’s just a few bucks for a million tokens. A million tokens is like the New Testament. We’ve got this.” Then by the time they realize the visible tokens are a small fraction of total usage, it definitely rings up. There are too many stories of companies exhausting an annual AI budget by April.

The interesting part: these AI companies are hemorrhaging money right now. We are paying a subsidized rate. I’m not saying it’s going to go like the dot-coms — but it is taking the same path. “I don’t need profit, I need eyeballs. I need to get as many people on my model as possible.” You tell that to the people who paid less than retail for products online — even less than gross cost including distribution — and those businesses never had a chance. I know a company I was consulting in 1999 selling cars online at a gross margin of negative seventeen hundred and fifty dollars on average. The more cars they sold, the faster they were going to go out of business.

Scott King: That happened to me with my auto insurance. It went up by almost double and I switched. I was a customer for twenty years. Pretty bad. So, Progressive, that’s you.

John Michelsen: Wow.

Scott King: John, I appreciate it. We talked about the variance of knowledge in the market — beginners, experts, everything in between. I really appreciate your time and love the stories. Thanks a lot.

John Michelsen: Yeah, anytime.