AI Agents—What Lies Beneath

Scott King

Hey Chris and John, there’s a recent article in TechCrunch that says OpenAI reportedly plans to charge up to $20,000 a month for a specialized AI agent. I can’t read the source article because they wanted $700 for access. It mentions a high-income knowledge worker agent could be priced around $2,000. I don’t know how many tasks that covers or if it’s just a Q&A tool, but a PhD-level research agent for $20,000 a month seems excessive. Especially when yesterday, someone asked ChatGPT how many Rs are in “strawberry,” and it said two because they appear together.

I can’t imagine spending $20,000 a month, which is nearly a quarter-million dollars annually, for something like that. What are your thoughts on specialized agents from OpenAI?

Chris Kraus

It’s interesting. I have sticker shock, to put it mildly.

I like that they’ve identified people want agents with specialized knowledge beyond just answering questions. The question is, what will it actually do? Is this a business model to satisfy investors, or do they genuinely believe this is the right move? We’ve found that people want agents to accomplish outcomes. No one we’ve talked to just wants a general agent to answer internet queries. It has to work with their data, processes, and context.

John, do you think this is just a play for funds, or are they trying to figure out profitability compared to other LLMs?

John Michelsen

I don’t know. There’s a lot about OpenAI’s strategy over the last several months that I don’t understand. I don’t even understand who, other than the roadshow guy, Sam, is still there. There’s been so much turnover in the leadership, and Sam spends a lot of time in front of audiences with an OpenAI picture in the background, but not at OpenAI. So I really don’t know who’s doing what there.

They’re not going to get far unless these agents are onboarded into the organizations hiring them. If you’re trying to establish the value of a high-paid employee and say, “I’m OpenAI, and I’m accomplishing the work of two, three, four of those employees, so I should at least cost the amount of one,” there’s the argument, right? That might have been said somewhere inside OpenAI’s leadership team.

Now, okay, let’s set a price. Is it 240? That’s the going rate for a strong engineer. We can justify that if we claim we’re three times as good, but we’re not. Right now, we’re a frustrating, sometimes helpful, sometimes not helpful coding buddy to an existing engineer. We’re certainly not the value of three engineers—just an assistant to one. So I’m with you, Chris, on the sticker shock.

Onboarding is another issue. If you think this artificial person replacing three employees will be capable of doing that work, you’d better show them how. Give them a login, a desk—at least notionally. They need accounts in your system, guidance on processes, and a clear understanding of how your company operates beyond the internet training data the model has been given.

A generalized agent won’t be sufficient for a company to get value from it. Even if it gets thoroughly onboarded, as soon as you specialize it without an orchestration capability, people end up making it valuable. People become your integration strategy.

They’ll have to figure out how to interface with that agent just to accomplish basic tasks. But if you integrate that agent with others and orchestrate them from a machine-run orchestration platform, you could get superhuman performance.

I struggle with this approach. It makes sense for OpenAI to pursue it—apparently, the vast majority of their revenue comes from paid chat sessions, while Anthropic gets most of theirs from API-based use. A lot of chatbot and coding assistant builders rely on API-based Anthropic models, like Claude 3.5 and now 3.7.

Meanwhile, when business or non-technical people talk about GPT, they mean OpenAI. I see how OpenAI is doubling down on its existing strength, but I don’t see how it plays out. They’re confusing less technical users with this O series—skipping numbers, introducing a mini and a not-mini version, then a new GPT model.

There’s no clear way to understand their lineup. It’s not like they have distinct branding—one as a sedan, one as a sports car, one as a truck. There’s no model for understanding the models. You almost need to ask their own model, “Can you explain GPT 4, 4.0, then 0.1.0.3, then 4.5? Now there’s GPT 4.5—can you help me make sense of this?”

I’m not trying to cast stones. OpenAI has been a fundamental driver of positive change. But I wonder what they’re thinking with some of these decisions. As an outsider, they seem a little in disarray.

Scott King

Maybe all the model numbers are just a bunch of skunkworks projects. The teams working on models in the office have the O models, and the remote teams have the numbered models. I have no idea.

But it’s interesting because there’s demand for specialized agents, especially from less technical users—businesspeople.

They’re asking questions, but they’re not always relevant to the business. There’s demand, but then they think, “I want to buy one of these—how do I do it?” And if OpenAI’s pricing is too high, it becomes inaccessible. Everyone would need an agent for everything, right?

At $2,000, it probably just assists a person, and as you said, people are still the integration strategy. You’d end up with so many of these agents, and ideally, they work together, or you build your own. That’s when companies start saying, “Let’s create a budget and build one.”

I want to talk about the surface level of all this. When I saw the iceberg analogy, I had to share it because, even though it’s cliché, it works. The Q&A agents that people interact with seem simple—you ask a question, and it responds. But can you trust the answer?

Building one is a different challenge. Rakesh shared an image on Twitter, or X, that lays it out. There are some good ideas, though I don’t completely agree with the order. Chris, what’s your take on what’s below the waterline?

Chris Kraus

Scott, I agree. People will face sticker shock with the build-versus-buy decision. The first thought is, “Can I just build this?” What I like about this iceberg analogy is that the small part above water is what you see, but everything below is what actually makes it work.

I have some concerns. Orchestration should be higher, and the way these tools are grouped feels a bit off. It suggests a focus on single-purpose, task-oriented agents. What interests me more is what’s missing.

Looking at this, I see ETL—Extract, Translate, Load—but that’s a batch process. What about real-time integration? If I wanted an agent to work with customer support, the first thing it would need to do is look up the customer’s details to get context.

Scott King

That’s a good point. There’s no Oracle, Salesforce, HubSpot, or Zoho. No ESB. Is there an API layer at least?

John Michelsen

There’s no integration bus.

Chris Kraus

ESB AI is missing, and we consider that part of orchestration. They’re looking at orchestration too narrowly—just agent orchestration and LLMs. We take a broader view. Orchestration is about understanding the business problem you want to solve.

Sometimes it’s customer support, where you need to look up tickets. Other times it’s accounts receivable or payable. It could be general questions about warranties and manuals or internal company processes like HR. The key is recognizing that you’ll have multiple agents, each needing specific connections to systems.

We define that as orchestration because understanding who you are is essential to getting the right answer the first time. Too often, when you ask an agent a question, it just responds with another question. Instead of getting an answer, you’re stuck in a loop.

How do we enable the right orchestration and ensure real-time system connections? Context changes constantly, so we have to design an experience that adapts. And an LLM isn’t the only AI available—people forget that.

You have categorizers. You have regressors that predict numbers and help automate decisions, eliminating the need to ask a manager, “Can I offer a 10% discount to keep this customer from churning?” There’s also natural language processing to determine whether a user is asking a general question or triggering an orchestration.

For example, if a customer says they want to change their plan or cancel a service, that’s not just a yes-or-no FAQ response saying, “You can cancel 30 days prior.” That’s an orchestration trigger to engage the customer, understand why they’re leaving, and attempt to retain them.

There are many types of AI. Categorizers are especially powerful because they continuously improve, becoming more accurate over time.

Chris Kraus

Yes, you can use an LLM for categorization, but fine-tuning it for precision—especially for subtle tasks—is difficult.

John Michelsen

Some language-based tasks can be classified with LLMs, but even that has limitations. To be fair to Rakesh, it’s always easy to critique someone else’s work, and he can’t fit everything into one diagram. We’re not taking shots at him—just analyzing what’s missing.

Scott King

I’m glad he put it together.

John Michelsen

Exactly. The real point is that it’s all this and even more. The big miss, from my perspective, is where are people? Where are people? I continue to be baffled that we focus on building sophisticated, outcome-oriented solutions but only think about interfacing with a single person.

We have no concept of workflow management for the people involved in getting a job done. Even the idea of interactions lasting longer than a few minutes is missing, when real outcomes can take months. It’s frustrating because this isn’t the first time our industry has done this. The industry is almost diseased with this forgetfulness—people are involved in outcomes, and they need to be equal participants in how a platform is built.

That’s not a knock on Rakesh. He’s making an important point—there are a lot of capabilities that need to be orchestrated to create a real agent. And an agent, in the end, is just an AI-driven way to perform a task. Once you have many agents, you need to orchestrate them, which is why we keep using that word.

So, looking at this, if I were evaluating a $20,000-a-month solution and didn’t even know how to make it productive, building it myself would be at least 10 times more expensive. Just in terms of the engineering skills, time, and resources needed, it would take a huge effort just to get something going.

That’s why we work in this space. We believe we have a great alternative, and we hope you take a look at it.

Chris Kraus

Yeah, I think that’s a good takeaway. Off-the-shelf solutions may not work because they don’t account for your company’s specific policies. And for three grand a month, are you really retraining a neural network to be specific to your business?

There’s no way to do that without hallucinations because there’s too much foundational knowledge in these models—just learning how to construct sentences and understand language introduces external context. You’ll never completely isolate it to just your business.

So, there are a lot of opportunities here.

What I like is that he’s brought attention to the fact that you could do it, but there’s a lot more to this. And John, as you said, orchestration involves people.

Scott, we’ve talked about this—you’ve used agents on phone and internet services, and they can’t actually answer the questions you need because they’re just call trees repurposed into chatbots. We need to be clear—we’re talking about much more advanced natural language capabilities to handle these interactions.

Can it actually solve a problem? Can it integrate with systems? And when it doesn’t know what to do, can it escalate to a person? Maybe the person joins the chat to help, or they’re a manager approving something in the background without you waiting on hold. Better yet, if it’s a repeatable approval, can we train an AI model to categorize yes or no and have that happen automatically?

There are interesting possibilities here, but it’s important to recognize that multiple types of AI will make this work. Once you build one, you’ll want 20 more because it’s so valuable. It’s not just a task agent helping with a single thing like a document or a spreadsheet. Once you see what it can do, you’ll want to scale it across multiple functions.

Scott King

Yeah, you’ll want a lot. But, as John said, constructing the first one is 10 times the cost. You probably don’t have the skills, infrastructure, or budget set aside. You might not even realize, oh, this is going to cost a million dollars to build—for just one.

And then, if you code it to a certain task or a specific process, that’s it. You think, we should build an agent for this, but if you’re not using a platform approach, you’re going to do it all over again. A platform approach already includes these features—security, orchestration, integration, and everything else.

John Michelsen

That’s right.

Scott King

It’d be like buying a car that only goes to the grocery store, another car that only does carpool, and another just for going to dinner. It sounds ridiculous, but people won’t believe it until they try it.

John Michelsen

That is what happens.

Scott King

Just like what happened years ago when we were talking about conversational automation—everyone thought we were full of it. Then ChatGPT came out, and suddenly, people said, My God, this is what these guys have been talking about forever.

John Michelsen

Sorry, Chris—but just yesterday, I had a great call with a new partner we’re onboarding, and he basically told the same story you just did, Scott.

He said, We’ve been delivering AI-based custom apps for people, and we’ve gotten pretty good at it, but it’s an enormous amount of labor. They’re a perfect example of the reality that open source is free—free of license costs—but it comes with significant human labor costs, which, by the way, are often much higher than software license costs.

But that reality doesn’t always register. They’ve completed several successful projects, but now they see that there’s no faster or easier way to move forward in their existing accounts. With a Krista deployment, the next solution is naturally better, faster, and cheaper to build—and then the one after that, and the next. It increases their momentum and ability to get more done.

With their old approach, they built everything manually—LangChain, Llama, Python, custom web components—integrated it all, onboarded bespoke content, and then, for the next project, they had to start from scratch. Every project is essentially File → New Project, repeating the entire process, which leads to nothing but more point solutions and siloed tools.

If there were no better way, and the return justified the effort, you’d accept it. But there is a better way.

Earlier that same day, I had another conversation with someone saying the same thing back to us:

“We’ve already got 30 AI projects we want to take on, but I don’t want to hire 30 vendors. I don’t want 30 different vendor plans. I can only run two or three of these at a time at most. I can’t imagine picking tools from this iceberg diagram 30 times over.”

Underneath it all, you realize that picking any single logo from the group means you may need to jump to another one—or even one that doesn’t exist yet. It’s too early in the market to commit to a single tool.

“All right, I’m going with X—wait, X is a company now. I can’t say X. I’m going with N. Is N a company? Do we know N? We need to look up N.”

Chris Kraus

Yeah, A-prime.

Scott King

I think Notion actually uses an N and a third.

John Michelsen

I bet every name’s taken. You can’t say anything without someone saying, That’s us. That’s copyrighted.

Yeah. All right. Back to my point—not my digression. We don’t know what we don’t know. What we need is to build in resiliency and adaptability.

Scott King

There’s also one that’s just the alphabet, so that’s every letter, right?

John Michelsen

Probably the most important attribute of an AI strategy isn’t, Hey, this thing can do X, let’s stick it in front of people and suddenly make or save a ton of money.

Because, frankly, none of those projects are guaranteed to make you better. It’s funny—we don’t think about the fact that optimizing a single step in a process might just result in extra inventory. It doesn’t necessarily make the overall outcome faster.

This has been proven in different ways—take traffic as an example. You optimize one segment to move faster, and it creates a massive slowdown later. People actually get home later than they would have if you had slowed that section down instead. There are plenty of studies showing that systems thinking needs to be applied.

To avoid a ton of unnecessary detail—systems thinking has to be applied here. That doesn’t align with the idea of, Let’s just buy a $20,000 agent and bring that in. Sure, an AI agent could do something faster than a human, but a person still has to instruct it, evaluate its responses, and move the result to the next step.

And none of that sounds particularly fast because that person is already busy. In reality, that agent is going to be sitting idle for 16 to 18 hours a day with nothing to do.

So how do we make these things truly valuable? The answer is that we let the machines do the work. We make them do more of the work. That’s why we always use the analogy: You don’t invoke Krista. Krista invokes you. Krista owns the outcome. When Krista needs your help, she’ll reach out to you in a conversational way.

You don’t go to a dashboard and check, Is there a screen I need to fill out? No. You’ll get a message saying exactly what Krista needs. That message might say, I don’t understand this—can you help? or I need you to do something in the real world because I don’t have hands and feet yet.

Chris Kraus

Yes.

John Michelsen

If you think about it the other way—someone builds an agent and says, Didn’t I do something valuable for you? Aren’t your outcomes optimized?

Well, not necessarily. If it takes too long to activate the agent, and if its output—even if faster than a human—doesn’t get consumed at the moment it’s produced, then it doesn’t actually speed anything up.

We have to think in terms of systems.

Scott King

I agree. That’s an incredible closing thought. Chris, anything you want to add?

Chris Kraus

Nope—just, Hey, check out Krista’s platform. We’ve spent the last four years thinking about this.

Scott King

Yeah. If you don’t want to spend 20 grand, call us. If you’re going to spend 2 million, then go build the iceberg.

All right—thanks, guys. Until next time.