For a few years the AI conversation was about who had the biggest model. That is the wrong altitude now. Models still matter, the way CPUs still matter — as a layer you buy, swap, and budget for. The margin is moving up the stack: into products, workflows, evaluation, and the data those products sit on.
This post is for people who ship software, buy inference, or have to explain to a board why "we use GPT/Claude/Grok" is not a strategy. It is not a market-sizing deck.
The stack, in one picture
At Davos 2026, NVIDIA's Jensen Huang described AI as a five-layer cake: energy, chips, computing infrastructure, models, and applications. Each layer has to be built and paid for. The application layer is the only one end users touch, and it is the reason the four layers underneath exist.
Two things follow.
First, the bottom of the cake is capital-intensive and crowded. Hyperscalers are pouring unprecedented capex into data centers, accelerators, and networking. That spend is real. It is also not where most product companies will differentiate.
Second, inference is now the production workload. Training still happens, but the day-to-day cost of AI is tokens out the door. Industry commentary through 2026 has inference taking a majority of AI compute, with agentic and reasoning workloads as the fastest-growing slice. If you run a product, you are in the inference business whether you meant to be or not.
Why models commoditize
A model is a capability. Capabilities leak.
Open weights, falling inference prices, and "good enough" alternatives mean the gap between the frontier API and the second-best option keeps shrinking for a large class of tasks. Routing, distillation, and small specialists eat the middle. The brand of the model still matters for a few flagship surfaces. It does not matter for most internal workflows.
That is the same pattern as cloud VMs, then containers, then managed databases. The undifferentiated layer gets cheaper and more interchangeable. Buyers stop paying a premium for "we have compute" and start paying for "this job is done."
What does not commoditize as fast:
- Proprietary data you can legally use in the loop
- Evaluation that matches the actual job (not a public leaderboard)
- Workflows that already live in the customer's day
- Distribution: the place the user already is
- Trust, audit, and on-call when the model is wrong
Those sit above the model.
Where the margin actually is
If the model is infrastructure, the product is the control plane.
Think of inference the way you think of a database. You do not advertise "we use Postgres." You advertise the workflow: the ticket that closes, the draft that ships, the claim that is coded, the incident that is triaged. The database is a line item. The workflow is the company.
Three practical consequences:
1. Switching cost lives in integration, not in the model card. Prompt libraries, tool schemas, eval sets, and human review queues are the lock-in. If those are thin, a competitor can swap your model next quarter.
2. Unit economics are tokens plus people. An agent that spends $0.04 of inference and $4 of human cleanup is not an agent product. It is a demo. Measure cost per completed job, not cost per million tokens.
3. Routing is a product decision. Different jobs want different models: cheap/fast for classification, stronger/slower for irreversible actions, local for data that cannot leave. The routing policy is yours. The vendors will all claim to be the only layer you need.
What to do this quarter
If you run a product or an internal platform:
- Treat the model API as a vendor, with a backup. Write a one-page "we can switch in 30 days" test and actually run it on one workflow.
- Put evals next to the feature, not in a slide. A frozen set of real tickets/emails/PRs beats a public benchmark.
- Own the workflow artifact: the ticket, the document, the PR, the claim. That is the up-stack asset.
- Budget inference as COGS, not as R&D theater. If you cannot say cost per successful task, you cannot say whether the feature should exist.
If you buy AI for a team:
- Ask "which job gets shorter?" not "which model is smartest?"
- Prefer tools that sit in the existing system of record. A new chat window is the down-stack move.
If you invest or advise:
- The crowded trade is GPUs and frontier brands. The quieter trade is the control plane: eval, routing, permissions, and industry workflow.
- Be suspicious of stories that stop at "we have access to a model." That is table stakes in 2026.
What not to do
Do not freeze the architecture on one vendor's chat API. Do not skip evals because the demo was impressive. Do not confuse "employees have Copilot seats" with "we captured workflow margin." Do not wait for the model layer to stabilize before you own the job — the model layer is supposed to keep moving. That is what infrastructure does.
A simple map
| Layer | What you buy | Where the margin is |
|---|---|---|
| Energy / chips / clusters | Capex, cloud commit | Hyperscalers and hardware |
| Models | API or weights | Labs, briefly; then price |
| Inference serving | Tokens, latency, region | Utilities, unless you own routing |
| Apps and workflows | Completed jobs | You, if you own the loop |
The punchline is not that models are worthless. It is that they are becoming plumbing. Plumbing has to be reliable, billed, and replaceable. The company that wins is the one whose product still works when the pipe is swapped.
This post is part of our operator notes on shipping AI. Follow the series at amtocbot.com.
No comments:
Post a Comment