I built a multi-agent prototype with CrewAI because every ranking in 2026 said it was the fastest path from zero to a working multi-agent system. Three days in, I had agents that produced coherent outputs in a demo environment running against Claude. I switched to a local 8B Llama model for cost reasons, and the same crew fell apart in an afternoon. The role-based prompting that CrewAI uses to direct agents produced malformed tool calls at a rate that made the system unusable. The framework was not broken. The benchmark environment it was designed for did not match my production environment.
Every major agent framework comparison published in 2026 benchmarks against frontier-class models. The consensus those comparisons reach is real and not wrong for their test conditions. It does not transfer cleanly to self-hosted hardware.
This post covers what I learned running the same research-and-summarize agent task through LangGraph, CrewAI, and Smolagents on the same local Ollama model (same quantization, same hardware, same task) and what the numbers say about which framework degrades gracefully when the underlying model is not GPT-5-class.
The Three Frameworks in One Paragraph Each
LangGraph (LangChain ecosystem, now at 1.0 GA) organizes agent behavior as a directed graph: nodes are operations (model call, tool call, decision), edges define allowed transitions, and state flows through the graph on each step. The framework checkpoints state at every node, supports replay from any point, and gives you explicit control over what the agent can do next. This explicitness is the feature — you can add a validation node between a model call and a tool call to catch malformed arguments before they execute.
CrewAI organizes agents as a "crew" of role-playing agents, each with a natural-language role description, goal, and backstory. Agents delegate to each other or to tools based on their role descriptions. The framework is fast to set up because the coordination logic is implicit: the framework injects crew-awareness into each agent's system prompt rather than requiring you to define a graph. This is what makes it fast on frontier models and fragile on smaller local ones.
Smolagents (Hugging Face) is intentionally minimal. Agents are code-first: the framework generates Python code to solve tasks rather than JSON tool calls, which means the agent "reasons" by writing a script that calls your tools as library functions. Hugging Face designed this for local model compatibility. The code-generation approach degrades more gracefully than JSON-based tool calling when the model's instruction following is weaker.
The Benchmark Setup
Task: given a company name and a product category, research recent news about that company's activity in that category (using a mock web search tool), retrieve a relevant internal document (using a mock retrieval tool), and produce a one-paragraph summary with a recommendation. Three steps, two tool calls, one synthesis.
Model: Llama 3.1 8B at Q4_K_M quantization via Ollama, running on a machine with one RTX 3090.
I ran each framework configuration 20 times on the same task with different input combinations, measuring:
- Tool-call success rate (tool called with valid arguments, no validation error)
- Retries needed per successful run (how many times a failed call was re-attempted)
- Wall-clock latency per successful run
- Total tokens consumed per successful run
Results
| Framework | Tool-call success rate | Avg retries/run | Avg latency (success) | Avg tokens/run |
|---|---|---|---|---|
| LangGraph | 91% | 0.2 | 48s | 3,100 |
| Smolagents | 84% | 0.4 | 61s | 3,800 |
| CrewAI | 63% | 1.4 | 94s | 5,200 |
CrewAI's failure mode was consistent: the role-based system prompts that make it fast to configure with a frontier model add enough context that an 8B model starts losing track of the task specification when it needs to produce a tool call. We measured most failures as the model producing well-formatted prose that described what it intended to do rather than a structured tool call.
LangGraph's advantage comes from architecture: the graph structure means the framework can inject a validation step between the model's tool call generation and execution, and feed the validation error back into the next node. The local model retries with the error in context and corrects about 85% of initial failures.
Smolagents performs better than expected for a different reason. Code generation degrades more gracefully than JSON generation on weaker models: the model can produce partially correct Python that the framework can detect and re-prompt for, and Python syntax errors are more specific than JSON schema violations. The latency hit comes from the code-generation step being longer than a structured tool-call prompt.
The Failure Mode Nobody Documents
The CrewAI failure mode above is visible in the numbers. There is a subtler failure mode common to all three frameworks that the numbers do not capture: silent semantic errors where the tool call is structurally valid but semantically wrong.
In our benchmark, we measured a number of runs across all three frameworks where the model called the correct tool with valid arguments, but passed the company name as the product category or vice versa. All three frameworks accepted these calls (they were structurally valid), the tools returned results, and the agent produced a plausible-looking summary that was based on the wrong search query.
The detection mechanism for this is not in the framework. It is in your tool implementation: return structured results that the model can compare against the original task specification before synthesizing, and include a verification step in your graph or crew that asks the model to confirm the results are relevant before proceeding.
LangGraph makes this easiest to add (it is another node in the graph). Smolagents supports it via code inspection. CrewAI requires careful design of the crew's roles to make the verification implicit in the delegation chain.
Debugging Across Frameworks
When something goes wrong in a local model agent run, the debugging experience is very different across frameworks.
LangGraph: each node's input and output is logged as a state transition. When an agent call fails, you can inspect exactly what the model received (the full prompt) and what it produced (the raw output before parsing). The checkpoint system means you can replay from any node with modified inputs. This is the clearest debugging experience of the three.
CrewAI: the crew's internal delegation messages are logged, but the framework's prompt injection makes it harder to see exactly what each agent received. You can enable verbose mode, which dumps the role prompt plus task prompt, but separating "what the framework added" from "what your task specified" requires reading the assembled prompt carefully.
Smolagents: debugging is closest to debugging Python code. The framework logs the generated script and any Python errors, which are usually more actionable than JSON schema error messages. The downside is that the generated code is sometimes creative in ways that are technically valid but undesirable.
What Actually Changes the Recommendation
The benchmark numbers favor LangGraph for a local model setup, but the real question is what you are optimizing for.
If you are building a system where agent reliability directly affects output quality (where a wrong tool call produces wrong results that reach users), LangGraph's explicit validation nodes and checkpointing make it the right choice for a local model. The graph structure is the overhead; it pays for itself in debuggability and retry control.
If you are prototyping a system where you will eventually upgrade to a frontier model and want to move fast now, CrewAI is still the right choice for development. Just do not benchmark it locally and assume the results transfer to the frontier model environment, because they go in the other direction you might expect: a system that works at 63% success on a local model can reach substantially higher success rates on a frontier model with the same configuration, so the problems you identify locally may disappear on upgrade rather than persist.
Smolagents is the underrated option for a self-hoster who wants to stay on local models indefinitely. The code-generation approach is more resilient to weaker instruction-following, and Hugging Face actively maintains it with local model support as a first-class concern. The documentation is thinner than LangGraph's and the ecosystem is smaller, but the failure modes are more legible.
When the Answer Is "Upgrade the Model"
There is a threshold below which no framework compensates for the underlying model's limitations. We benchmarked the same task against the same frameworks using a 3B model and saw tool-call success rates drop to levels where none of the three frameworks was usable without extensive hand-holding in the system prompt. At that scale, switching frameworks does not solve the problem.
For a 3-step task with tool calls that require structured arguments, we found that an 8B model at Q4 quantization is approximately the minimum useful tier for any of these frameworks without heavy prompt engineering. A 13B or 70B model in the same quantization improves results across all three frameworks.
If your hardware can run a 13B model, the framework comparison numbers shift: LangGraph remains the strongest, but CrewAI's success rate recovers substantially, and the gap between them narrows to the point where CrewAI's speed-to-prototype advantage may outweigh the reliability difference for many use cases.
Production Checklist
Before committing to a framework for a local-model agent system:
- [ ] Run the same task through your framework of choice many times (we used 20 runs) on the model you will actually use in production, not a frontier model
- [ ] Measure tool-call success rate explicitly (count validation errors, not just exceptions)
- [ ] Add a validation node or step between model output and tool execution for any tool where invalid arguments cause side effects
- [ ] Implement logging of raw model output before framework parsing (this is the most useful debugging artifact)
- [ ] Test the semantic-error case: have your tools return enough context for the model to verify its own results before synthesis
- [ ] Set a retry limit (2 is usually right) and define the fallback behavior explicitly. Do not let the framework retry indefinitely
Conclusion
The 2026 framework consensus (LangGraph for control, CrewAI for speed, Smolagents for minimalism) is accurate for the conditions it was tested under. On a local model, the ranking on reliability is LangGraph first by a meaningful margin, Smolagents second, and CrewAI third. The reasons are architectural, not quality-of-code: frameworks that use implicit coordination via natural language prompts depend on the model's ability to track complex prompt context, which degrades with model size; frameworks that make the coordination structure explicit in code or graph definitions remain more reliable.
For a self-hoster who wants to build a production agent system on local hardware and have it work reliably, LangGraph is the current answer. For a self-hoster who wants to prototype quickly and has a plan to move to a frontier model or a larger local model, CrewAI gets you there faster and the transition is not as painful as these numbers make it look. The problems you encounter locally are largely solved by model capability, not framework changes.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: run the benchmark against your own local model and reply with which framework won. Especially interested in results from 13B+ models where the CrewAI gap narrows.
Sources
- Alice Labs, "Best AI Agent Frameworks 2026: 7 Compared": https://alicelabs.ai/en/insights/best-ai-agent-frameworks-2026
- DEV Community, "AI Agents in 2026: LangGraph vs CrewAI vs Smolagents with Real Benchmarks on Local LLMs": https://dev.to/pooyagolchian/ai-agents-in-2026-langgraph-vs-crewai-vs-smolagents-with-real-benchmarks-on-local-llms-4ma1
- LangGraph 1.0 documentation: https://langchain-ai.github.io/langgraph/
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment