AI’s Diligence Isn’t Enough: Why Volume Can Fail When It Matters Most

Imagine dating someone who always shows up with the right words, never lies, and seems perfect on paper. But when the moment to commit arrives, they hesitate or walk away. In the world of AI, this scenario is more than a metaphor — it’s a real challenge. No matter how thorough or diligent, an AI can miss the most critical opportunity if it lacks focus and prioritization. That’s the core lesson from a groundbreaking experiment in AI decision-making, where even the most comprehensive model stumbled at the decisive moment.

The Experiment: Putting AI Through Its Paces

In mid-2026, researchers at Firmulate ran a live test called the Crucible League, involving four leading AI models — including a new competitor, Kimi K3, and the well-known GPT-5.6-sol. The goal? To simulate a small software company’s worst week, complete with customer crises, internal temptations, and the pressure to close a significant €55,000 deal. Each model faced identical scenarios: same customers, same issues, same incentives to cheat or cut corners. Every decision was carefully recorded and auditable, providing a transparent view into how each AI behaved under pressure.

The Key Findings: Diligence Alone Is Not Enough

All four models demonstrated impressive abilities: they identified every crisis and refused every manipulation attempt — even the most sophisticated social engineering tactics, like fake CEO messages or staged reporter inquiries. Kimi K3, notable for its discipline, refused all of these requests without hesitation. Meanwhile, the other models also showed integrity, leaning into their analysis and resisting shortcuts.

The Hidden Weakness: The Critical Data in the Files

Despite this, only two models managed to close the deal effectively. The decisive factor was not their ability to spot crises or refuse manipulation — it was their capacity to find and use information buried deep within the company’s own documents. The files contained a crucial piece of data that, if read, would have secured the full €4,583 monthly recurring revenue (MRR). The models that examined these references won the customer and signed the contract at full price.

Discipline Versus Prioritization

The most thorough participant in the experiment was Opus 4.8, which learned over 80 rules and performed detailed analyses. Yet, it still finished last because it failed to escalate its findings appropriately — instead leaving some attempts into locked departments, risking missed opportunities. This highlighted a fundamental insight: diligence and volume of rules do not guarantee impact. Prioritization, focus, and discipline to act on the most critical information are what truly matter.

Social Engineering and Trust

All models successfully refused social engineering attempts, like staged CEO messages or background questions. For example, Kimi K3 regarded such requests as potential impersonation or approval bypasses, refusing to act without proper verification. This demonstrates that AI can be trained to maintain integrity against manipulative tactics, a vital trait for trustworthy automation.

The Real-World Company: A Living Testbed

The experiment was conducted in a simulated but realistic environment: a live company with 13 synthetic employees, managing real money mechanics, burning €105,000 monthly against a backdrop of €2,300 MRR. Every decision, from customer support to strategic negotiations, was versioned and observable at Firmulate’s live site, firmulate.com/live. This setup allows enterprises to run their own scenario wargames against an AI model, testing how their future AI workforce might perform before deploying it in real operations.

Implications for Business and AI Development

The core takeaway is that diligent, comprehensive AI models are not enough if they lack focus on critical priorities. The models that succeeded were those that identified and acted on key pieces of information — like the buried data — rather than simply analyzing everything exhaustively. For organizations deploying AI in customer relations, sales, or decision-making, this underscores the importance of guiding principles: teach your AI to prioritize and escalate key insights, not just process volume.

What This Means for Your Business

If AI is to touch your CRM, support queue, or forecasting tools, ask not only if it can write well — but if it can finish what it starts, read your files properly, and stay honest under pressure. Volume of work is secondary to quality and focus. As the experiment shows, even the most diligent AI can leave deals on the table if it doesn’t prioritize effectively.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI