The copilot keeps answering when the language model goes dark
A free tier capped at 50 API calls a day forced every design decision in this system. The constraint is gone now. The architecture it produced stayed, because it earned its place.
I built an AI copilot that keeps answering even when its language model goes dark. That design came from a hard limit during early development: 50 API calls a day.
Why it matters
A system that works in a demo and a system that survives production are not the same thing. The gap shows up when something fails: when the model quota runs out mid-afternoon and the system has to keep answering with nobody available to intervene.
When compute is unlimited, the architecture can afford to be fragile. Take that budget away, and every design decision changes. The constraint I built under is gone now. The architecture it forced stayed, because it earned its place.
The systems enterprises trust are the ones that still work when the compute budget runs out.
How it works
I built a LangGraph multi-agent supply-chain ERP copilot and deployed it to Azure Kubernetes Service running GPT-4o. During early development the LLM ran on a free tier capped at 50 requests per day. That limit shaped the whole design.
The system researches every question two ways at once, then double-checks its own findings before answering. In detail: a text-to-vector model (BGE-large-en-v1.5, 1024 dimensions) finds documents by meaning, BM25 finds them by keyword, the two lists merge through Reciprocal Rank Fusion, a second model re-ranks them, and a quality gate drops any chunk it judges irrelevant before the LLM ever sees it.
Every tool call passes through a typed contract. 6 servers validate the input against a schema before any solver or database query runs, so a malformed request is rejected before it reaches the math.
It also runs without the LLM at all. The async stack handles concurrent user sessions without blocking. When the model is unavailable, a keyword classifier and pattern matcher take over, and all 7 operations research solvers run untouched. A procurement analyst running a morning check still gets an answer, even if the model went down overnight.
What caught me out: when the quota ran out, the knowledge-graph agent didn’t throw an error. It silently returned nothing. A silent failure is worse than a loud one. The fix was a deterministic fallback that fires on any LLM exception, not just the quota limit, so the system always returns something it can stand behind.
The hardest part of building under constraints is accepting that the constraints might be the design.