A hand-tuned prompt scored 91%. A compiled one scored 99%.
Route a supply-chain question wrong and an inventory query comes back with a vehicle-routing answer. I replaced the hand-written classifier prompt with a DSPy-compiled one, then spent longer on the measurement than on the compile.
A hand-tuned prompt scored 91%. An optimizer-written one scored 99%. Both measured on queries neither had seen during training.
Why it matters
The first decision an AI system makes is what kind of question it received. Route it wrong and a supply-chain manager asking about inventory gets a vehicle routing answer. The recommendation looks professional. The numbers are for the wrong problem.
The common starting point is a hand-written prompt describing each category. It works until a user phrases something the author never considered, and every miss means another patch.
There is another way. Label enough examples to define what correct means, then let an optimizer write the prompt against that target.
I stopped tuning the prompt and started defining the evaluation. Eight points came from that switch.
How it works
Stanford’s DSPy framework treats prompts as compiled programs: define the inputs, outputs, and a quality metric, then let an optimizer write the instructions and pick the examples. Program, don’t prompt.
I applied this to the intent classifier in a LangGraph supply-chain ERP copilot that routes across 10 bounded categories. I hand-labeled 100 queries, 10 per category. The optimizer rewrote the instructions and kept the demonstrations that scored best.
The measurement took longer to design than the compile. A single train/test split on 100 examples is too noisy to trust: one held-out example moves the number 5 points. So I ran 5-fold cross-validation stratified by intent, so every query was scored once, by a fold model that never trained on it. Baseline 91.0%. Compiled 99.0%. Plus 8 points, give or take 4.5, with every fold improving.
That accuracy costs tokens: the compiled prompt carries its learned instructions and four demonstrations into every call.
What surprised me was the labeling. Deciding what “correct” means for 100 ambiguous questions forced out rules I had never written down. Should “find me a cheaper route for 500 units” be network routing or supplier allocation? It depends on whether the user controls the logistics network or the supplier contracts. That distinction lived in my intuition until labeling made it explicit.
A limit worth stating: those 100 labels are mine alone, so the number holds for the queries I anticipated. Production traces are the next labeling round.