Executive summary

• Problem. On 100 FinQA test questions, base Llama-3-8B gave the checkable thought-then-answer format 0% of the time.
• What I did. On one RTX 4090 I ran supervised fine-tuning (SFT) with LoRA, built AI-labeled preference pairs with Zephyr-7B-β (RLAIF), and trained Direct Preference Optimization (DPO) on them.
• Result. SFT raised format adherence to 100% and reached 58% official FinQA execution accuracy. DPO scored 59%, a gap I can't tell apart from noise (McNemar p = 1.0). I traced that null result to 3 causes, and found a stop-token bug that had inflated an earlier hallucination figure.

Problem statement

Why it matters. An analyst who gets a number from an AI model has to decide whether to trust it. If the model also shows which table rows it used and the arithmetic it ran, the analyst can check the number in seconds. If it returns free prose, checking means redoing the work.

Technical formulation. FinQA (Chen et al., 2021) pairs questions with text and tables from company financial reports, plus a gold reasoning program and an executable answer. I framed the task as conditional generation: given the context and question, the model writes a <thought> block with the supporting facts and the reasoning program, then an <answer> block with the final number.

Scope. In scope: format learning through SFT, preference tuning through RLAIF and DPO, and deterministic evaluation on held-out test questions. Out of scope: retrieval (the model receives the question's own context) and unanswerable questions (I evaluated no unanswerable split).

Success criteria. Two questions I can measure. Does the model produce the checkable format? Does it compute the right number, scored with FinQA's official execution accuracy?

Data engineering

Source. FinQA's public training split: 6,251 examples (Chen et al., 2021, CC BY 4.0). I held out 5% to watch for overfitting, because an overfit SFT model makes a poor starting point for DPO.

Formatting. I converted each example to a chat conversation whose assistant turn holds the supporting-fact indices and the reasoning program inside <thought>, followed by <answer>.

Preference data (RLAIF). For each training prompt, the SFT model wrote 2 answers, at temperatures 0.3 and 0.9. Zephyr-7B-β chose the better one with a written pairwise rubric. I dropped pairs with cosine similarity above 0.85, since near-identical pairs give the preference loss little to separate. That left 1,647 pairs.

What my audit of that data found.

• Both answers were wrong in 49% of pairs and both right in 30%. Only 20% had exactly one correct answer (13% right-wrong, 7% wrong-right), and in those Zephyr picked the correct one 65.3% of the time.
• No pair taught refusal: the set holds 0 refusal pairs.
• 91.6% of prompt-plus-chosen sequences exceed 1,024 tokens while max_length was left unset, so many were probably truncated.

Architecture

Baseline. Base Llama-3-8B, prompted with the same context and question.

Stage 1: SFT. LoRA (r = 16, alpha = 32) on all attention and MLP projections, cosine schedule with 5% warmup, paged AdamW, gradient checkpointing, effective batch 48, 2 epochs on one RTX 4090. The scripts also support multi-GPU DDP. Training loss fell from about 2.7 to 0.785 and token accuracy reached 80.8%. Both are training-time numbers. The test results are in the next section.

Stage 2: RLAIF labels. Described above. I kept the annotator (Zephyr-7B-β) out of the evaluation, so the model that labels preferences doesn't grade the result.

Stage 3: DPO. TRL's DPO trainer on a 4-bit NF4 base with a LoRA adapter, learning rate 5e-6, 1 epoch, beta 0.3 in the committed run.

Why DPO gave no measurable gain. I found 3 causes:

1. Weak label signal. Most pairs had no correct-versus-wrong contrast, and where they did, the annotator was right 65.3% of the time.
2. Truncation. Most sequences were probably cut before the answer.
3. Adapter mismatch. The committed DPO adapter came from an older training design, and the current evaluation setup loads it on top of the SFT-merged model, which would apply the SFT update twice.

What I changed in the code for the re-run. I now merge the SFT adapter into the base and train a fresh LoRA, so the reference policy is exactly the SFT model, as the DPO derivation requires (Rafailov et al., 2023, Sec. 4). Zephyr also judges every pair twice with the answer order swapped and keeps only consistent verdicts, the position-bias fix described by Zheng et al. (2023, Sec. 3.4). Both changes are in the code and haven't been run yet.

Evaluation and results

Setup. The first 100 FinQA test questions, one sample per model at temperature 0.1. I scored the final number with FinQA's official execution accuracy and report 95% confidence intervals.

The base model never saw the answer format, so its 0% is guaranteed by construction. A 3-shot base model and an Instruct model are the fair comparison, and both are on the re-run plan.

Reading the numbers. SFT and SFT+DPO differ on only 6 questions, 3 in each direction (McNemar p = 1.0), so DPO gave no measurable gain. With 100 questions each interval spans about 19 points. For reference, FinQANet reached 61.24% execution accuracy on FinQA (Chen et al., 2021, Table 2). My SFT result is in the same range. I make no head-to-head claim: FinQANet uses a retriever and the full test set, and I used neither.

Error analysis: the stop-token bug. My chat template ends each assistant turn with <|eot_id|> (token 128009). The base tokenizer's end token is <|end_of_text|> (128001), so generation never stopped at the first one. At least 99 of 100 SFT responses kept going after the answer and invented new turns. An LLM judge scored the whole run-on text, which produced the 87% hallucination figure. I now stop at <|eot_id|> and cut at the first </answer>.

Status. Research prototype, evaluated offline.

Links

• FinQA alignment repository on GitHub
• DPO adapter on Hugging Face (from the committed run, see the limits above)