60 of the 150 questions I test my AI on point to a table or figure. My AI could not see a single one
My AI assistant searches research papers before it answers, and its search skipped every table and figure. Fixing that exposed a second blind spot, in the scorecard I use to grade it.
60 of the 150 questions I test my AI on point to a table or a figure as evidence.
My AI could not see a single one.
Why it matters
I build an AI assistant that answers questions about research papers and cites its sources. Before answering, it searches the paper for passages to read.
The public question set I test on gives each paper’s text, plus a caption and a picture for each table and figure. In that data, a table’s cells exist only in the picture. My search read the text and skipped the rest.
What surprised me was the size: 40% of my test questions, against the 13% the set’s authors report across all 5,049. The 26 questions whose evidence sat only in a table or figure accounted for 7 of my AI’s 22 refusals to answer.
So I made the abstract, captions and table contents searchable. The share of evidence marked by human annotators that reached my AI rose from 65% to 83%.
Then a code review found a second blind spot, in my own scorecard. It counts a table as found when only its caption arrives. And captions rarely hold the answer: across the 60 table-or-figure questions, only 15% of answer words appear in the caption.
I now check what my AI can see, and what my scorecard counts as seen, before I judge how well it reasons.
How it works
QASPER’s figures_and_tables field holds a caption and an image file name. Its authors set these questions aside in their baselines and left multimodal question answering to future work.
The papers come from arXiv with LaTeX source, so I parse tabular environments into Markdown, match each to its caption (Dice ≥ 0.6 on caption words) and index caption plus rows. 3,259 of 3,631 table captions matched a body.
Gold-evidence recall on the 150 questions, no LLM judge:
- 0.485 → 0.646: 20 passages instead of 10, none deleted
- 0.829 with abstract, caption and table chunks
- Table/figure questions with a gold caption in context: 0 → 42 of 60
What I can’t claim yet: that the retrieved table holds the exact number asked for, or that the answers improved. I tuned on these 150 questions, so both get tested on fresh ones next.
The 4 checks I now run first on a document AI:
- What share of test questions rely on tables or figures?
- Does the search index contain them?
- Are refusals concentrated in those questions?
- Does the score credit the caption or the cell?
I wrote up how I ran each check. Happy to share it, or to run it on your system if it reads reports or spec sheets full of tables.