Without column names, GPT-4 scored 0.55 out of 1 on GitHub tables. On SAP customer tables: 0.03
Company tables store codes, not their meaning. A study of GPT-4 on real SAP customer tables shows what an AI does with them: it sees what happened, then guesses what it means.
Without column names, GPT-4 scored 0.55 out of 1 on a public benchmark of GitHub tables. On SAP customer tables: 0.03.
Adding the names lifted SAP only to 0.17. One possible cause, per the researchers: column names that are often abbreviations.
Why it matters
Take a sales order from a public SAP dataset. It holds fields like Sales Group, Plant, Shipping Point and Payment Terms. Each holds a code.
Your colleagues know what those codes stand for. The table doesn’t say.
SAP defines Sales Group: “A group of sales people who are responsible for processing sales of certain products or services.” In the published sales data, that column holds only short codes.
The sentence exists. It never appears in the table itself, as the infographic shows side by side.
So an AI reading company tables often sees what happened, then guesses what it means. Whoever acts on its answer inherits that guess.
I now think a reliable AI application on company data starts long before the first question: with writing down what every code means.
How it works
The numbers come from a VLDB 2024 workshop study on column type annotation: labelling what kind of information each column holds. The score is support-weighted F1, from 0 to 1.
With names, the scores are 0.98 on a public GitTables subset against 0.17 on 100 real SAP customer tables. That gap looks wider, but the authors explain why existing evaluations usually leave names out: the labels are derived from them, so the task would become trivial.
The SAP set also has 5,063 possible types against 122. So the fairer comparison stays inside the SAP data: names lifted a 2023 GPT-4, shown 3 rows per table, only from 0.03 to 0.17. The prompt held a list of type labels and one solved example, but no descriptions of the fields.
One next step the authors suggest: helpful context in the prompt, such as example values for each column type. That is part of what Anthropic calls context engineering: “curating and maintaining the optimal set of tokens (information) during LLM inference.”
SAP’s SALT-KG benchmark (EurIPS 2025 workshop) builds part of the missing layer: metadata on 990 fields, including an entry for every column of its 500,908 anonymized sales orders from a real ERP system.
Next post: what happened when SAP added that metadata to prediction models.
Where does the meaning of your company’s data codes live today: in a maintained data dictionary, or in one senior colleague’s head?