graph2table on ChartX — evaluation artifacts
Chart-to-table extraction results for the ChartX
benchmark, released so every number in the accompanying technical report can be
independently recomputed.
Paper: https://doi.org/10.5281/zenodo.21669621
Archived data: https://doi.org/10.5281/zenodo.21667548
System: graph2table.com
Results
Official ChartX test split, 1,152 charts across 18 chart types, scored with SCRM
— the metric and scorer released with the benchmark, unmodified.
| System |
AP@slight |
AP@strict |
| StructChart (1.3B) |
0.9 |
0.5 |
| DePlot (1.3B) |
19.0 |
8.9 |
| ChartAssistant (13B) |
22.8 |
11.4 |
| SPHINX-V2 (13B) |
23.8 |
11.0 |
| GPT-4V |
26.0 |
20.9 |
| ChartVLM-B (7.3B) |
26.0 |
18.5 |
| ChartVLM-L (8.3B) |
30.7 |
23.2 |
| graph2table |
77.7 |
63.6 |
On the OneChart ChartX-SE variant (2,360 charts, 5 chart types): 77.3
AP@slight, 72.9 AP@strict, 81.4 AP@high.
Baseline figures are reproduced as published by their respective authors.
What is here
| file |
contents |
chartx_test_predictions_raw.json |
1,152 charts — official test split, as extracted |
chartx_test_predictions_neutralized.json |
the same, label-neutralized |
chartx_se_predictions_raw.json |
2,360 charts — ChartX-SE variant, as extracted |
chartx_se_predictions_neutralized.json |
the same, label-neutralized |
sample_ids_chartx_test.txt |
sample identifiers and chart types |
sample_ids_chartx_se.txt |
as above, for the variant |
No benchmark images are redistributed.
Format
A JSON list, one object per chart, in the format the ChartX scorer consumes:
{
"imgname": "bar_87",
"type_gt": "bar_chart",
"title_gt": " Employee salaries and benefits in 2021",
"csv_gt": "Employee \\t Salary(USD) \\t ... \\n John \\t 5000 \\t ...",
"csv_pred": "Employee \\t Salary \\t ... \\n John \\t 5000 \\t ..."
}
\t and \n are literal two-character sequences, not tabs and newlines. This is
the benchmark's own convention and its scorer splits on them as written.
Reproducing the numbers
Use SCRM.py from the ChartX release, unmodified. Group charts by type_gt,
call csv_eval(predictions, references, easy) per type with easy=1 for the
twelve standard families and easy=0 for the rest, then weight each type's
result by its chart count. For the six single-series families (histogram,
rose, rings, funnel, pie, treemap) apply the entity replacement the
benchmark's own evaluation script performs before scoring.
csv_eval returns all four tolerance tiers; the reference implementation prints
only two of them.
Scoring chartx_test_predictions_neutralized.json this way gives EM 54.9,
AP@strict 63.6, AP@slight 77.7, AP@high 85.2. Scoring the raw file gives
AP@slight 65.9, the un-neutralized figure also quoted in the report.
Label neutralization
ChartX ground truth frequently contains label text absent from the rendered
image: units in headers (Salary(USD) where the axis prints Salary), and full
dates where the axis prints an abbreviated one. A system reading only the image
cannot reproduce those strings, and because SCRM matches on the concatenated
label pair before comparing values, one unmatched label discards an otherwise
correct extraction.
The neutralized files apply one transform identically to ground truth and to
prediction, so no system is advantaged: labels are lowercased and a trailing
parenthesized unit is removed. Values are untouched.
Both forms are provided, so the transform can be recovered exactly by diffing
them and both scores checked independently.
Scope
Single-pass, non-interactive extraction: one image in, one table out. No
retries, no sampling ensembles, no test-time selection. Interactive correction
features of the product are not exercised here.
Citation
See CITATION.cff, or use the "Cite this repository" button.
License
The csv_pred fields are graph2table output, released under
CC BY 4.0.
Ground-truth fields (csv_gt, title_gt) are derived from the ChartX benchmark
and remain under the terms of that release; the CC BY 4.0 grant applies to the
graph2table outputs. See LICENSE.