seniorReportedcurated
How do you ingest a 200-page PDF of charts and tables where the answer exists only inside a line chart, with no matching text?
30-second answer
- Text extraction returns almost nothing for a chart, so by default it is not indexed at all, which is a silent recall hole.
- The options are OCR for scanned text, a vision model to caption each chart and embed the caption with a pointer to the page image, table extraction for tabular regions, or page-image embedding such as ColPali when layout is the content.
- For a price that exists only in a chart, the caption must include axis labels, series names and period, or retrieval will never match the question.
- The cost is a vision-model call per figure, and captions are model output that can invent a trend, so sample them against the rendered image.
- Index the image reference alongside the caption so the generator can be given the actual figure, and evaluate per modality because text recall says nothing about chart retrieval.
Likely follow-ups
Open a question to rehearse the answer.
How do you evaluate caption quality when a vision model invents a plausible number for an axis value?
- Render the figure and check the caption's claims against it on a sample, since a wrong caption makes the figure retrievable for claims it does not support.
- Test with figures whose values you can confirm from a source table, so an invented axis value is detectable rather than plausible.
- Ask for captions that quote axis labels verbatim, and reject captions stating values the image does not print.
Would you index the page image, the caption, or both, and what does each cost at query time?
- Index the caption for matching and keep the image reference for generation, which separates retrieval from what the model reads.
- Page-image embedding matches layout-heavy questions, but its token cost is paid when the page image is created at ingest and again when it is attached to a query, so reserve it for pages where layout is the content.
- Attaching images to the prompt spends prefill tokens and time to first token, so cap images per request and cache rendered pages.
How do you keep chart captions fresh when the source PDF is re-issued with revised figures?
- Key the caption on document version plus a figure hash, and regenerate only the figures whose bytes changed.
- Store the caption with its source span and bounding box, so a revised figure can be traced and its old caption retired.
- Propagate revisions through the same change feed as text, or the old caption keeps answering after the PDF is re-issued.
Red flags
- assumes OCR recovers chart data
- misses unindexed content
Concepts this tests
Tags
Source: www.reddit.com
More in Data & storage
- How do you choose a chunk size, and what concretely goes wrong at each extreme?
- Users say a rate-limits answer quotes correct numbers but never says what they apply to. You trace it to one chunk. Diagnose and fix.
- Page 1 of a financial report says "all amounts in thousands". How do you stop chunks from losing that document-wide context?