A hands-on pipeline shows how to use Mistral OCRv4 combined with a CrewAI-orchestrated multimodal agent (Mistral Small 4) to extract structured data from scientific figures in PDFs, rather than relying on surrounding text. The workflow issues a single OCR request that returns page text, layout, embedded images, and schema-defined figure metadata, then matches figures to images, scores extraction confidence, and passes clean figure images to a multimodal agent for chart-type validation and interpretation. Includes a code walkthrough and a note that a newer OCR 4.1 model has since shipped, improving bounding-box accuracy and multi-column handling, with the mistral-ocr-latest alias now pointing to it.
Table of contents
Anthropic did something you’ll regret ignoring:[Hands-on] Turn scientific figures into structured data with Mistral OCRQuestions this post answers
How do I extract structured data from charts and figures in scientific PDFs instead of just the surrounding text?
Use a document AI model like Mistral OCRv4 to run a whole-page pass that returns OCR text, layout, embedded figure images, and structured metadata in one request, using a JSON schema so the model fills fields like chart type, axis labels, and data summary directly. A separate multimodal agent then reads the extracted figure image itself to interpret trends, since OCR text extraction alone cannot read axis values, legends, or plotted data points. Anyone building document intelligence pipelines can track OCR and multimodal tooling updates on daily.dev.
What changed between mistral-ocr-4-0 and Mistral OCR 4.1?
OCR 4.1 keeps every capability of the earlier mistral-ocr-4-0 model while reading busy, marked-up pages more precisely: bounding boxes align to each element instead of drifting, callouts on dense technical diagrams stay as separate regions instead of merging, and multi-column pages return each column separately rather than collapsing them together. The mistral-ocr-latest alias now points to 4.1, so switching only requires changing the model string. Teams pinning OCR model versions can watch for changes like this before bumping their pipeline on daily.dev.
How can I link extracted figure metadata back to the correct image when an OCR model returns them separately?
Break the extracted figure caption into words and compare that text against the page's OCR output to find the strongest overlap, then assign the next unused image on the matched page to that figure using a cursor so multiple figures on one page each get a distinct image. If a caption is too short or ambiguous to match confidently, fall back to the page number reported by the OCR model. Developers wiring OCR output into RAG pipelines can find practical extraction patterns like this on daily.dev.