Hybrid Document Extraction (VLM + OCR)

Off-the-shelf OCR breaks on real-world documents: skewed photos, crumpled receipts, mixed layouts, handwriting on cheques. And cloud document-AI APIs mean shipping sensitive financial documents off-site. Teams need extraction that is both accurate on messy inputs and private.

Extract structured data (line items, totals, dates, payees, account numbers, arbitrary fields) from any document reliably, even messy, photographed or handwritten, without sending a single page to a third-party API.

A hybrid technique. Classical computer vision preprocesses the image (detection, deskew, region proposals); PaddleOCR detects and recognizes text with its precise position and layout; and an open-source vision-language model reads the whole page to understand structure and emit clean output (structured markdown / key-value / line items). Fusing OCR's exact text with the VLM's layout understanding makes it robust where either alone fails. And because every model is open-source, it runs on-prem with no data leaving your network.

1. 1Ingest a document image (scan, phone photo, or PDF page)
2. 2Computer-vision preprocessing: detect, deskew and propose regions
3. 3PaddleOCR detects and recognizes text with its position and layout
4. 4An open-source VLM reads the page and structures the content
5. 5Return clean structured data with each value mapped to its source box

• PaddleOCR: open-source text detection + recognition
• Open-source vision-language model (layout & semantic understanding)
• Classical CV: deskew, region detection, bounding boxes
• Our fusion step combining OCR text with VLM structure
• Structured output (markdown / key-value / line items)

• Handles messy real-world docs: crumpled, photographed, skewed, handwritten
• Receipts, invoices, cheques and forms → structured data
• Line items, totals, dates, payees and account numbers extracted
• Bounding boxes tie every value back to its source region
• 100% open-source models: fully on-prem, no third-party API

A modular pipeline: an image-preprocessing / CV stage, a PaddleOCR text stage, and an open-source VLM understanding stage, fused by our own logic into structured output. Because every model is open-source and self-hosted, the whole pipeline deploys on your own hardware (GPU for throughput or CPU for smaller volumes) with no document ever leaving your environment.

Output is structured and auditable: clean markdown / key-value data alongside bounding-box overlays that map each extracted value to exactly where it appeared on the page, so results are easy to verify and correct.

• PaddleOCR (Paddle)
• Open-source VLM (self-hosted)
• OpenCV / computer-vision preprocessing
• On-prem / self-hosted inference

Runs fully on-premise on open-source models (GPU for speed or CPU for lower volumes), so sensitive financial documents never leave your infrastructure. No third-party document-AI API, and no per-page cloud cost.