qvac-lib-inference-addon-onnx-ocr-fasttext

This library provides Optical Character Recognition (OCR) capabilities for QVAC runtime applications, leveraging the ONNX Runtime for efficient inference.

The library supports two OCR pipelines:

• EasyOCR pipeline (default): Uses CRAFT detector + language-specific recognizers.
• DocTR pipeline: Uses DBNet detector + CRNN/PARSeq recognizer, matching the OnnxTR Python library.

Table of Contents

• Supported Platforms
• Installation
• Building from Source
• Usage
• Output Format
• Glossary
• Supported Languages
• DocTR Pipeline
• Contributing
• License
• Support

Supported Platforms

Platform	Architecture	Min Version	Status
macOS	arm64, x64	14.0+	Tier 1
iOS	arm64	17.0+	Tier 1
Linux	arm64, x64	Ubuntu 22+	Tier 1
Android	arm64	12+	Tier 1
Windows	x64	10+	Tier 1

Installation

Prerequisites

Install Bare Runtime:

npm install -g bare

Note: Make sure the Bare version is >= 1.19.3. Check this using:

bare -v

Installing the Package

Install the latest version of the package:

npm install @qvac/ocr-onnx@latest

Building from Source

If you want to build the addon from source (for development or customization), follow these steps:

Prerequisites

Before building, ensure you have the following installed:

1. vcpkg - Cross-platform C++ package manager

   git clone https://github.com/microsoft/vcpkg.git
   cd vcpkg && ./bootstrap-vcpkg.sh -disableMetrics
   export VCPKG_ROOT=/path/to/vcpkg
   export PATH=$VCPKG_ROOT:$PATH
   

2. Build tools for your platform:

   • Linux: sudo apt install build-essential autoconf automake libtool pkg-config
   • macOS: Xcode command line tools
   • Windows: Visual Studio with C++ build tools

3. Node.js and npm (version 18+ recommended)

4. Bare runtime and build tools:

   npm install -g bare-runtime bare-make
   

Building the Addon

1. Clone the repository:

   git clone git@github.com:tetherto/qvac-lib-inference-addon-onnx-ocr-fasttext.git
   cd qvac-lib-inference-addon-onnx-ocr-fasttext
   

2. Install dependencies:

   npm install
   

3. Build the addon:

   npm run build
   

This command will:

• Generate CMake build files (bare-make generate)
• Build the native addon (bare-make build)
• Install the addon to the prebuilds directory (bare-make install)

Verifying the Build

After building, verify everything works by running the Hyperdrive example:

bare examples/example.hd.js

This example will:

• Download the detector and recognizer models from Hyperdrive (cached locally in models/hd/)
• Load the OCR model
• Run text recognition on a test image
• Display the detected text with confidence scores

Examples

The examples/ folder contains several examples to help you get started:

Example	Description
example.hd.js	Downloads models from Hyperdrive and runs OCR
example.fs.js	Basic OCR using local model files
exampleGPU.fs.js	OCR with GPU acceleration enabled
example.logger.js	OCR with custom logging
visualize_ocr.js	Runs OCR and saves results to JSON for visualization
draw_boxes.py	Python script to draw bounding boxes on images using OCR results

Usage

The library provides a straightforward workflow for image-based text recognition. The pipeline is selected via the pipelineMode parameter: 'easyocr' (default) or 'doctr'.

1. Configure Parameters

EasyOCR Mode (default)

const args = {
  params: {
    // Required
    langList: ['en'],                        // Language codes for recognizer selection
    pathDetector: './models/ocr/detector_craft.onnx',
    pathRecognizer: './models/ocr/recognizer_latin.onnx',
    // Or use prefix: pathRecognizerPrefix: './models/ocr/recognizer_',

    // Shared optional
    useGPU: true,                            // Enable GPU/NPU acceleration (falls back to CPU)
    timeout: 120,                            // Max inference time in seconds

    // EasyOCR-specific optional
    magRatio: 1.5,                           // Detection magnification ratio (1.0–2.0)
    defaultRotationAngles: [90, 270],        // Rotation angles to try (use [] to disable)
    contrastRetry: false,                    // Re-process low-confidence regions with adjusted contrast
    lowConfidenceThreshold: 0.4,             // Confidence threshold below which contrast retry triggers
    recognizerBatchSize: 32                  // Text regions per batch (lower = less memory on mobile)
  },
  opts: {
    stats: true                              // Enable performance statistics
  }
}

DocTR Mode

const args = {
  params: {
    pipelineMode: 'doctr',                   // Select DocTR pipeline
    langList: ['en'],                        // Language codes (defaults to ['en'] for DocTR)
    pathDetector: './models/doctr/db_mobilenet_v3_large.onnx',
    pathRecognizer: './models/doctr/crnn_mobilenet_v3_small.onnx',

    // Shared optional
    useGPU: false,                           // Enable GPU/NPU acceleration (falls back to CPU)

    // DocTR-specific optional
    straightenPages: false,                  // Apply perspective transform to straighten text regions
    decodingMethod: 'greedy'                 // 'greedy' (all models) or 'attention' (PARSeq only)
  },
  opts: {
    stats: true                              // Enable performance statistics
  }
}

Shared Parameters (both pipelines)

Parameter	Type	Default	Required	Description
pipelineMode	string	'easyocr'	No	Pipeline to use: 'easyocr' or 'doctr'.
langList	string[]	—	Yes (EasyOCR), optional for DocTR	Language codes (ISO 639-1). In EasyOCR mode, determines the recognizer model. In DocTR mode, defaults to ['en'].
pathDetector	string	—	Yes	Path to the detector ONNX model (CRAFT for EasyOCR, DBNet for DocTR).
pathRecognizer	string	—	Yes	Path to the recognizer ONNX model. In EasyOCR mode, can be omitted if pathRecognizerPrefix is provided.
pathRecognizerPrefix	string	—	No	EasyOCR only. Prefix path for recognizer model; the library appends the language suffix (e.g., recognizer_latin.onnx).
useGPU	boolean	true	No	Enable GPU/NPU/TPU acceleration. Falls back to CPU if unavailable.
timeout	number	120	No	Maximum inference time in seconds.

EasyOCR-Specific Parameters

Parameter	Type	Default	Description
magRatio	number	1.5	Detection magnification ratio (1.0-2.0). Higher values improve detection of small text but increase processing time.
defaultRotationAngles	number[]	[90, 270]	Rotation angles to try for text detection. Use [] to disable rotation.
contrastRetry	boolean	false	Re-process low-confidence regions with adjusted contrast.
lowConfidenceThreshold	number	0.4	Confidence threshold (0-1) below which contrast retry is triggered.
recognizerBatchSize	number	32	Number of text regions processed per batch. Lower values reduce memory on mobile.

DocTR-Specific Parameters

Parameter	Type	Default	Description
straightenPages	boolean	false	Apply perspective transform to straighten detected text regions before recognition.
decodingMethod	string	'greedy'	Decoding method: 'greedy' (all models) or 'attention' (PARSeq only).

2. Create Model Instance

Import the library and create a new instance with the configured arguments.

const { ONNXOcr } = require('@qvac/ocr-onnx')

const model = new ONNXOcr(args)

3. Load Model

Asynchronously load the ONNX models specified in the parameters.

try {
  await model.load()
  console.log('OCR model loaded successfully.')
} catch (error) {
  console.error('Failed to load OCR model:', error)
}

4. Run OCR

Pass the path to the input image file to the run method. Supported formats: BMP, JPEG, and PNG.

const imagePath = 'path/to/your/image.jpg'

try {
  const response = await model.run({
     path: imagePath,
     options: {
       paragraph: true,           // Group results into paragraphs (default: false)
       rotationAngles: [90, 270], // Override default rotation angles for this run
       boxMarginMultiplier: 1.0   // Adjust bounding box margins
     }
  })
  // ... process the response (see step 5)
} catch (error) {
  console.error('OCR failed:', error)
}

Runtime Options

Option	Type	Default	Description
paragraph	boolean	false	Group detected text regions into paragraphs based on proximity.
rotationAngles	number[]	Uses defaultRotationAngles	Override default rotation angles for this specific run.
boxMarginMultiplier	number	1.0	Multiplier for bounding box margins around detected text.

5. Process Output

The run method returns a QvacResponse object. Use its methods to handle the OCR results as they become available.

// Option 1: Using onUpdate callback
await response
  .onUpdate(data => {
    // data contains OCR results for a chunk or the final result
    console.log('OCR Update:', JSON.stringify(data))
  })
  .await() // Wait for the entire process to complete

// Option 2: Using async iterator (if supported by QvacResponse in the future)
// for await (const data of response.iterate()) {
//   console.log('OCR Chunk:', JSON.stringify(data))
// }

// Access performance stats if enabled
if (response.stats) {
  console.log(`Inference stats: ${JSON.stringify(response.stats)}`)
}

See Output Format for the structure of the results.

6. Release Resources

Unload the model and free up resources when done.

try {
  await model.unload()
  console.log('OCR model unloaded.')
} catch (error) {
  console.error('Failed to unload model:', error)
}

Output Format

The output is typically received via the onUpdate callback of the QvacResponse object. It's a JSON array where each element represents a detected text block.

Each text block contains:

1. Bounding Box: An array of four [x, y] coordinate pairs defining the corners of the box around the detected text. Coordinates are clockwise, starting from the top-left relative to the text orientation.
2. Detected Text: The recognized text string.
3. Confidence Score: A numerical value indicating the model's confidence in the recognition (range may vary, often 0-1).

[ // Array of detected text blocks
  [ // First text block
    [ // Bounding Box
      [x1, y1], // Top-left corner
      [x2, y2], // Top-right corner
      [x3, y3], // Bottom-right corner
      [x4, y4]  // Bottom-left corner
    ],
    "Detected Text String", // Recognized text
    0.95 // Confidence score
  ],
  [ // Second text block
    [ /* Bounding Box */ ],
    "Another piece of text",
    0.88
  ]
  // ... more text blocks
]

Example:

[[
  [
    [10, 10],
    [150, 12],
    [149, 30],
    [9, 28]
  ],
  "Example Text",
  0.85
]]

The box coordinates are always provided in clockwise direction and starting from the top-left point with relation to the extracted text. Therefore, it is possible to know how extracted text is rotated based on this.

(Note: The exact structure and timing of updates might depend on internal buffering and the paragraph option.)

Glossary

• Bare – Small and modular JavaScript runtime for desktop and mobile.
• QVAC – QVAC is our open-source AI-SDK for building decentralized AI applications.
• ONNX – Open Neural Network Exchange is an open format built to represent machine learning models. Learn more.

Supported Languages

Language support is determined by the recognizer model used. Each recognizer model supports a specific set of languages. The library automatically selects the appropriate model based on the langList parameter.

Recognizer Model	Languages
recognizer_latin.onnx	af, az, bs, cs, cy, da, de, en, es, et, fr, ga, hr, hu, id, is, it, ku, la, lt, lv, mi, ms, mt, nl, no, oc, pi, pl, pt, ro, rs_latin, sk, sl, sq, sv, sw, tl, tr, uz, vi
recognizer_arabic.onnx	ar, fa, ug, ur
recognizer_cyrillic.onnx	ru, rs_cyrillic, be, bg, uk, mn, abq, ady, kbd, ava, dar, inh, che, lbe, lez, tab, tjk
recognizer_devanagari.onnx	hi, mr, ne, bh, mai, ang, bho, mah, sck, new, gom, sa, bgc
recognizer_bengali.onnx	bn, as, mni
recognizer_thai.onnx	th
recognizer_zh_sim.onnx	ch_sim
recognizer_zh_tra.onnx	ch_tra
recognizer_japanese.onnx	ja
recognizer_korean.onnx	ko
recognizer_tamil.onnx	ta
recognizer_telugu.onnx	te
recognizer_kannada.onnx	kn

See supportedLanguages.js for the complete language definitions.

DocTR Pipeline

The DocTR pipeline (pipelineMode: 'doctr') provides an alternative OCR engine based on the OnnxTR project. It uses DBNet for text detection and CRNN or PARSeq for text recognition.

DocTR Models

Model	Type	Description
db_resnet50.onnx	Detector	DBNet with ResNet50 backbone (higher accuracy)
db_mobilenet_v3_large.onnx	Detector	DBNet with MobileNetV3 backbone (faster, mobile-friendly)
parseq.onnx	Recognizer	PARSeq attention-based recognizer (supports attention decoding)
crnn_mobilenet_v3_small.onnx	Recognizer	CRNN with MobileNetV3 backbone (faster, mobile-friendly, greedy decoding only)

EasyOCR vs DocTR

Feature	EasyOCR	DocTR
pipelineMode	'easyocr' (default)	'doctr'
Detector	CRAFT	DBNet (ResNet50 / MobileNetV3)
Recognizer	Language-specific CRNN models	PARSeq or CRNN (single model)
Language support	50+ languages via separate models	French vocab (126 chars)
Mobile	Supported	Supported (MobileNet models recommended)
Perspective correction	N/A	straightenPages option
Image resizing	Auto-resize to 1200px	Full resolution (no resize)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

Support

For questions, bug reports, or feature requests, please open an issue on GitHub.