qvac-lib-infer-llamacpp-embed

This native C++ addon, built using the Bare Runtime, simplifies running text embedding models to enable efficient generation of high-quality contextual text embeddings. It provides an easy interface to load, execute, and manage embedding model instances.

Table of Contents

• Supported platforms
• Installation
• Building from Source
• Usage
  • 1. Import the Model Class
  • 2. Create a Data Loader
  • 3. Create the args obj
  • 4. Create config
  • 5. Instanstiate the model
  • 6. Load the model
  • 7. Generate embeddings for input sequence
  • 8. Unload the model
• API behavior by state
• Quickstart Example
• Other Examples
• Benchmarking
• Tests
• Glossary
• License

Supported platforms

Platform	Architecture	Min Version	Status	GPU Support
macOS	arm64, x64	14.0+	✅ Tier 1	Metal
iOS	arm64	17.0+	✅ Tier 1	Metal
Linux	arm64, x64	Ubuntu-22+	✅ Tier 1	Vulkan
Android	arm64	12+	✅ Tier 1	Vulkan, OpenCL (Adreno 700+)
Windows	x64	10+	✅ Tier 1	Vulkan

Dependencies:

• qvac-lib-inference-addon-cpp (≥1.1.2): C++ addon framework
• qvac-fabric-llm.cpp (≥7248.2.1): Inference engine
• Bare Runtime (≥1.24.0): JavaScript runtime
• Linux requires Clang/LLVM 19 with libc++

Installation

Prerequisites

Ensure that the Bare Runtime is installed globally on your system. If it's not already installed, you can install it using:

npm install -g bare@latest

Installing the Package

npm install @qvac/embed-llamacpp@latest

Building from Source

See build.md for detailed instructions on how to build the addon from source.

Usage

1. Import the Model Class

const GGMLBert = require('@qvac/embed-llamacpp')

2. Create a Data Loader

Data Loaders abstract the way model files are accessed. Use a FileSystemDataLoader to load model files from your local file system. Models can be downloaded directly from HuggingFace.

const FilesystemDL = require('@qvac/dl-filesystem')

// Download model from HuggingFace (see examples/utils.js for downloadModel helper)
const [modelName, dirPath] = await downloadModel(
  'https://huggingface.co/ChristianAzinn/gte-large-gguf/resolve/main/gte-large_fp16.gguf',
  'gte-large_fp16.gguf'
)

const fsDL = new FilesystemDL({ dirPath })

3. Create the args obj

const args = {
  loader: fsDL,
  logger: console,
  opts: { stats: true },
  diskPath: dirPath,
  modelName
}

The args obj contains the following properties:

• loader: The Data Loader instance from which the model file will be streamed.
• logger: This property is used to create a QvacLogger instance, which handles all logging functionality.
• opts.stats: This flag determines whether to calculate inference stats.
• diskPath: The local directory where the model file will be downloaded to.
• modelName: The name of model file in the Data Loader.

4. Create config

The config is a dictionary (object) consisting of hyper-parameters which can be used to tweak the behaviour of the model.
All parameter values should be strings.

const config = {
  device: 'gpu',
  gpu_layers: '99',
  batch_size: '1024',
  ctx_size: '512'
}

Parameter	Range / Type	Default	Description
-dev	"gpu" or "cpu"	"gpu"	Device to run inference on
-ngl	integer	0	Number of model layers to offload to GPU
--batch-size	integer	2048	Tokens for processing multiple prompts together
--pooling	{none,mean,cls,last,rank}	model default	Pooling type for embeddings
--attention	{causal,non-causal}	model default	Attention type for embeddings
--embd-normalize	integer	2	Embedding normalization (-1=none, 0=max abs int16, 1=taxicab, 2=euclidean, >2=p-norm)
-fa	"on", "off", or "auto"	"auto"	Enable/disable flash attention
--main-gpu	integer, "integrated", or "dedicated"	—	GPU selection for multi-GPU systems
verbosity	0 – 3 (0=ERROR, 1=WARNING, 2=INFO, 3=DEBUG)	0	Logging verbosity level

IGPU/GPU selection logic:

Scenario	main-gpu not specified	main-gpu: "dedicated"	main-gpu: "integrated"
Devices considered	All GPUs (dedicated + integrated)	Only dedicated GPUs	Only integrated GPUs
System with iGPU only	✅ Uses iGPU	❌ Falls back to CPU	✅ Uses iGPU
System with dedicated GPU only	✅ Uses dedicated GPU	✅ Uses dedicated GPU	❌ Falls back to CPU
System with both	✅ Uses dedicated GPU (preferred)	✅ Uses dedicated GPU	✅ Uses integrated GPU

5. Instantiate the model

const model = new GGMLBert(args, config)

6. Load the model

await model.load()

Optionally you can pass the following parameters to tweak the loading behaviour.

• close?: This boolean value determines whether to close the Data Loader after loading. Defaults to true
• reportProgressCallback?: A callback function which gets called periodically with progress updates. It can be used to display overall progress percentage.

For example:

await model.load(false, progress => process.stdout.write(`\rOverall Progress: ${progress.overallProgress}%`))

Progress Callback Data

The progress callback receives an object with the following properties:

Property	Type	Description
action	string	Current operation being performed
totalSize	number	Total bytes to be loaded
totalFiles	number	Total number of files to process
filesProcessed	number	Number of files completed so far
currentFile	string	Name of file currently being processed
currentFileProgress	string	Percentage progress on current file
overallProgress	string	Overall loading progress percentage

7. Generate embeddings for input sequence

The model outputs a vector for the input sequence.

const query = 'Hello, can you suggest a game I can play with my 1 year old daughter?'
const response = await model.run(query)
const embeddings = await response.await()

When opts.stats is enabled, response.stats includes runtime metrics such as total_tokens, total_time_ms, tokens_per_second, and backendDevice ("cpu" or "gpu"). backendDevice reflects the resolved device used at runtime after backend selection/fallback logic, not only the requested config.

8. Release Resources

Unload the model when finished:

try {
  await model.unload()
  await fsDL.close()
} catch (error) {
  console.error('Failed to unload model:', error)
}

API behavior by state

The following table describes the expected behavior of run and cancel depending on the current state (idle vs a job running). cancel can be called on the model (model.cancel()) or on the response (response.cancel()); both target the same underlying job.

Current state	Action called	What happens
idle	run	Allowed — starts inference, returns QvacResponse
idle	cancel	Allowed — no-op (no job to cancel); Promise resolves
run	run	Throw — second run() throws "a job is already set or being processed" (can wait very briefly for previous job completion)
run	cancel	Allowed — cancels current job; Promise resolves when job has stopped

When run() is called while another job is active, the implementation first waits briefly for the previous job to settle. This preserves single-job behavior while still failing fast when the instance is busy. If the second run cannot be accepted (timeout or addon busy rejection), it throws:

• "Cannot set new job: a job is already set or being processed"

Cancellation API: Prefer cancelling from the model: await model.cancel(). This cancels the current job and the Promise resolves when the job has actually stopped (future-based in C++). You can also call await response.cancel() on the value returned by run(); it is equivalent and targets the same job. Both are no-op when idle.

Quickstart Example

Clone the repository and navigate to it:

cd qvac-lib-infer-llamacpp-embed

Install dependencies:

npm install

Run the quickstart example (uses examples/quickstart.js):

npm run quickstart

Other Examples

• Batch Inference – Demonstrates running multiple prompts at once using batch inference.
• Native Logging – Demonstrates C++ addon logging integration.

Benchmarking

We conduct rigorous benchmarking of our embedding models to evaluate their retrieval effectiveness and computational efficiency across diverse tasks and datasets. Our evaluation framework incorporates standard information retrieval metrics and system performance indicators to provide a holistic view of model quality.

Running Benchmarks

For instructions on running benchmarks yourself, see the Benchmark Runner Documentation.

The benchmarking covers:

• Retrieval Quality:

  • nDCG@k: Quality of ranked results based on relevance and position
  • MRR@k: Position of the first relevant result per query
  • Recall@k: Coverage of relevant results in the top k
  • Precision@k: Proportion of top k results that are relevant

Results are continuously updated with new releases to ensure up-to-date performance insights.

Tests

Integration tests are located in test/integration/ and cover core functionality including model loading, inference, tool calling, multimodal capabilities, and configuration parameters.
These tests help prevent regressions and ensure the library remains stable as contributions are made to the project.

Unit tests are located in test/unit/ and test the C++ addon components at a lower level, including backend selection, cache management, chat templates, context handling, and UTF8 token processing.
These tests validate the native implementation and help catch issues early in development.

Glossary

• Bare Runtime - Small and modular JavaScript runtime for desktop and mobile. Learn more.

License

This project is licensed under the Apache-2.0 License – see the LICENSE file for details.

For questions or issues, please open an issue on the GitHub repository.