@qvac/bci-whispercpp

Brain-Computer Interface (BCI) neural signal transcription addon for qvac, powered by the tetherto/qvac-ext-lib-whisper.cpp fork of whisper.cpp.

Transcribes multi-channel neural signals (e.g., 512-channel microelectrode array recordings) into text using a BCI-trained whisper model running natively via GGML. Output matches the Python BrainWhisperer reference model exactly.

Table of Contents

• Architecture
• Results
• Neural Signal Format
• Installation
• Quickstart
• Model Conversion
• Usage
• Configuration
• Tests
• Error Range
• whisper.cpp Patches
• Resources
• Glossary
• License

Architecture

Neural Signal (512ch, 20ms bins)
    │
    ▼
┌──────────────────────────────┐
│   NeuralProcessor (C++)      │
│   - Gaussian smoothing       │  std=2, kernel=100
│   - Day-specific projection  │  low-rank (A·B) + month + softsign
│   - Pad to 3000 frames       │  mel-major layout for whisper.cpp
└──────────────┬───────────────┘
               │  mel features (512 × 3000)
               ▼
┌──────────────────────────────┐
│   whisper.cpp (patched)      │
│   - conv1 (k=7, 512→384)    │  BCI-trained embedder weights
│   - conv2 (k=3, stride=2)   │
│   - Positional encoding      │  learned time PE + sinusoidal day PE
│   - 6-layer encoder          │  windowed attention (w=57) on layers 0–3
│   - 4-layer decoder (LoRA)   │  beam search, length_penalty=0.14
└──────────────┬───────────────┘
               │
               ▼
          Text output

Results

Native GGML inference matches the Python BrainWhisperer reference on all test samples:

Sample	Ground Truth	GGML Native Output	WER
0	"You can see the code at this point as well."	"You can see the good at this point as well."	10.0%
1	"How does it keep the cost down?"	"How does it keep the cost down?"	0.0%
2	"Not too controversial."	"Not too controversial."	0.0%
3	"The jury and a judge work together on it."	"The jury and a judge work together on it."	0.0%
4	"Were quite vocal about it."	"We're quite vocal about it."	20.0%
Average			6.0%

Neural Signal Format

Binary files with the following layout:

Offset	Type	Description
0	uint32	Number of timesteps
4	uint32	Number of channels
8	float32[]	Feature data (row-major: features[t * channels + c])

Each timestep represents a 20ms bin of neural activity. Channels correspond to individual electrodes in a microelectrode array (typically 512 channels).

Installation

cd packages/bci-whispercpp
npm install
VCPKG_ROOT=/path/to/vcpkg npm run build

Prerequisites

• Bare runtime >= 1.24.0
• CMake >= 3.25
• vcpkg with VCPKG_ROOT environment variable set

Quickstart

To run an example you need the BCI model files and (for batch mode) the test fixtures. The download script fetches the model files from the QVAC model registry (no GitHub CLI, no auth) and the neural-signal fixtures from the public release tarball (the fixtures aren't in the registry yet).

cd packages/bci-whispercpp
npm install   # installs @qvac/registry-client (devDependency)

# Download model files (ggml-bci-windowed.bin + bci-embedder.bin) + test fixtures
npm run download-models
# node scripts/download-models.js --models    # models only (from the registry)
# node scripts/download-models.js --fixtures  # fixtures only
# node scripts/download-models.js --force     # re-download even if present

The models land in models/ and the neural-signal fixtures in test/fixtures/. Then run an example:

# Transcribe all bundled fixture samples and print WER
bare examples/transcribe-neural.js --batch

# Transcribe a single neural signal file
bare examples/transcribe-neural.js test/fixtures/neural_sample_0.bin

# Streaming transcription over a sliding window
bare examples/transcribe-stream-neural.js test/fixtures/neural_sample_0.bin

By default the examples look for models/ggml-bci-windowed.bin (with bci-embedder.bin alongside it). Override with WHISPER_MODEL_PATH=/path/to/ggml-bci-windowed.bin or by passing the model path as the final argument.

The model files come from the qvac model registry (engine @qvac/bci-whispercpp, S3 source qvac_models_compiled/bci-whispercpp/...). If you already have them locally, skip the download step and point WHISPER_MODEL_PATH at your copy. The fixtures URL can be overridden with BCI_FIXTURES_URL.

Model Conversion Prerequisites

• Python 3 with numpy, torch, and transformers (pip install numpy torch transformers)

Model Conversion

Convert a trained BrainWhisperer checkpoint. This produces two files, both required for inference:

File	Size	Description
ggml-bci-windowed.bin	~84 MB	GGML model: whisper encoder/decoder (LoRA-merged), tokenizer, positional embedding, windowed attention header
bci-embedder.bin	~24 MB	Day projection weights: low-rank A·B matrices per recording day, month projections, session-to-day mapping

python3 scripts/convert-model.py \
  --checkpoint /path/to/epoch=93-val_wer=0.0910.ckpt

Both files are written to models/ by default. All flags are optional:

Flag	Default	Description
--output	models/ggml-bci-windowed.bin	GGML model output path
--embedder-output	models/bci-embedder.bin	Embedder weights output path
--day-idx	1	Day index for baked positional embedding
--window-size	57	Windowed attention size (0 to disable)
--last-window-layer	3	Last encoder layer with windowed attention
--f32	off	Use f32 for all tensors (avoids f16 precision loss, ~2x larger)

Important: By default both files must be in the same directory at runtime — the addon resolves bci-embedder.bin next to the GGML model file and will fail if it is missing. To store the embedder elsewhere, pass an explicit path via files.embedder (see Usage).

Usage

The package's default export is the high-level BCIWhispercpp class. It owns model lifecycle, an inference queue, and a sliding-window streaming driver on top of the native addon.

const BCIWhispercpp = require('@qvac/bci-whispercpp')

1. Construct an instance

const bci = new BCIWhispercpp({
  files: { model: './models/ggml-bci-windowed.bin' },
  opts: { stats: true }            // optional — surfaces runtime stats on response.stats
}, {
  whisperConfig: { language: 'en', temperature: 0.0 },
  bciConfig:     { day_idx: 1 },   // session day index for day-specific projection
  miscConfig:    { caption_enabled: false }
})

By default the companion bci-embedder.bin must sit next to files.model — the native addon resolves it relative to the model path and will fail to load otherwise. To keep the embedder in a different location, pass its path explicitly via files.embedder:

const bci = new BCIWhispercpp({
  files: {
    model: './models/ggml-bci-windowed.bin',
    embedder: './weights/bci-embedder.bin' // optional override
  }
})

2. Load the model

await bci.load()

load() is idempotent — calling it again unloads the existing model and re-initialises with the current config. There is no progress callback today.

3. Transcribe (batch mode)

Use this when you have the full neural signal up-front. transcribe() accepts the raw bytes (header + body); transcribeFile() is a convenience wrapper that reads the file for you.

const fs = require('bare-fs')

const response = await bci.transcribeFile('./signal.bin')
// or: const response = await bci.transcribe(new Uint8Array(fs.readFileSync('./signal.bin')))

const segments = await response.await()
const text = segments.map(s => s.text).join('').trim()
console.log(text)

if (response.stats) console.log(response.stats) // when opts.stats: true

Concurrent calls are serialised — a second transcribe() waits for the first to settle.

4. Transcribe (streaming mode)

transcribeStream() consumes a stream of bytes (async iterable, sync iterable, Uint8Array, or array of chunks) and decodes a sliding window over the body as data arrives. The first 8 bytes of the stream must be the standard [T u32 LE, C u32 LE] header (T is ignored in stream mode; C must be non-zero).

const response = await bci.transcribeStream(chunkIterable, {
  windowTimesteps: 1500,   // default
  hopTimesteps:    500,    // default — must be < windowTimesteps
  emit:            'delta' // 'delta' (default) | 'full'
})

response.onUpdate(segments => {
  // emit: 'delta' — newly-discovered tail segments since the last window.
  //                 Each segment carries native fields (text, t0, t1, ...) plus
  //                 windowStartTimestep so you can map back to the stream timeline.
  // emit: 'full'  — single { text } entry with the full running transcript.
  for (const s of segments) process.stdout.write(s.text)
})

await response.await()     // resolves when the stream ends and the final window decodes

Streaming constraints:

Option	Constraint
windowTimesteps	positive integer, ≤ 2900 (MAX_WINDOW_TIMESTEPS)
hopTimesteps	positive integer, < windowTimesteps
emit	'delta' or 'full'

Only one stream may be active at a time. response.stats is not populated for streams.

5. Cancel / unload / destroy

await bci.cancel()    // abort an in-flight job or stream; instance remains usable
await bci.unload()    // release native resources; bci.load() can be called again
await bci.destroy()   // permanent — instance cannot be reused

6. Word Error Rate helper

The package re-exports computeWER(hypothesis, reference) for evaluation:

const { computeWER } = require('@qvac/bci-whispercpp')
const wer = computeWER('how does it keep the cost down', 'how does it keep the cost down?')

Output shape

response.await() resolves to an array of segments; response.onUpdate(cb) receives the same shape per emission:

[
  { text: ' How does it keep the cost down?', t0: 0, t1: 280, /* ... */ }
]

In streaming delta mode each segment is annotated with windowStartTimestep. In full mode the array contains a single { text } entry.

Tests

Script	Purpose
npm run test:unit	JS unit tests (brittle-bare test/unit/*.test.js) — no model required
npm run test:integration	JS integration tests against the native addon — requires WHISPER_MODEL_PATH
npm run test:cpp	C++ unit tests (GoogleTest); bare-make rebuilds the addon with BUILD_TESTING=ON
npm run test:dts	Type-checks the published index.d.ts
npm test	Runs test:unit + test:integration

# JS unit tests
npm run test:unit

# JS integration tests
WHISPER_MODEL_PATH=./models/ggml-bci-windowed.bin npm run test:integration

# C++ unit tests
VCPKG_ROOT=/path/to/vcpkg npm run test:cpp

# .d.ts typecheck
npm run test:dts

Integration tests require both ggml-bci-windowed.bin and bci-embedder.bin to be present in the same directory, plus the neural-signal fixtures. The quickest way to get them is npm run download-models (see Quickstart); alternatively produce the models yourself via Model Conversion.

Configuration

BCIWhispercpp accepts two arguments:

new BCIWhispercpp(args, config)

args

Field	Type	Description
files.model	string	Required. Path to BCI GGML model file. By default bci-embedder.bin must sit alongside it.
files.embedder	string	Optional explicit path to the embedder weights file. Overrides the default lookup of bci-embedder.bin next to files.model.
logger	object	Optional logger; wrapped in @qvac/logging. Defaults to a noop logger.
opts.stats	boolean	When true, runtime stats are surfaced on response.stats for batch jobs. Default false.

config.whisperConfig

The convenience defaults below are surfaced explicitly. Any other whisper_full_params key is forwarded untouched to whisper.cpp — see Advanced configuration.

Parameter	Type	Default	Description
language	string	"en"	Language code
temperature	number	0.0	Sampling temperature
n_threads	number	0 (auto)	Number of threads

config.bciConfig

Parameter	Type	Default	Description
day_idx	number	0	Session day index for the day-specific low-rank projection at runtime. Distinct from the conversion-time --day-idx flag, which bakes a positional embedding into ggml-bci-windowed.bin.

config.contextParams

These keys back the whisper_context. Changing any of them between jobs forces a full model reload (unload → re-init → warmup), which can take several seconds.

Parameter	Type	Description
model	string	Optional override; usually set via args.files.model.
use_gpu	boolean	Enable GPU acceleration (Metal on macOS by default).
flash_attn	boolean	Enable flash attention.
gpu_device	number	Select a non-default GPU device.

config.miscConfig

Parameter	Type	Default	Description
caption_enabled	boolean	false	Format segments with <|start|>..<|end|> markers.

streamOpts (passed to transcribeStream())

Parameter	Type	Default	Constraint	Description
windowTimesteps	number	1500	positive integer, ≤ 2900 (MAX_WINDOW_TIMESTEPS)	Decode window size in 20 ms timesteps.
hopTimesteps	number	500	positive integer, < windowTimesteps	How far the window advances between decodes (~33% overlap by default).
emit	string	'delta'	'delta' or 'full'	'delta' emits the newly-discovered tail per window with native segment fields plus windowStartTimestep. 'full' emits a single { text } entry with the running transcript.

The encoder accepts up to ~3000 timesteps per forward pass; MAX_WINDOW_TIMESTEPS = 2900 keeps a safety margin so partial flush windows always fit.

Advanced configuration

whisperConfig is a thin pass-through to whisper.cpp's whisper_full_params. For the full surface (decoding strategy, beam search, VAD, suppression, callbacks, etc.) refer to the upstream reference:

• whisper_full_params in whisper.cpp
• Concrete shapes used in production: see the examples directory and @qvac/transcription-whispercpp for richer usage patterns (VAD, chunking, live streaming).

whisper.cpp Patches

The BCI patches live in the tetherto/qvac-ext-lib-whisper.cpp fork (v1.8.4.2) and are consumed via the qvac-registry-vcpkg port:

Feature	Description
Variable conv1 kernel	Read n_audio_conv1_kernel from model header (k=7 for 512ch BCI vs k=3 for audio)
Windowed attention	Attention mask with configurable window size/layer params in header
BCI SOS tokens	BCI-specific start-of-sequence token handling
Graph placement fix	Correct encoder-graph mask population for the encoder graph refactor

Error Range

All errors thrown by this package are QvacErrorAddonBCI instances (extending QvacErrorBase from @qvac/error) and use codes in the range 26001–27000.

Code	Name	When
26001	FAILED_TO_LOAD_WEIGHTS	Native addon failed to load the GGML model
26002	FAILED_TO_CANCEL	cancel() failed at the addon layer
26003	FAILED_TO_APPEND	Append to processing queue failed
26004	FAILED_TO_DESTROY	destroy() failed at the addon layer
26005	FAILED_TO_ACTIVATE	addon.activate() failed during load()
26006	INVALID_NEURAL_INPUT	Batch input rejected by the addon
26007	JOB_ALREADY_RUNNING	transcribe() called while a job is in flight
26008	MODEL_NOT_LOADED	Inference called before load() or after destroy()
26009	MODEL_FILE_NOT_FOUND	files.model missing/unreadable, or files.embedder provided but missing/invalid
26010	BUFFER_LIMIT_EXCEEDED	Neural signal buffer exceeded the addon limit
26011	FAILED_TO_START_JOB	Addon refused to start the job
26012	INVALID_CONFIG	Constructor / context configuration rejected
26013	EMBEDDER_WEIGHTS_INVALID	bci-embedder.bin failed validation
26014	STREAM_ALREADY_ACTIVE	transcribeStream() called while one is already active
26015	INVALID_STREAM_INPUT	Bad stream input type or streamOpts
26016	INVALID_STREAM_HEADER	Stream [T u32, C u32] header malformed (e.g. C == 0)
26017	WINDOW_TOO_LARGE	windowTimesteps exceeds MAX_WINDOW_TIMESTEPS (2900)

Codes are also re-exported via require('@qvac/bci-whispercpp/lib/error').ERR_CODES for programmatic matching.

Resources

• whisper.cpp fork (Tether): tetherto/qvac-ext-lib-whisper.cpp
• Sibling package — audio transcription: @qvac/transcription-whispercpp
• vcpkg registry: qvac-registry-vcpkg
• BrainWhisperer reference (Python): the model checkpoints converted by scripts/convert-model.py

Glossary

• Bare — small modular JavaScript runtime for desktop and mobile. Learn more.
• QVAC — Tether's open-source SDK for building decentralized, local-first AI applications.
• GGML — tensor library / file format used by whisper.cpp for native inference.
• BCI — Brain-Computer Interface; here, microelectrode-array recordings of neural activity decoded into text.
• Day index (day_idx) — selects the day-specific low-rank projection (A·B) baked into bci-embedder.bin. Sessions recorded on different days use different projections.
• Windowed attention — encoder attention mask restricted to a local window (w=57 over layers 0–3 by default), configured at model conversion time.

License

Apache-2.0