turboquant-search

Vector search for JSON datasets. Build quantized indexes and search with WASM SIMD.

Takes any JSON array, embeds text fields into vectors, compresses them with 3-bit quantization, and runs similarity search entirely via WebAssembly SIMD, in the browser or Node.js.

Install

npm install turboquant-search

Quick Start

import { TurboSearch } from 'turboquant-search';

// Build from any JSON array
const ts = await TurboSearch.from(products, {
  fields: ['name', 'description', 'tags'],
});

// Text search
const results = await ts.search('wireless audio bluetooth', { topK: 5 });
// => [{ index: 0, score: 0.94, data: { name: 'Wireless Headphones', ... } }]

// Find similar items
const similar = ts.similar(0, { topK: 5 });

// Save for later
await ts.save('./products.index.json');

// Load a pre-built index
const loaded = await TurboSearch.load('./products.index.json');

// Clean up
ts.destroy();

CLI

# Build an index
npx tqs build --input products.json --fields "name,description,tags" --output search.json

# Inspect an index
npx tqs info search.json

API

TurboSearch.from(data, options)

Build a search index from a JSON array.

Option	Type	Default	Description
fields	string[]	required	Fields to embed
dim	number	384	Embedding dimensions
bits	number	3	Quantization bits
seed	number	42	Random seed
embedder	Embedder	keyword	Custom embedder

TurboSearch.load(pathOrUrl)

Load a pre-built index from a file or URL.

Instance Methods

ts.search(query, { topK })     // text search
ts.similar(index, { topK })    // find similar items
ts.vectorSearch(vec, { topK }) // search by embedding
ts.save(path)                  // save index to disk
ts.size                        // number of indexed items
ts.destroy()                   // clean up WASM

Custom Embedder

// Works with any embedding source: transformers.js, OpenAI, Gemini, Cohere, etc.
import { pipeline } from '@xenova/transformers';

const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');

const ts = await TurboSearch.from(data, {
  fields: ['text'],
  embedder: {
    async embed(text, dim) {
      const output = await extractor(text, { pooling: 'mean', normalize: true });
      return new Float32Array(output.data);
    },
  },
});

Scalability

Items	Index Size	Search Time
100	~14 KB	<1ms
10,000	~1.4 MB	~5ms
50,000	~7 MB	~15ms
100,000	~14 MB	~30ms

How It Works

1. Text extraction - concatenates specified JSON fields per item
2. Embedding - TF-IDF keyword hashing into 384-dim vectors (or your custom embedder)
3. Quantization - 3-bit TurboQuant compression (1,536 bytes to ~144 bytes per vector)
4. Search - WASM SIMD dot products, returns top-K results

License

MIT