SochDB Bulk Operations Tool

The sochdb-bulk tool is a specialized high-performance utility designed for "offline" heavy lifting. It bypasses the standard transaction and RPC layers to build, inspect, and convert HNSW index files directly, enabling massive throughput for initialization and migration tasks.

It is the same Rust binary shipped in the core engine (crate version 2.0.3, defined at sochdb-tools/src/main.rs as #[command(name = "sochdb-bulk")]). The Python SDK (pip install sochdb, currently 0.5.9) also exposes a sochdb-bulk console script that wraps this native binary.

Where it fits

sochdb-bulk builds a .hnsw artifact on disk. The gRPC server (sochdb-grpc-server, engine 2.0.3) can then load that file via mmap. See the gRPC Server page for serving the index.

Capabilities

• Index building: parallelized, zero-copy HNSW construction from raw f32 binary or NumPy .npy files.
• Benchmark queries: run a single query vector against a built index to validate recall/latency.
• Index inspection: low-level metadata extraction from .hnsw index files (info subcommand).
• Format conversion: transcode .npy to raw f32 binary.

Performance

The bulk path achieves close to 100% of pure-Rust throughput by avoiding the Python FFI layer (per the binary's own module docs at sochdb-index/src/bin/bulk_load.rs):

Dimension	Throughput (single-threaded HNSW build)
128D	~9,500 vec/s
768D	~1,600 vec/s

Multi-threaded builds (--threads/-t greater than 1) scale further on multi-core machines.

Numbers are indicative

Throughput depends heavily on dimension, M, ef_construction, dataset size, and hardware. Treat the figures above as ballpark guidance, not a guarantee. Always benchmark with the query subcommand on your own data before deploying an index.

Installation

The sochdb-bulk binary is built from the Rust workspace, or installed alongside the Python SDK.

# Via the Python SDK (pulls the prebuilt native binary)
pip install sochdb
sochdb-bulk --help

# Or build from the Rust workspace
cargo build --release --bin sochdb-bulk
./target/release/sochdb-bulk --help

Commands

sochdb-bulk is a clap-based CLI with four subcommands: build-index, query, info, and convert. A global -v/--verbose flag enables debug logging.

build-index

The core command. It takes a raw vector file (or .npy) and produces a fully optimized, memory-mappable HNSW graph file (.hnsw).

Algorithm:

1. Load: memory-maps (or, with --direct-read, reads) the input file for zero-copy access.
2. Order (optional): with --ordering, reorders vectors for cache locality before insertion.
3. Insert: inserts in contiguous batches (--batch-size, default 1000) using insert_batch_contiguous (zero-copy from the mmapped slice). Multi-threaded when --threads is greater than 1.
4. Save: serializes the graph topology to disk with compression (save_to_disk_compressed).

Flags (from sochdb-tools/src/main.rs):

Flag	Default	Purpose
-i, --input	(required)	Input vector file (raw f32 or .npy)
-o, --output	(required)	Output .hnsw index file
-d, --dimension	auto for .npy	Vector dimension (required for raw f32)
-f, --format	auto from extension	raw_f32 or npy
--ids	none	Optional ID file (raw u64); defaults to sequential 0..N
-m, --max-connections	16	HNSW M (max connections per node)
-e, --ef-construction	100	Build-time ef_construction
--batch-size	1000	Insertion batch size
-t, --threads	0 (auto)	Number of build threads
--quiet	(off)	Suppress the progress bar
--direct-read	(off)	Use direct read() instead of mmap (avoids page-fault storms)
--prefault	(off)	Prefault the mmap region for residency
--telemetry	(off)	Capture page-fault telemetry during the build
--ordering	none	Locality ordering: none, random_projection (rp), kmeans, block

No --metric flag on the CLI

build-index does not accept a --metric flag. The bulk builder constructs the index from HnswConfig::default() with only max_connections and ef_construction overridden, which means the distance metric is the engine default (Cosine), ef_search is 500, and precision is F32. To build with a non-cosine metric, use the Python native sochdb 2.0.3 package (HnswIndex(dimension, metric="euclidean")) instead — see the Python SDK guide.

Usage:

sochdb-bulk build-index \
    --input large_dataset.npy \
    --output prod.hnsw \
    --dimension 1536 \
    --max-connections 32 \
    --ef-construction 200 \
    --threads 8

query (benchmark mode)

Runs a single query vector against a built index. Useful for validating recall/latency before deploying the index to a live server.

Flag	Default	Purpose
-i, --index	(required)	.hnsw index file
-q, --query	(required)	Query vector file (single vector, raw f32)
-k, --k	10	Number of neighbors to return
-e, --ef	none	Search ef parameter

Usage:

sochdb-bulk query \
    --index prod.hnsw \
    --query test_vector.f32 \
    --k 10 \
    --ef 128

--ef is accepted but not yet applied

The query handler currently calls the default search(query, k) path and does not forward --ef to the search. The flag is parsed for forward-compatibility but does not change ef_search today. To tune ef_search at query time, set it on the index when serving (or use the Python SDK's set_ef_search / per-search ef_search).

info

Prints low-level metadata for a .hnsw file: vector count, dimension, max layer, average connections, and on-disk file size.

Usage:

sochdb-bulk info --index prod.hnsw

convert

A utility for efficient format transcoding (memory-mapped, no Python loop).

Supported input formats (auto-detected from extension, or via --from-format):

• npy: standard NumPy binary format (.npy).
• raw_f32: flat little-endian array of f32 (aliases: raw, f32, bin). No header — requires --dimension.

Supported output format (--to-format):

• raw_f32 only (aliases: raw, f32). Converting to .npy is not currently implemented in the convert subcommand.

Flag	Default	Purpose
-i, --input	(required)	Input file
-o, --output	(required)	Output file
--from-format	auto from extension	Input format override
--to-format	(required)	Output format (raw_f32)
-d, --dimension	none	Dimension (required when input is raw f32)

Usage:

sochdb-bulk convert \
    --input data.npy \
    --output data.f32 \
    --to-format raw_f32

Serving large pre-built indexes (64 MB gRPC limit)

When the server loads or accepts large vector payloads, message size matters. In sochdb-grpc-server only the VectorIndexService is configured for large messages:

.max_decoding_message_size(64 * 1024 * 1024)  // 64 MB
.max_encoding_message_size(64 * 1024 * 1024)  // 64 MB

All other gRPC services use the tonic default (4 MB decode). This 64 MB ceiling on VectorIndexService is what allows large batch vector inserts and bulk-loaded index payloads to pass over gRPC.

VectorIndexService runs without the auth interceptor

In the default server, VectorIndexService is the one service registered without the auth interceptor (every other service uses .with_interceptor(auth)). If you expose the server beyond loopback, secure the network path accordingly. See gRPC Server for auth, TLS, and RBAC.

Batch vector inserts are zero-copy on the build side: build-index feeds contiguous f32 slices straight from the mmapped file into insert_batch_contiguous, and the Python native HnswIndex.insert_batch likewise requires C-contiguous float32[N, D] arrays for a zero-copy path.

Integration Workflow

The typical workflow for a production deployment:

1. Data Science Team: generates embeddings and saves them as .npy.
2. DevOps Pipeline: runs sochdb-bulk build-index to generate the .hnsw artifact (optionally sochdb-bulk query to validate recall).
3. Deployment: copies the .hnsw file to the production server.
4. Runtime: sochdb-grpc-server loads the pre-built .hnsw file via mmap, served through VectorIndexService (64 MB message limit).