SochDB Bulk Operations

High-throughput paths for loading large datasets into SochDB — vector indexes and relational tables — while avoiding per-call FFI overhead.

Versions on this page

Examples use the native Python engine sochdb 2.0.3 (PyO3), the pure-Python sochdb SDK 0.5.9 (ctypes), the Node.js SDK @sochdb/sochdb 0.5.3, and the sochdb-bulk CLI from core engine 2.0.3.

Deep Dive: See the Bulk Operations Reference for CLI internals and advanced usage.

Why Use Bulk Operations?

Inserting vectors one at a time crosses the Python/Rust (or Node/Rust) boundary for every call. The overhead comes from:

• An O(N·d) memcpy per call crossing the language boundary
• Host-language allocation tax (Python reference counting / GC pressure, V8 array boxing)
• GIL contention in multi-threaded scenarios (Python)

The bulk paths eliminate this by handing the engine one large, contiguous buffer per operation and releasing the GIL while the index builds:

1. In-process zero-copy — pass a single C-contiguous float32 NumPy array; the engine reads it directly with no copy.
2. Accumulate-then-flush — buffer many chunks in pre-allocated arrays, then build the graph in one call.
3. Out-of-process — for datasets too large for RAM, memory-map a file and build with the sochdb-bulk CLI.

Which Python package?

SochDB ships two importable packages both named sochdb:

• The native engine (sochdb 2.0.3, PyO3) exposes HnswIndex, build_index_from_numpy, build_index_from_file, TableDatabase, etc. — used for the zero-copy index-build examples below.
• The pure-Python SDK (sochdb 0.5.9, ctypes) exposes Database, Namespace, Collection, VectorIndex, and BatchAccumulator.

Each example notes which package its imports come from.

In-Process Index Build (Recommended)

The fastest way to build an HNSW index from in-memory embeddings is the native zero-copy path. It requires a 2D float32 array in C-contiguous (row-major) order.

build_index_from_numpy

# sochdb 2.0.3 (native PyO3 engine)
import numpy as np
from sochdb import build_index_from_numpy

# 10K x 768D embeddings, float32, C-contiguous
embeddings = np.random.randn(10000, 768).astype(np.float32)

# m / ef_construction default to recommended_hnsw_params(D) when omitted
index = build_index_from_numpy(embeddings, metric="cosine")

index.save("my_index.hnsw")
print(len(index), "vectors indexed")

build_index_from_numpy(embeddings, *, m=None, ef_construction=None, metric="cosine", ids=None) builds entirely in-process — no disk I/O, no subprocess. When m or ef_construction are left as None they are filled from recommended_hnsw_params(D) (dimension-aware: M=16 for D <= 128, M=24 for 129–512, M=32 for 513+; ef_construction = max(200, m*8)).

HnswIndex.insert_batch

# sochdb 2.0.3 (native PyO3 engine)
import numpy as np
from sochdb import HnswIndex

index = HnswIndex(dimension=768, m=32, ef_construction=200, metric="cosine")

# Must be C-contiguous float32, shape (N, 768)
vectors = np.ascontiguousarray(
    np.random.randn(10000, 768).astype(np.float32)
)

# Zero-copy: the engine reads the NumPy buffer directly, GIL released
n = index.insert_batch(vectors)            # auto-generates sequential IDs
print(f"Inserted {n} vectors")

# Or with explicit uint64 IDs:
ids = np.arange(10000, dtype=np.uint64)
index.insert_batch_with_ids(ids, vectors)

insert_batch(vectors) and insert_batch_with_ids(ids, vectors) both require a C-contiguous float32 array. A non-contiguous array raises ValueError — call np.ascontiguousarray(vectors) first. insert_batch auto-generates sequential IDs; insert_batch_with_ids takes a 1D uint64 array whose length must equal the row count.

Contiguity is mandatory for the zero-copy path

Slicing, transposing, or stacking can produce a non-C-contiguous array. If insert_batch raises ValueError: Non-contiguous array, wrap the array: vectors = np.ascontiguousarray(vectors, dtype=np.float32).

Accumulate-then-flush with BatchAccumulator

When embeddings arrive in chunks (e.g. from a data loader), the BatchAccumulator from the pure-Python SDK (sochdb 0.5.9) collects them into pre-allocated buffers with zero FFI calls, then builds the index in a single flush().

# sochdb 0.5.9 (pure-Python ctypes SDK)
from sochdb.vector import VectorIndex, BatchAccumulator

index = VectorIndex(dimension=1536)            # m=32, ef_construction=256 by default

acc = BatchAccumulator(index, estimated_size=50_000)
for batch_ids, batch_vecs in data_loader:      # uint64[], float32[N, 1536]
    acc.add(batch_ids, batch_vecs)             # pure numpy memcpy, no FFI
acc.flush()                                    # single bulk insert_batch() FFI call

print(len(index), "vectors indexed")

BatchAccumulator(index, *, estimated_size=0) appends chunks via acc.add(ids, vectors) (a uint64 ID array plus a 2D float32 array) or acc.add_single(id, vector) into geometrically-grown buffers. acc.flush() performs the single bulk insert_batch() FFI call and returns the inserted count; acc.count(), acc.save(dir), and acc.load(dir) are also available. The accumulator reports a typical 4–5x speedup over incremental single inserts.

VectorIndex itself also exposes insert_batch(ids, vectors) -> int for one-shot insertion when all data is in a single array.

Out-of-Process Build (Large / Out-of-Memory Datasets)

For datasets that do not fit in RAM, build from a file. The native engine spawns the sochdb-bulk CLI, which memory-maps the input.

# sochdb 2.0.3 (native PyO3 engine)
from sochdb import build_index_from_file

stats = build_index_from_file(
    "embeddings.npy",          # .npy or raw .f32
    "index.hnsw",
    dimension=768,             # auto-detected for .npy
    m=16,
    ef_construction=100,
    batch_size=1000,
)
print(stats)                   # {"vectors", "elapsed_secs", "rate", "output_size_mb", "command"}

build_index_from_file(input_path, output_path, *, dimension=None, m=16, ef_construction=100, batch_size=1000, quiet=False) returns a dict of build statistics. It requires the sochdb-bulk binary on PATH or in the workspace target/ directory.

Prefer in-process for in-memory data

build_index_from_numpy() is significantly faster than the subprocess path when your embeddings already fit in memory, because it avoids serializing to disk and spawning a process. Only reach for build_index_from_file() when the dataset is too large for RAM.

bulk_build_index() is deprecated

The legacy bulk_build_index(embeddings, output, *, ids=None, m=16, ef_construction=100) still exists in the native package but emits a DeprecationWarning. It now delegates to the in-process build and saves the index. Use build_index_from_numpy() instead.

Bulk Table Loading with TableDatabase.load_csv

The native engine's columnar TableDatabase (sochdb 2.0.3) can bulk-load a CSV directly in Rust, releasing the GIL during the insert.

# sochdb 2.0.3 (native PyO3 engine)
from sochdb import TableDatabase

db = TableDatabase.open("data.sdb")

# Register the schema first; column types: int64, uint64, float64, text, binary, bool
db.register_table("events", [
    ("user_id", "uint64"),
    ("score", "float64"),
    ("label", "text"),
])

# CSV must have a header row whose columns match the registered schema, in order
rows = db.load_csv("events", "events.csv")
print(f"Loaded {rows} rows")

load_csv(table, csv_path) -> int reads and converts the CSV entirely in Rust, parsing each field according to the registered column type, and commits in batches (every 100K rows). The table must already be registered with register_table(...). Returns the number of rows inserted.

TableDatabase is native-only

TableDatabase lives only in the native sochdb 2.0.3 package. It is a builder/columnar store with no SQL execute method. The pure-Python 0.5.9 SDK does not have a TableDatabase class — for SQL-style tables there, use db.execute(...) against the KV-backed SQL subset.

Node.js Batch Insert

The Node.js SDK (@sochdb/sochdb 0.5.3) batches vectors through a single FFI call. Note that EmbeddedDatabase.open() is synchronous.

HnswIndex.insertBatch

import { HnswIndex } from '@sochdb/sochdb';

const index = new HnswIndex({ dimension: 768 }); // maxConnections=32, efConstruction=256

const ids = ['a', 'b', 'c'];
const vectors: number[][] = [
  /* 768-length number[] per row */
];

// Single FFI call; vectors are flattened into a contiguous Float32Array internally
index.insertBatch(ids, vectors);

console.log(index.length, 'vectors indexed');

Collection.insertMany

import { open } from '@sochdb/sochdb';

const db = open('data.sdb'); // synchronous

const ns = db.getOrCreateNamespace('default');
const col = ns.getOrCreateCollection({ name: 'docs', dimension: 768 });

const vectors: number[][] = [/* one number[768] per item */];
const metadatas = [{ title: 'a' }, { title: 'b' }];

// Uses the native HNSW batch fast path under the hood
const ids: string[] = await col.insertMany(vectors, metadatas);
console.log(`Inserted ${ids.length} vectors`);

String IDs are hashed internally

HnswIndex accepts string IDs but hashes them to a numeric key; the reverse mapping returns only the low 64 bits, so a string round-trip can be lossy. Use Collection/insertMany (which maintains its own string-to-numeric map) when you need to recover original string IDs.

sochdb-bulk CLI

The sochdb-bulk binary builds and queries indexes from files. It ships with the core engine (the sochdb-tools crate) and is also bundled in the Python wheel.

# Build from raw f32 file
sochdb-bulk build-index \
    --input embeddings.bin \
    --output index.hnsw \
    --dimension 768

# Build from NumPy .npy file (dimension auto-detected)
sochdb-bulk build-index \
    --input embeddings.npy \
    --output index.hnsw

# With custom HNSW parameters
sochdb-bulk build-index \
    --input data.f32 \
    --output index.hnsw \
    --dimension 768 \
    --max-connections 32 \
    --ef-construction 200 \
    --threads 8

# Query an index
sochdb-bulk query \
    --index index.hnsw \
    --query query.f32 \
    --k 10

# Get index info
sochdb-bulk info --index index.hnsw

build-index

sochdb-bulk build-index [OPTIONS] --input <FILE> --output <FILE>

Options:
  -i, --input <FILE>         Input vector file (raw f32 or .npy)
  -o, --output <FILE>        Output index file
  -d, --dimension <DIM>      Vector dimension (auto-detected for .npy)
  -f, --format <FORMAT>      Input format: raw_f32, npy (auto-detected)
      --ids <FILE>           Optional ID file (raw u64)
  -m, --max-connections <N>  HNSW M parameter [default: 16]
  -e, --ef-construction <N>  HNSW ef_construction [default: 100]
      --batch-size <N>       Batch size for insertion [default: 1000]
  -t, --threads <N>          Number of threads (0 = auto) [default: 0]
      --direct-read          Use direct read instead of mmap
      --prefault             Prefault mmap pages for memory residency
      --quiet                Suppress progress bar
  -v, --verbose              Enable verbose logging

query

sochdb-bulk query [OPTIONS] --index <FILE> --query <FILE>

Options:
  -i, --index <FILE>   Index file
  -q, --query <FILE>   Query vector file (single vector, raw f32)
  -k, --k <N>          Number of neighbors [default: 10]
  -e, --ef <N>         Search ef parameter

info

sochdb-bulk info --index <FILE>

Input Formats

Raw float32 (Recommended)

The simplest and fastest format — raw bytes, memory-mappable.

File layout:

• vectors.f32 — N × D × 4 bytes of row-major float32 data
• vectors.json (optional) — metadata: {"n": 10000, "dim": 768, "metric": "cosine"}
• ids.u64 (optional) — N × 8 bytes of uint64 IDs

Creating raw f32 from Python:

import numpy as np

embeddings = np.load("embeddings.npy").astype(np.float32)
# Row-major float32 bytes; ensure C-contiguous first
np.ascontiguousarray(embeddings).tofile("embeddings.f32")

NumPy .npy

Standard NumPy format, auto-detected from the extension.

Requirements:

• dtype: float32 (<f4)
• order: C-order (fortran_order: False)
• shape: 2D (N, D)

import numpy as np
embeddings = np.random.randn(10000, 768).astype(np.float32)
np.save("embeddings.npy", embeddings)

Building the CLI from Source

# Build release binary
cargo build --release -p sochdb-tools

# Binary location
./target/release/sochdb-bulk --help

# Run benchmarks
cargo bench -p sochdb-tools

# Install to PATH
cargo install --path sochdb-tools

Bundling with the Python Package

The Python wheel can bundle the native binary:

cd sochdb-python-sdk

# Build and install binary for current platform
python build_native.py

# Then build the wheel
pip wheel .

The binary is installed to src/sochdb/_bin/<platform>/sochdb-bulk.

Performance Tips

1. Pass C-contiguous float32 — this is what enables the zero-copy insert_batch path; non-contiguous arrays are rejected.
2. Prefer build_index_from_numpy() for in-memory data — it avoids disk I/O and subprocess spawn.
3. Use BatchAccumulator for streamed chunks — buffer in numpy, build once.
4. Use raw f32 for out-of-core builds — fastest to parse and memory-mappable.
5. Use all CPU cores — set --threads 0 (CLI) for auto-detection; the in-process build releases the GIL and uses Rayon.
6. Pre-normalize vectors if using cosine similarity, and store on NVMe for very large indexes.

Troubleshooting

"Could not find sochdb-bulk binary"

The file-based build paths (build_index_from_file, the sochdb-bulk CLI) require the binary. The SDK searches, in order:

1. System PATH:

   cargo install --path sochdb-tools
   # Or: export PATH="$PATH:/path/to/target/release"

2. Cargo target directory (development):

   cargo build --release -p sochdb-tools
   # Auto-detected when running from the workspace

3. Bundled in the wheel:

   pip install sochdb
   # Binary at: site-packages/sochdb/_bin/<platform>/sochdb-bulk

If you only need in-memory builds, use build_index_from_numpy() — it needs no external binary.

Platform Support

The bundled binary supports:

Platform	Wheel Tag	Notes
Linux x86_64	manylinux_2_17_x86_64	glibc >= 2.17
Linux aarch64	manylinux_2_17_aarch64	ARM servers
macOS	macosx_11_0_universal2	Intel + Apple Silicon
Windows	win_amd64	Windows 10+ x64

"Dimension mismatch"

• For the in-process path, the array's second axis must equal the index dimension — insert_batch raises ValueError otherwise.
• For raw f32 CLI input: pass -d 768 (or provide a sidecar meta.json).
• For .npy: the dimension is auto-detected from the header.

"Non-contiguous array"

insert_batch/insert_batch_with_ids require C-contiguous float32. Fix with:

import numpy as np
vectors = np.ascontiguousarray(vectors, dtype=np.float32)

"Out of memory"

For very large datasets (10M+ vectors):

• Use the out-of-process sochdb-bulk build with a smaller --batch-size and mmap input.
• Build per-shard indexes (see MultiShardHnswIndex in the native package, a Python threaded scatter-gather wrapper for 100M–1B scale).
• Use a 64-bit system with sufficient RAM.

"GLIBC_2.xx not found" (Linux)

Your system glibc is older than the wheel requires:

ldd --version  # Check glibc version; needs 2.17 or higher

Solutions:

1. Use a newer distro (Ubuntu 14.04+, CentOS 7+).
2. Use a container with newer glibc.
3. Build from source with your system's glibc.

Architecture

The in-process path passes a NumPy buffer straight into Rust; the out-of-process path memory-maps a file and lets the CLI build:

In-process (build_index_from_numpy / insert_batch)
┌──────────────────────┐
│ NumPy float32 (N, D) │  C-contiguous, no copy
└──────────┬───────────┘
           │ zero-copy pointer, GIL released
           ▼
┌──────────────────────┐
│ Rust HNSW build      │  Rayon-parallel insertion
└──────────────────────┘

Out-of-process (build_index_from_file / sochdb-bulk CLI)
┌──────────────────────┐
│ embeddings.npy/.f32  │
└──────────┬───────────┘
           │ subprocess.run()
           ▼
┌──────────────────────┐
│ sochdb-bulk CLI      │  mmap file, HNSW insertion, save
└──────────┬───────────┘
           ▼
┌──────────────────────┐
│ index.hnsw           │
└──────────────────────┘

The key insight: subprocess spawn is O(1), while per-call FFI marshalling is O(N·d) per batch. The in-process zero-copy path is fastest for in-memory data; the subprocess path wins when the dataset cannot fit in RAM.

See the Python SDK Guide and HNSW Vector Search for index configuration details.