SochDB v2.0 Release Notes (2.0.0 → 2.0.3)

Engine version: 2.0.3  •  Release theme: Major Architecture Overhaul

v2.0 is the largest release in SochDB's history. It promotes the database from an embedded/IPC engine into a "thick server, thin client" architecture: a single sochdb-grpc-server binary now ships authentication, RBAC, TLS/mTLS, change-data-capture, Prometheus metrics, a WebSocket gateway, and a PostgreSQL wire endpoint. Under the hood, a Volcano-model SQL engine replaces the old single-table executor, and the HNSW vector index gained staged parallel construction plus higher-recall defaults.

This page summarizes what changed across the 2.0.0 → 2.0.3 line. For SDK-specific upgrade steps see the Python, Node.js, and Go guides.

Version & component matrix

The core engine and the language SDKs version independently in v2.0.

Component	Version	Install
Core engine (Rust workspace, server, MCP)	2.0.3	cargo add sochdb
Python SDK	0.5.9	pip install sochdb
Node.js SDK	0.5.3	npm install @sochdb/sochdb
Go SDK	0.4.5	go get github.com/sochdb/sochdb-go

---

License note (read this first)

SochDB v2.0 uses a split license. There has not been a full relicense to Apache.

• The core engine — the Rust workspace, the sochdb crate, the sochdb-grpc-server, and sochdb-mcp — is AGPL-3.0-or-later, with commercial licensing available for organizations that cannot adopt AGPL terms.
• The language SDKs — Python, Node.js, and Go — are Apache-2.0.

Plan your dependency graph accordingly

If you embed the AGPL core engine (for example, the in-process Rust crate, the embedded PyO3 native module, or a self-hosted sochdb-grpc-server you modify and distribute), AGPL obligations apply. If you only consume a SochDB server over the network through an Apache-2.0 SDK, the SDK's Apache-2.0 terms govern your client code. For commercial licensing of the core engine, contact the maintainers.

---

The thick server: sochdb-grpc-server

v2.0 consolidates production features into one binary. All configuration is via CLI flags and environment variables.

# Plaintext, loopback, all gateways on their default ports
sochdb-grpc-server --host 127.0.0.1 --port 50051

Default ports

Service	Flag	Default	0 disables
gRPC	-p, --port	50051	—
Prometheus metrics (HTTP /metrics)	--metrics-port	9090	✅
WebSocket gateway	--ws-port	8080	✅
PostgreSQL wire protocol	--pg-port	5433	✅

No --config file flag

The server is configured by flags and environment variables, not a config file. The make server-run target and sochdb-server-config.toml in the repo are stale — the binary's clap parser has no --config flag and will reject it. Use the flags documented here.

Authentication & RBAC

Authentication is off by default (every request resolves to an anonymous principal with Read + Write + ManageCollections). Pass --auth to require credentials.

sochdb-grpc-server --auth --api-key "my-secret-key"
# or supply the key via env
SOCHDB_API_KEY="my-secret-key" sochdb-grpc-server --auth

Clients present credentials in gRPC metadata:

• authorization: Bearer <token> — preferred. With auth enabled the bearer token is validated as a JWT.
• x-api-key: <key> — fallback; internally rewritten to Bearer <key>.

RBAC roles are Owner, Editor, Viewer (plus a Custom { name, capabilities } variant) — not Admin/ReadWrite/ReadOnly.

Role	Capabilities
Owner	Admin, Read, Write, ManageCollections, ManageIndexes, ViewMetrics, ManageBackups, ManageUsers
Editor	Read, Write, ManageCollections, ManageIndexes
Viewer	Read, ViewMetrics

Roles can be scoped globally, per-namespace, or per-collection via server-side role bindings, or carried in JWT claims.

JWT is validation-only (HS256)

The server validates JWTs (exp, optional iss/aud) and reads sub, tenant_id, role, and capabilities claims. There is no token-issuance API — tokens must be minted by your external IdP or caller and signed with the shared secret (SOCHDB_JWT_SECRET / the jwt-secret mounted secret).

Note a practical contradiction: with --auth, both JWT and API-key checking are enabled, but because JWT validation runs first, a bare --api-key is only reachable via the x-api-key path when JWT is disabled. For API-key-only deployments, plan accordingly.

Credential hashing (correcting a common misconception):

• API keys are stored as SHA-256(key) by default, or HMAC-SHA256(pepper, key) when SOCHDB_API_KEY_PEPPER is set. Keys are never stored in plaintext.
• argon2 is used only for user passwords (register_user / verify_password), not for API keys.

TLS & mTLS

sochdb-grpc-server \
  --tls-cert /etc/sochdb/tls/server.pem \
  --tls-key  /etc/sochdb/tls/server.key \
  --tls-ca   /etc/sochdb/tls/ca.pem      # adding --tls-ca enables mTLS client-cert verification

TLS is enabled when both --tls-cert and --tls-key are present. Adding --tls-ca turns on mutual TLS (client certificate verification). Certificates support hot reload.

Kubernetes secrets

--secrets-path (env SOCHDB_SECRETS_PATH) reads a mounted directory for jwt-secret, api-keys, encryption-key, tls-cert, tls-key, and tls-ca. Equivalent env overrides (SOCHDB_JWT_SECRET, SOCHDB_API_KEYS, SOCHDB_ENCRYPTION_KEY) are also supported, with periodic auto-refresh.

Change Data Capture & subscriptions

A WAL-derived, log-structured CDC ring buffer (default capacity 65,536 events) feeds a streaming SubscriptionService:

• Subscribe — stream SubscribeEvents, optionally filtered by table and operation type, resuming from a sequence number (start_sequence > 0) or from latest (0).
• WatchKey — stream changes for a single key.
• ListSubscriptions / CancelSubscription.

Operation types are INSERT, UPDATE, DELETE, and SCHEMA_CHANGE. Events carry sequence, timestamp_us, txn_id, table, key, and after_value.

where_predicate is accepted but not yet enforced

SubscribeRequest includes a where_predicate field, and table filtering and operation-type filtering are fully enforced. However, SQL WHERE-predicate filtering is not yet applied in the streaming loop — the field is accepted but ignored. Filter by table and operation for now.

CDC events are after-image only

The current implementation populates after_value; before images are not yet captured. Slow-subscriber WAL replay beyond the ring buffer is also not yet implemented — a subscriber that falls behind the buffer's capacity receives an overrun error and must resync.

Prometheus metrics

GET /metrics (Prometheus text format) and GET /health are served on --metrics-port (default 9090, bound 0.0.0.0). Exposed series include sochdb_grpc_requests_total{service,method}, sochdb_grpc_request_duration_seconds, sochdb_sql_queries_total{statement_type}, sochdb_transactions_total{outcome}, sochdb_storage_bytes, sochdb_wal_bytes, sochdb_uptime_seconds, and sochdb_build_info{version,rustc}. A separate gRPC health service is also mounted on the gRPC port (unauthenticated, for Kubernetes probes).

WebSocket gateway

A JSON message gateway on --ws-port (default 8080, path /) accepts message types sql, kv_get, kv_put, kv_delete, subscribe, and ping; replies are result, error, event, and pong.

{ "id": "1", "type": "sql", "payload": { "query": "SELECT 1", "params": [] } }

WebSocket subscriptions are not CDC-wired in the default binary

The gateway understands a subscribe message, but the default server starts it with an in-memory KV store and no CDC log attached, so live subscribe over WebSocket is not wired to CDC out of the box. Use the gRPC SubscriptionService for change streaming.

PostgreSQL wire protocol

--pg-port (default 5433) exposes a Postgres-compatible endpoint you can hit with psql:

psql -h 127.0.0.1 -p 5433 -d sochdb

With --pg-data-dir (env SOCHDB_PG_DATA_DIR) set, the endpoint runs real SQL (SELECT/INSERT/UPDATE/DELETE/DDL including JOINs) against a persistent on-disk database. Without it, a placeholder executor simply echoes queries.

pg-wire is loopback-only, simple-query, no auth

The Postgres endpoint speaks the simple query protocol only (no extended/prepared statements), has no SSL/TLS (cleartext), and uses trust auth (no password). Because a writable SQL database would otherwise be exposed unauthenticated, the server logs a loud warning if you bind a non-loopback --host. Treat it as a local development/tooling convenience, and provide --pg-data-dir for real SQL.

At-rest encryption (library API, not yet a server flag)

At-rest encryption is a library capability, not a runtime toggle

v2.0 ships an EncryptionEngine implementing AES-256-GCM-SIV (nonce-misuse-resistant AEAD) with a 32-byte key, designed to encrypt data blocks, WAL entries, and checkpoint files. The API is implemented and unit-tested, and a key can be supplied via the encryption-key secret or SOCHDB_ENCRYPTION_KEY. However, there is currently no CLI flag that wires it into sochdb-grpc-server's main path — treat it as an available API / planned server wiring, not a runtime toggle.

---

Volcano SQL engine

v2.0 replaces the previous single-table executor with a Volcano (row-at-a-time iterator) query engine. The production path runs through a storage-backed SqlBridge, with the Volcano executor providing the operator implementations.

Operators: SeqScan, IndexSeek, Filter, Project, Sort, Limit, HashJoin, NestedLoopJoin, MergeJoin, HashAggregate, Values, and Explain.

Aggregates & GROUP BY / HAVING

SELECT department, COUNT(*), AVG(salary), MAX(salary)
FROM employees
GROUP BY department
HAVING COUNT(*) > 5
ORDER BY AVG(salary) DESC;

The Volcano HashAggregate operator supports COUNT, COUNT(DISTINCT ...), SUM, AVG, MIN, and MAX with GROUP BY and HAVING.

MEDIAN and STDDEV live in a separate aggregate path

MEDIAN and STDDEV (sample, n-1, via Welford's online variance) are implemented in the dedicated SQL aggregate engine, not in the Volcano HashAggregate operator. If you depend on MEDIAN/STDDEV, route through that path. DISTINCT (as a SELECT DISTINCT clause) is not yet implemented in the planner.

Joins

The executor implements INNER, LEFT, RIGHT, FULL, and CROSS joins. The planner chooses a strategy from the join condition: equi-join ON a = b → hash join; non-equi ON → nested-loop join; USING(col) → hash join; merge join on sorted inputs. NATURAL JOIN currently falls back to a cross join rather than a true natural join.

SELECT u.name, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

Compatibility matrix may understate JOIN support

The repo's SQL compatibility matrix still marks multi-table joins as Partial/Planned; the executor and bridge in v2.0 actually implement all four outer/inner join types plus cross joins. The matrix is stale relative to the engine.

EXPLAIN

EXPLAIN SELECT * FROM users WHERE age > 30 ORDER BY name LIMIT 10;

EXPLAIN emits a textual plan tree (scan, filter, aggregate, project, sort, limit, and join nodes) under a QUERY PLAN column. It runs through the Volcano path.

SQL vector search

Vector search in SQL is expressed with VECTOR_SEARCH(column, query_vector, k, metric) where metric is one of COSINE, EUCLIDEAN, or DOT_PRODUCT. The Vector(dims) and Embedding(dims) data types are first-class column types.

Known SQL gaps

Not yet supported: SELECT DISTINCT, window functions, CTEs (WITH), subqueries in WHERE/SELECT, INTERSECT/EXCEPT, real NATURAL JOIN, and full CAST coercion (CAST currently passes the inner value through without type conversion). See SQL Compatibility for the full matrix.

---

HNSW: staged construction & higher recall

v2.0 reworks HNSW build and tuning for better out-of-the-box recall.

Staged parallel construction

Index building now uses a three-phase staged parallel construction (sequential scaffold → parallel waves with deferred back-edges → back-edge consolidation), followed by additive graph refinement. This delivers high recall without the memory blow-up of full exact rebuilds: a 1M-vector build that skips the exact layer-0 rebuild completes in ~195s at recall@10 ≈ 0.968.

New default configuration

Parameter	v2.0 default	Notes
max_connections (M)	32	raised from 16
max_connections_layer0 (M0)	64	standard 2·M
ef_construction	256	raised from 200 (helps hard high-dim embeddings)
ef_search	500	single fixed default
metric	Cosine
quantization_precision	F32	F16 / BF16 also available

With M = 32 the index "clears 95% recall out of the box": deep-1M recall@10 moves from 0.967 (M=16/M0=32) to 0.988 (M=32). ef_construction = 256 further helps hard real embeddings (e.g. Cohere) where higher M alone left recall near 0.90.

There is no dimension-aware ef_search split

ef_search is a single fixed default of 500 — there is no 500/1500 dimension-aware split in the core engine. The dimension-aware logic that does exist is the brute-force flat-scan threshold: for small indices the search uses an exact, SIMD-parallel flat scan when the vector count is at or below 10,000 (≤128 dims), 4,000 (≤384 dims), or 1,000 (higher dims). Above the threshold it uses the HNSW graph.

Quantization, distance metrics, search helpers

• Precisions: F32, F16, BF16; optional Product Quantization (off by default, intended for large/high-dim sets).
• Distance metrics: Cosine (default), Euclidean, DotProduct.
• search_exact for brute-force exact k-NN, and search_smart which auto-selects exact vs. approximate search by dataset size.
• An AdaptiveSearchConfig can binary-search the minimum ef that hits a target recall (default target 0.95).

Hybrid search (vector + BM25 + grep)

The unified fusion engine ranks three lanes — dense vector, BM25 lexical, and an indexed grep lane (trigram-prefiltered, regex-verified) — and combines them with Reciprocal Rank Fusion.

RRF-k is fixed in the core

In the core/Rust fusion engine, the RRF k constant is fixed at 60.0; there is no adaptive RRF-k. (The Python HybridSearchIndex does expose an adaptive_rrf_k=True parameter — adaptivity only applies in that Python context.)

---

Python: PyO3 native engine

Python ships two importable packages, both named sochdb:

1. The pure-Python ctypes SDK (v0.5.9) — the broad embedded + server SDK with Database, Namespace, Collection, Queue, AgentMemory, temporal-graph, semantic cache, and StudioClient. Prefer this for general usage.
2. The PyO3 native engine (v2.0.3) — a compiled extension exposing low-level index primitives: HnswIndex, BM25Index, RRFFusion, ThreeLaneHybridIndex, MultiShardHnswIndex, TableDatabase, the build_index* helpers, and recommended_hnsw_params.

Zero-copy batch ingestion + TableDatabase

The native engine supports zero-copy batch vector ingestion (passing NumPy arrays directly to Rust without per-vector FFI copies) and a TableDatabase for columnar table storage close to the engine.

import numpy as np
from sochdb import HnswIndex, recommended_hnsw_params  # PyO3 native engine (v2.0.3)

params = recommended_hnsw_params(dimension=768, n_vectors=100_000)
index = HnswIndex(
    dimension=768,
    m=params["m"],
    ef_construction=params["ef_construction"],
)

# Zero-copy batch insert from a contiguous NumPy array
vectors = np.random.rand(10_000, 768).astype(np.float32)
ids = np.arange(10_000, dtype=np.uint64)
index.insert_batch_with_ids(ids, vectors)

results = index.search(vectors[0], k=10)

from sochdb import Database  # pure-Python ctypes SDK (v0.5.9) — general usage

db = Database.open("file://./data")
db.put("users/alice", {"name": "Alice", "team": "search"})
print(db.get("users/alice"))

MultiShardHnswIndex is Python-only

MultiShardHnswIndex is a threaded scatter-gather wrapper that exists only in the Python native package. It is not a core-engine type and is not exposed by the server — do not treat it as a server/core feature.

Some Python "agent" patterns are examples, not SDK classes

Context-builder, policy-hooks, tool-routing, and graph-overlay appear as example patterns in the sochdb-python-examples repository — they are not importable classes in the Python SDK. (Several of these are real modules in the Rust and Node.js SDKs; see those guides.)

---

Node.js & Go notes

• Node.js (v0.5.3): commit() returns Promise<void>; EmbeddedDatabase.open() is synchronous; there is no routing module in the Node SDK.
• Go (v0.4.5): the SDK is remote-first by default. The embedded FFI engine is gated behind the sochdb_embedded build tag.

---

MCP server

The standalone sochdb-mcp server (stdio, JSON-RPC) exposes SochDB to LLM clients. The authoritative tool names use underscores: sochdb_query, sochdb_get, sochdb_put, sochdb_delete, sochdb_context_query, sochdb_list_tables, sochdb_describe, memory_search_episodes, memory_get_episode_timeline, sochdb_grep, sochdb_peek, and sochdb_expand, among others.

Dot-named tools in mcp.json/README are stale

The dot-named catalog (sochdb.query, memory.search_episodes, …) in the bundled mcp.json and README is illustrative/stale. The actual exposed tools are the underscore names above.

---

Deployment

• Docker: the sochdb-grpc-server binary ships as a slim Debian image; EXPOSE 50051, health-checked with grpc_health_probe.
• Helm: the chart (appVersion 2.0.3, license AGPL-3.0-or-later) deploys a StatefulSet with a long startup probe (up to ~10 min) to tolerate WAL replay/migrations during boot, TCP readiness/liveness on 50051, group-commit durability, and optional Prometheus ServiceMonitor wiring.
• Unified connect() URIs (per SDK): file://./data (embedded), ipc:///tmp/sochdb.sock (local IPC), grpc://host:50051 (gRPC), grpcs://host:443 (gRPC over TLS).

---

Upgrade summary

• License: core engine is AGPL-3.0-or-later (commercial licensing available); SDKs are Apache-2.0. Audit your dependency graph.
• Server: new sochdb-grpc-server with --auth, --tls-*, --metrics-port, --ws-port, and --pg-port. Configure via flags/env, not a config file.
• RBAC: roles are Owner / Editor / Viewer. JWTs are validation-only (HS256); mint them externally.
• SQL: new Volcano engine with aggregates, joins, and EXPLAIN. MEDIAN/STDDEV come from a separate aggregate path; DISTINCT, window functions, and CTEs are not yet supported.
• HNSW: new defaults (M=32, M0=64, ef_construction=256, ef_search=500). Higher recall; rebuild indices to benefit from the new build quality.
• Python: prefer the v0.5.9 ctypes SDK for general use; use the v2.0.3 PyO3 native engine for low-level index primitives and zero-copy batch ingestion.