TOON Format

TOON (Tabular Object-Oriented Notation) is SochDB's native, schema-aware data format. It is compact and optimized for LLM consumption — "the native format for SochDB, like JSON is for MongoDB."

TOON is the native query-result / output format: SochQL results, the MCP tools (sochdb_query, sochdb_context_query), and the context-query / token-budget pipeline all emit TOON by default.

Token efficiency

TOON typically uses 40-66% fewer tokens than equivalent JSON for tabular data. This is a measured range across real workloads, not a code-level guarantee — the savings depend on field names, value sizes, and the tokenizer.

---

Format

The text format is a one-line header followed by rows:

name[count]{field1,field2,...}:
row1
row2
...

• name — table / collection name (identifier)
• [count] — number of rows
• {field1,field2,...} — comma-separated field names
• : — terminates the header; rows follow

The header is parsed in the core engine (SochTable::parse_header in sochdb-core/src/soch.rs) by locating [, ], {, }, and the trailing :. The core SochTable::format() emits the header and then one row per line, separated by newlines.

Row separator differs by layer

The core engine (SochTable::format()) writes one row per line (newline-separated). The Python SDK helper (Database.to_toon, see below) packs the rows onto a single line, separated by ;. Both are valid TOON; the header grammar is identical. The examples below note which convention they use.

Examples

Simple table (core / newline rows):

users[3]{id,name,email}:
1,Alice,alice@example.com
2,Bob,bob@example.com
3,Charlie,charlie@example.com

With explicit field types — append :type to a field name:

orders[2]{id:uint,amount:float,status:text}:
1001,99.99,pending
1002,149.50,shipped

Python SDK helper (;-separated rows on one line):

users[2]{name,email}:Alice,alice@ex.com;Bob,bob@ex.com

---

Type system

Field types are optional. When omitted, fields default to text. The core SochType enum (sochdb-core/src/soch.rs) and its parse function recognize the following type strings:

Type string	SochType variant	Notes
null	Null	Null value
bool	Bool	Boolean
int / i64	Int	Signed 64-bit integer
uint / u64	UInt	Unsigned 64-bit integer
float / f64	Float	64-bit floating point
text / string	Text	UTF-8 string (default)
binary / bytes	Binary	Binary data
array(T)	Array(T)	Homogeneous array of T
ref(table)	Ref(table)	Reference to another table
T?	Optional(T)	Nullable T

Composite forms compose: array(int), int?, ref(users). There is no dedicated vector type tag in SochType — vectors are represented as arrays of floats.

Optional / nullable fields

Append ? to make a field nullable:

users[2]{id:uint,email:text,phone:text?}:
1,alice@ex.com,555-0100
2,bob@ex.com,null

---

Codec API (Rust)

The core text and binary codecs live in sochdb-core/src/soch_codec.rs. The text codec wraps the official toon-format crate; the binary codec is hand-rolled.

use sochdb_core::soch::SochTable;
use sochdb_core::soch_codec::{
    SochDocument, SochTextEncoder, SochTextParser, SochDbBinaryCodec,
};

// --- Simple table helper (soch.rs) ---
let text = "users[2]{id,name}:\n1,Alice\n2,Bob";
let table = SochTable::parse(text)?;   // -> SochTable
let out = table.format();              // -> String (TOON text)

// --- Document codec (soch_codec.rs) ---
let doc: SochDocument = SochTextParser::parse(text)?;
let text_out: String = SochTextEncoder::encode(&doc);

// --- Binary codec ---
let bytes: Vec<u8> = SochDbBinaryCodec::encode(&doc);
let doc2: SochDocument = SochDbBinaryCodec::decode(&bytes)?;

note

There is no SochCodec facade type. Use SochTable::format() / SochTable::parse() for the simple table representation, or SochTextEncoder / SochTextParser / SochDbBinaryCodec for the SochDocument representation.

---

Binary format

For internal storage and high-performance scenarios, SochDbBinaryCodec encodes a SochDocument to bytes. The layout is not a fixed-size header — it is the magic bytes, a varint schema version, then the value tree:

[ "TOON" magic : 4 bytes ] [ schema version : varint ] [ value tree ... ]

• Magic: TOON_MAGIC = [0x54, 0x4F, 0x4F, 0x4E] (ASCII "TOON").
• Schema version: a varint (current version is 1).
• Value tree: type-tagged values; integers, string lengths, array/object/ binary lengths, and ref ids are all varint-encoded.

Type tags (SochTypeTag, #[repr(u8)])

enum SochTypeTag {
    Null      = 0x00,
    False     = 0x01,
    True      = 0x02,
    PosFixint = 0x10,  // 0-15 embedded in 0x10-0x1F
    NegFixint = 0x20,  // -16 to -1 embedded in 0x20-0x2F
    Int8      = 0x30,
    Int16     = 0x31,
    Int32     = 0x32,
    Int64     = 0x33,
    Float32   = 0x40,
    Float64   = 0x41,
    FixStr    = 0x50,  // 0-15 length embedded in lower nibble
    Str8      = 0x60,  // 1-byte varint length prefix
    Str16     = 0x61,
    Str32     = 0x62,
    Array     = 0x70,
    Ref       = 0x80,
    Object    = 0x90,
    Binary    = 0xA0,
    UInt      = 0xB0,  // unsigned integer (varint)
}

Varint encoding

Lengths and large integers use LEB128-style variable-length encoding (write_varint / read_varint): 7 payload bits per byte, high bit as the continuation flag.

Value range	Bytes
0 – 127	1
128 – 16383	2
16384 – 2097151	3
...	...

---

Using TOON from the SDKs

Python (SDK 0.5.9)

The pure-Python SDK (pip install sochdb) exposes Database.to_toon / Database.from_toon static helpers, and selects the wire/context format via the WireFormat and ContextFormat enums. These helpers use the ;-separated-rows convention.

from sochdb import Database, WireFormat, ContextFormat

records = [
    {"id": 1, "name": "Alice", "email": "alice@ex.com"},
    {"id": 2, "name": "Bob",   "email": "bob@ex.com"},
]

# Encode to TOON (optionally project a subset of fields)
toon = Database.to_toon("users", records, ["name", "email"])
# -> "users[2]{name,email}:Alice,alice@ex.com;Bob,bob@ex.com"

# Parse TOON back to structured data
name, fields, parsed = Database.from_toon(toon)
# name == "users"; fields == ["name", "email"]
# parsed == [{"name": "Alice", "email": "alice@ex.com"}, ...]

# Select the output format for queries / context packaging
fmt = WireFormat.TOON        # default; also JSON, COLUMNAR
ctx = ContextFormat.TOON     # default; also JSON, MARKDOWN

WireFormat values are TOON (default), JSON, COLUMNAR. ContextFormat values are TOON (default), JSON, MARKDOWN.

Rust (core 2.0.3)

use sochdb_core::soch::SochTable;

let text = "users[2]{id,name}:\n1,Alice\n2,Bob";
let table = SochTable::parse(text)?;
for row in &table.rows {
    // access row fields
}
let toon: String = table.format();

---

Where TOON is used

• SochQL query results are returned in TOON (soch_ql.rs: "Query result in TOON format").
• MCP tools emit TOON by default. sochdb_query returns results in TOON, and sochdb_context_query defaults to TOON output (both also accept json and markdown via the format argument). MCP tool names use underscores.
• Context-query / token-budget paths in sochdb-query (context_query.rs, token_budget.rs, streaming_context.rs) assemble LLM context in TOON. The context-query OutputFormat is Soch (TOON, default), Json, or Markdown.
• Storage uses TOON for columnar emission via SochCursor ("Cursor for iterating over columnar data and emitting TOON format").

---

Best practices

For LLM context

1. Prefer TOON for tabular data in prompts — it is the default and the most token-efficient of the available formats.
2. Short field names reduce tokens further.
3. Omit types when they are inferable; they default to text.

For storage

1. Use the binary codec (SochDbBinaryCodec) for internal storage.
2. Use the text codec for debugging and logging.

Token optimization

# More tokens (explicit types)
users[2]{id:uint,name:text,email:text}:
1,Alice,alice@ex.com
2,Bob,bob@ex.com

# Fewer tokens (inferred types)
users[2]{id,name,email}:
1,Alice,alice@ex.com
2,Bob,bob@ex.com

# Even fewer (short names)
u[2]{i,n,e}:
1,Alice,alice@ex.com
2,Bob,bob@ex.com

---

See Also

• Architecture — internal format and engine details
• Performance — benchmarks and optimization notes