Helix and LMDB.

The N=10,000 study is complete: five alternating trials per engine and mode, 1,500,000 timed calls, 630,000 timed writes with immediate full-value read-back, and 600,000 final records checked again after reopen. All checks passed. The base population was verified twice before timing as well.

Durable: measured on a shared desktop, pinned cores. Fast: preliminary. All durable trial medians were below 3. The first fast triple sampled one-minute load 3.08, above the director's median threshold; its trials remain in the table. The other fast trials passed. No trial was dropped or selectively rerun. The complete per-trial load table follows the measurements.

Large-N Helix is memory-blocked in this pinned version/workload. A bounded calibration and independent HTTP reproduction establish growth outside the in-process harness; see the evidence below. Its large-N comparison cells remain unmeasured. The three-trial Kiban/LMDB large-N subset completed with zero value errors; both modes are preliminary because later median loads exceeded 3. No Helix patch or substitute query supplies published numbers.

Interpretation of the small dataset

Kiban's durable remember/update p50p50The median: half of all operations were faster than this. medians were 524/557 µs, versus Helix's 2,425/1,245 µs and raw LMDBLMDBA widely used embedded database built on a B+tree and memory-mapped files, known for very fast reads.'s 1,278/1,278 µs. Kiban also had lower median p99p9999% of operations were faster than this. It shows how bad the slow cases get. write latency in this campaign. The configured durability guarantees match; the persistence operations differ.

Kiban appends a WALWALWrite-ahead log. Every change is appended here first, so it can be replayed after a crash. record and syncs that WAL. LMDB commits copy-on-write pages and synchronized metadata. The separate excluded sync diagnostic confirms these paths; this is more specific than attributing the difference to an LSM versus a B+tree in general.

Kiban loses to raw LMDB on recent-for-agent p50: 2.75 versus 2.11 µs in durable mode, with non-overlapping trial ranges. Point-read p50 is effectively tied at 0.80 µs, with overlapping ranges. Helix's recent p50 is 156 µs; its faithful query traverses and materializes that agent's memories, sorts by creation sequence, then takes K.

Both KV engines seek directly into a newest-first key range and decode K values. This structural work difference is visible in the query source, rather than inferred from HTTP latency.

In the preliminary fast results, Kiban's median insert/update p50 is lower than both competitors. Raw LMDB retains the recent-scan advantage. Point-read ranges overlap, and some Kiban maximum-latency observations are worse than LMDB's; the tail columns are retained without selecting favorable trials. Fast verified throughput includes generator/oracle work and is substantially lower than reciprocal API-call throughput. It is not an engine capacity test.

RSSRSSResident set size: the physical memory a process is actually using. counts mapped resident pages and the complete process, including bulk preload and oracle allocations; it is not a heap-only comparison. Kernel file cache outside a process is excluded. The Helix RSS result needs a version-specific allocation explanation before making memory-efficiency claims. The on-disk numbers include all database files after close. Allocated bytes are filesystem st_blocks * 512 accounting, not measured device writes.

Method and layers

Kiban engine source is unchanged from 361305a; implementation commit be1df72 adds this standalone harness. Helix is the requested v2.3.5, source 17e7ecf764aecd553e1f54ca25320d654153a9aa (internal crate version 1.3.3), with heed3 0.22.0 and lmdb-master3-sys 0.2.5. This is not Helix v3. All three run from one release binary using Helix's MiMalloc allocator and a common upstream Helix profile: opt-level 2, LTO and one codegen unit.

Binary SHA-256: d3901e63d7dd7b24529c5d2768762ae3d8ff40a6d7439a3d264d40d45e1c84ce.

The timed Helix entry uses public graph storage and the exact generated transaction/traversal bodies. Its generated handlers require Request/JSON, so only those wrappers are replaced by typed inputs and owned benchmark fields. There is no HTTP server, router, socket, JSON transport, task queue or async runtime.

Graph node/property encoding, UUIDs, required unique indexes, edge maintenance, per-query Bump arena, traversal/sort and transaction commit remain. The layer statement and generated-code equivalence include the original query excerpt and a machine-checkable source extraction.

Unused automatic BM25 indexing is explicitly disabled; W1–W5 do not query full-text search. Schema, required indexes and query bodies are preserved. The old HTTP study kept BM25 enabled and used different compiler/allocator settings, so its measurements are not pooled or subtracted here. Kiban and raw LMDB share the same ordered primary key and JSON persistent-value codec. That JSON is storage encoding, not transport.

Helix exposes its environment through the public storage API; enabling fast flags does not patch Helix or claim a corresponding server configuration switch. Both LMDB environments use a 10 GiB map, 200 database slots and 200 reader slots; WRITEMAP and MAP_ASYNC are absent. Synchronous LMDB commit flushes dirty data and writes metadata through an O_DSYNC descriptor.

This is per-call persistence, not one identical fsyncfsyncForces a file and its metadata onto the disk. syscall per engine. Explicit final sync/flush is outside foreground metrics. Fast reopen checks prove orderly persistence after that final sync; no power-loss durability claim is made.

Each process warms a separate discarded 100-record database. It then bulk-loads 10,000 base memories untimed (Kiban WriteBatch chunks; Helix/LMDB one transaction, relaxed flags, explicit final sync), verifies every base value, closes/reopens, and verifies the base again. Timed W1 inserts fresh IDs 10,000–19,999. W2/W3/W5 use 10,000 calls; W4 overwrites 10,000 records.

Final population is 20,000. All values, ordered lists and final/reopened state are checked against the same seed 20261016, 64 agents, K=10 and 200-byte text plus metadata. Warmup, bulk load, base verification, final full verification and reopen are excluded from the foreground latency/throughput metrics. The raw resource files retain their times.

The CPU Ultra 9 185H exposes P-core CPUs 0–11. Every engine process is pinned to CPUs 1–4, two physical P-cores including their SMT siblings; the driver uses E/LP-E CPUs 12–21. Actual process affinity is checked. Host: 22 logical CPUs, 30.8 GiB RAM, Fedora 44 / Linux 7.2.10-200.fc44.x86_64, WD SN740 NVMe with LUKS/Btrfs/zstd:1, Rust/Cargo 1.98.1.

Other desktop processes share the host; cores are not exclusively reserved and frequency/governor are not fixed.

The director's gate is each trial's median one-minute load below 3; maxima are reported. Driver samples every 200 ms, with actual engine-side boundary samples for short fast trials. Timing boundaries exclude preload/warmup; the load interval includes final in-process verification. Gaps above 2.5 seconds fail coverage. Readback/command/count/flag/affinity errors invalidate a run.

Small N uses five alternating repetitions of Kiban, Helix, LMDB for each mode; large N uses three alternating Kiban/LMDB trials, with Helix excluded as documented below. Durable W1/W4 are capped at 10,000 calls for large N; the 100,000-call mixed workload contains 10,000 writes. No million-record per-write-fsync preload is performed.

Reproduction and exact layer/method details. Compact raw evidence contains every stage/config/resource/reopen JSON, commands, source/binary hashes and byte-exact gzip load log. Full databases, source snapshots and logs remain under ignored target/enginebench-results/small-10000. engine_audit.py independently rechecks counts, flags, affinities, persistence markers, source/binary hashes and load classification; it passed with zero errors. The earlier HTTP evidence is retained separately.

Validation passed: five engine Rust tests (including all engines/modes, bulk preload/reopen and deliberate corruption rejection), four original harness Rust tests, thirteen Python harness checks, main cargo test (320 unit + 11 integration; five stress tests ignored), format/Clippy, core cargo deny and site-facts validation. No core crate dependency or source changed.

Excluded sync diagnostic

Same binary; N=10,000 untimed base, 1,000 calls per measured workload (2,100 timed writes), plus warmup, bulk initialization, verification and close. The C interposer observes libc fsync/fdatasync and pwrite/pwrite64; it is not a full syscall or CPU stack tracer. These instrumented runs are not comparison trials.

All observed sync calls succeeded. Remaining fast-mode calls include startup, final sync and Kiban maintenance; the counts are whole-process observations.

N=10,000, durable

Measured on a shared desktop, pinned cores. Untimed verified/reopened base: 10,000 records. W1 inserts 10,000 fresh records; W2/W3/W5: 10,000 calls each; W4: 10,000 overwrites. Final population: 20,000. 5 alternating trials per engine.

N=10,000, fast

Load during every trial

Sampled every 200 ms plus actual engine-side boundary samples; median [minimum, maximum]. Warmup is excluded using engine-recorded timestamps; final sync and full in-process verification are included. Engine open is excluded. The separate reopen process is excluded. All three load windows are retained.

RSS and disk after every small-N run

Helix large-N exclusion and memory evidence

This exclusion is specific to Helix v2.3.5 / 17e7ecf764aecd553e1f54ca25320d654153a9aa, this schema and this W3 query. The small-N measurements above are retained unchanged. The large calibration loaded 1,000,000 base records, inserted 1,000 fresh records, and ran 1,000 calls per workload with fast flags.

All five foreground stage markers passed value checks, but the 6 GiB guard stopped the process during final verification, so there is no complete/reopen marker and no valid comparison trial.

Its maximum observed RSS was 6,443,012,096 bytes. Calibration W3 took 16.30 seconds for 1,000 requests, projecting roughly 163 minutes for W3 alone across six 100,000-call trials. That projection is not a measured benchmark cell.

The owner's requested harness check took under ten minutes. The direct adapter's recent function drops its local traversal Vec and Bump arena when returning K owned records. The common runner consumes/drops the returned many Vec inside each operation; only fixed-size oracles/histograms persist, and logs contain one row per stage. There is no accumulating response Vec or record log.

A separate Python standard-library client then sent 1,000 POST /Recent requests to an independent pinned Helix HTTP process opening a copy of the 1,001,000-record database. Requests used {"agent": q % 64, "k": 10}; all ten IDs, agent/creation metadata, order, revision type and text size were checked before each response was discarded.

Server and client were pinned to CPUs 1–4 and 12–21 respectively. The client streams one RSS row per request.

The database file was about 673 MiB, so continued multi-GiB growth after repeated visits to the same agents cannot be explained solely by making that file resident. HTTP reproduction one-minute load was 2.46 [2.14, 2.58], sampled with every request; raw values are retained.

These RSS observations are diagnostic evidence, not HTTP latency cells pooled with the engine study. They establish retention in Helix's independent server path, excluding the in-process harness's result handling as its cause.

The stock server's BM25-enabled configuration first attempted to build an unused full-text index and reached MDB_MAP_FULL at its 10 GiB map before serving requests. The successful reproduction changed only generated application config bm25: Some(true) to Some(false), matching the engine study. It rebuilt a separate HTTP executable, whose SHA-256 is 2380637190af2e3d536b6bbe26537665ebe094b2b608db52a6527bdb1cdc284a.

All pinned engine sources were verified unchanged; the HTTP W3 storage/traversal core is byte-identical to the frozen adapter. The installed original HTTP executable/config were restored. The comparison binary was never rebuilt. The config delta, actual generated HTTP source, version/commit, server logs, query-count/RSS curve and hashes are archived separately.

Exact HelixQL:

QUERY Recent(agent: U32, k: U32) =>
    memories <- N<Agent>({agent_num: agent})::Out<Remembers>::ORDER<Desc>(_::{created_at})::RANGE(0, k)
    RETURN memories

The likely mechanism is visible in pinned ImmutablePropertiesMap: its Copy map stores raw arena-allocated Value pointers, populated by values.add(index).write(value), with no Drop implementation. Value::String owns a String; bump arena teardown does not run those destructors. W3 decodes roughly 15,640 neighbors per query before sorting/taking ten.

The HTTP curve and source support this explanation; no heap-profiler claim assigns every RSS byte to that mechanism. Mapped pages and allocator overhead also contribute. This is a version-specific allocation issue, not an inherent graph storage cost.

The minimal standalone client and setup instructions are suitable for an upstream reproduction. Helix is memory-blocked in both large-N mode tables: only fast mode was calibrated, but the read-only W3 allocation path is shared with durable mode. No durable large-N Helix trial is implied.

Large-N Kiban/LMDB results

All 12 large-N subset trials completed: 4,920,000 foreground calls, 1,440,000 timed writes with immediate full-value read-back, 24,000,000 base-value checks before timing, and 12,660,000 final records checked after a new-process reopen. The recorded campaign interval was about 5.7 minutes, including bulk loads and reopen processes.

The frozen executable/hash is identical to small N; no benchmark rebuild occurred. Durable W1/W4 each use 10,000 timed writes; W2/W3/W5 use 100,000 calls. Fast uses 100,000 for all five workloads. The different final populations are explicit beside each table.

Both modes are preliminary, measured on a shared desktop with pinned cores. First-cycle medians passed; cycles two and three did not. Every trial is retained, with median one-minute load 2.53–4.07 and maximum 4.07. Sampling coverage passed for all trials.

CPU sets, seed, warmup and verification rules match small N. Helix has no valid large-N trial; its unmeasured cells below are memory-blocked, not a latency extrapolation or a substitute storage path.

Kiban loses the large-N point-read comparison to raw LMDB: durable p50 medians are 5.89 versus 1.31 µs, and fast 5.63 versus 1.38 µs, with non-overlapping trial ranges. LMDB also has the lower recent-list p50 and better fast mixed verified throughput (196,638 versus 160,223 ops/s).

With this resident dataset, LMDB's mmap reads avoid Kiban's SST/block-cache lookup and decoding path; the larger dataset exceeds Kiban's default 32 MiB block cacheblock cacheRecently read file blocks kept in memory, so they need no disk read.. That is a plausible structural explanation, not a CPU-profile attribution or a cold-cache result.

Kiban has lower remember/update p50 in both modes here. Durable medians are 541/541 µs versus LMDB's 1,180/1,311 µs. Kiban synchronizes an appended WAL record, while LMDB's single-writer copy-on-write commit persists modified pages and metadata. Fast mode removes per-write durability from both: Kiban defers flush/compaction work and LMDB still performs a transaction commit/page update.

These are foreground acknowledgment latencies, not sustained compaction-drained write capacity. The tables retain max latency and throughput: durable mixed p50 favors LMDB's majority reads while total verified throughput favors Kiban's less costly writes. Fast mixed throughput favors LMDB.

Kiban's lower end RSS does not establish lower peak memory. Whole-process peak RSS during preload/work was roughly 1,614 MiB for Kiban versus 1,329 MiB for LMDB, including the bulk path, oracle, allocator and mapped pages. Fast Kiban end RSS varied from 270 to 955 MiB; that full range is retained.

LMDB files were larger after these operations, but neither engine was force-compacted or vacuumed to a common minimum-size state. No general engine winner is claimed.

Compact large-N evidence contains all stage/config/resource/reopen JSON, command/source/binary hashes, compressed load samples and an independent zero-error audit. Full source snapshots/databases/logs remain at target/enginebench-results/large-1000000-kv.

N=1,000,000, durable

N=1,000,000, fast

Load during every trial

Sampled every 200 ms plus actual engine-side boundary samples; median [minimum, maximum]. Warmup is excluded using engine-recorded timestamps; final sync and full in-process verification are included. Engine open is excluded. The separate reopen process is excluded. All three load windows are retained.

RSS and disk after every large-N run

Large-N read diagnosis: cache budget

Cache budget explains most of the observed point-read gap, with a residual that needs profiling. The contemporaneous W2 p50 gap narrowed from 4.35 µs (32 MiB) to 0.70 µs (512 MiB): about 84% closure (80.6–85.9% when computed within each alternating cycle). At 1 GiB the residual was 0.58 µs, about 87% closure. These are differences between p50 medians, not an additive breakdown of read work.

This is read-only and uses fast mode: reads pay no per-call fsync. W5 retains its stage-4 key sequence with writes converted to reads, explicitly mixed-key-reads-only, rather than 90/10. The original bulk-load/sync/verify/reopen/verify method warms caches; every revision stays zero and no W1 inserts expand the population. Compare against this sweep’s control/reference, rather than subtracting the older post-update measurements.

Cells are median [minimum, maximum] across three trials. RSS is the whole direct benchmark process after reads.

Every Kiban preload produced one L0 SST and no other levels, unchanged throughout. The separate replay’s resident cache was 32.00/256.00/309.14/309.14 MiB. Direct Kiban exposes no public cache counters, so the hit rates above are proxy counters from an excluded SharedKiban replay: fresh cache, identical files, full verification warmup, identical keys, all values checked.

Its time/RSS is excluded. W3 already had 99.94% replay hits at 32 MiB; larger caches did not improve its latency.

The block working set fits at 512 MiB, yet point reads remain slower than LMDB with lower end RSS. More capacity alone does not account for the remaining gap. Profile SSTSSTSorted string table: an immutable file of sorted keys and values. Once written, it never changes. filter/index/restart-block lookup, cache locking and value materialization next. Source shows Kiban copies a found value into a Vec before JSON decoding; LMDB decodes directly from its mmap-backed slice.

Both return the same owned fields. No profile here assigns the residual to those candidates. Median peak RSS includes preload: about 1,614–1,617 MiB for Kiban versus 1,329 MiB for LMDB.

One new diagnostic executable (430dd88d…) served all configurations; the original frozen binary (d3901e63…) remains unchanged. No core source/dependency changed. All 4.5 million timed calls and 15 million records checked after independent reopen passed, with zero timed writes.

Full percentiles, max, throughput, RSS/peak/disk, counters and raw evidence; one-command reproduction and detailed method. Full binary/source hashes are archived.

All timed-interval sampling coverage checks passed (200 ms plus engine-side boundaries). Bulk load, full verification and counter replay are excluded from that interval; their recent activity can contribute to system load. Process-wide raw load samples are retained too.