[tae898]

Pin bumps for three comparators, from the 2026-10-23 subset readiness review (.notes/bench/SUBSET-READINESS-20261023.md, item 3). Only these engines' entries are touched.

Versions

Engine	Before	After	Pin
SurrealDB server	v3.2.4 sha256:6a500236...	v3.3.4, also latest	amd64 manifest sha256:0d25256bbbe25235beb0ae8d785065fbafade32169956c5b468daebc67de31df
DuckDB (+ DuckPGQ, VSS)	1.5.4	1.5.6, not 2.0.0	duckdb==1.5.6 in dbbench:duckdb and dbbench:dense; DuckPGQ build 61f5d082
python-arango	8.3.5	8.3.6	PKGS[client]

SurrealDB SDK 2.0.0 is still the newest SDK, so it stays.

Why DuckDB stays on 1.5.x

DuckDB 2.0.0 was tried first (laptop, PyPI wheel). The vss extension loads and its HNSW index works. The duckpgq community build 29ca668a22 exists and LOADs, but it registers no syntax: CREATE PROPERTY GRAPH and GRAPH_TABLE ... MATCH both answer Parser Error: syntax error, with or without SET enable_legacy_parser or SET allow_parser_override_extension. The graph arm cannot be measured on 2.0.0, and one engine wears one version per page (DECISIONS #103d), so every DuckDB arm goes to 1.5.6, where INSTALL/LOAD duckpgq, CREATE PROPERTY GRAPH, GRAPH_TABLE and VSS all work. The reason and the lift condition are written beside the pin (build_images.sh, COMPARATORS.md). 2.0.0 notes read: new PEG parser default, single-arrow lambdas off, ICU moved to an extension, files written by 2.0 unreadable by older DuckDB; none of it was needed once the extension failed. Re-check when a 2.x DuckPGQ build parses.

SurrealDB 3.3

The GitHub releases carry no notes. Compared surreal start --help between the two images: the only additions are --graph-fold-interval (background fold of graph adjacency keys, default 5s) and a Postgres wire listener, both left at default. rocksdb:...?sync=never and ?sync=every still print Sync mode: never (handled by the OS and Sync mode: every transaction commit, the line the runner reads back (BUGS F165). The September "higher numbers are pre-releases" note in COMPARATORS.md is removed.

Changed

build_images.sh, runner.py (digest on the six served SurrealDB arms), COMPARATORS.md, docstrings and comments that name the live version, version_consistency_check.py message, and stage_drafts_2611.sha256 (the 14 stage drafts that bake in the digest; reviewed, only those hashes moved). python-bindings/ keeps duckdb 1.5.4: its frozen results are at that version.

Tests

test_fairness_headers, test_memory_probe, test_stage_generator_2611, test_stage_generator_after, test_engine_pin, test_vendor_memory, test_milvus_compose, test_jpype_pin, test_land_stage_pin, test_surreal_reconnect, test_index_before_load_note, test_cell_cost_backends: 189 passed, 1 skipped. version_pin_check.py: 0 disagreements.

Not merged. Not rebased on the parallel pin PRs: expect a trivial conflict in stage_drafts_2611.sha256 (re-run python3 test_stage_generator_2611.py --rewrite after the last merge) and in build_images.sh / runner.py.

🤖 Generated with Claude Code

[tae898]

Smoke result, 2026-10-23 (laptop, micro scale, cpuset 8-11, one rep, old pins from origin/main versus the new images dbbench:{duckdb,client,dense}-bump and the v3.3.4 digest). Chain: l1tpc, l2, l3d, e2, lifecycle for DuckDB, DuckPGQ, DuckDB VSS, SurrealDB served and ArangoDB.

• Equal to before: 12 of 24 cells have identical answer fields (digests, row counts) and the other 11 cells have none of those fields to compare. Includes DuckDB l1tpc oltp/olap, DuckPGQ olap (12 graph questions), DuckDB VSS and the cross-model duckdb_e2, and ArangoDB with python-arango 8.3.6. Row engine_version reads 1.5.6, surrealdb-server:3.3.4, arangodb:3.12.12.
• One difference to look at: surrealdb_graph_server oltp: graph_delete and graph_delete_edges digests differ (1c61013c… -> 17d7ae00…, 675d5fdb… -> 46152f30…). The hop3f cell is censored in both at its 135 s budget but at different start counts (39 vs 34 of 50), so the delete targets may depend on how far the censored cell got; not yet shown to be a 3.3.4 answer change. Needs a rerun of that cell before the SurrealDB graph lane is trusted on 3.3.4.
• Failed identically on old and new, so not caused by the bump, but not covered: duckpgq_graph oltp (cell failed at connect on both; olap passes), surrealdb_tpc_server olap (900 s timeout on both), and all three l4 cells (l4_tsbs.py usage error from the runner on both). The l4/ts lane, DuckPGQ oltp and SurrealDB TPC olap are therefore unsmoked on both versions.
• Timings were not compared (laptop, one rep, shared host).

[tae898]

Not merged: the answer check fails on SurrealDB v3.3.4.

I ran the old-vs-new smoke (l2 graph, micro, surrealdb_graph_server, one rep) and the OLTP delete phase leaves one person and its edge behind on v3.3.4, with no error and delete_ops 1000. The delete is BEGIN; DELETE $n<->knows; DELETE $n; COMMIT; over the 1000 ids the write pass created. The result digests res_graph_delete_digest and res_graph_delete_edges_digest must be those of an empty scan (res_graph_delete_n 0).

run	server	rows left after delete	leftover id
smoke	3.2.4	0	none
rerun	3.2.4	0	none
smoke	3.3.4	1	10000248
rerun 1	3.3.4	1	10000392
rerun 2	3.3.4	0	none

So 2 of 3 runs on 3.3.4 lose one delete, 0 of 2 on 3.2.4. A different id each time, so it is not one bad row. Every other answer matches between old and new (SAME or equal recall, torn counts and heads for l1tpc, l2 olap, l3d, e2); the duckdb and arangodb rows are equal, so this is the SurrealDB bump only. Needs a look at whether a 3.3.x transaction drops a statement under the lane's concurrent load (and an upstream issue if so) before the pin moves. The merge of main into this branch is pushed (1fdb4afea5, CI green), so nothing else is blocking.

Also in the smoke, identical on old and new and so not caused by the bump: surrealdb_tpc_server olap times out at 900 s at micro, duckpgq_graph oltp errors after connect-star, and the l4 cells fail on the smoke's --scale micro (not a valid l4 scale).

[tae898]

The DuckDB 1.5.6 and python-arango 8.3.6 part moved to #487 (merged). #483 stays open for the SurrealDB bump only; it needs a rebase onto main (SurrealDB stays at v3.2.4 on main, with the #483 reason beside the pin). A repeat study of the l2 OLTP delete leftover (alternating 3.2.4 and 3.3.4, fresh container per run) is running before the pin moves.

🤖 Generated with Claude Code

[tae898]

Repeat of the full l2 cell (surrealdb_graph_server, micro, OLAP then OLTP, same command as the smoke), alternating versions, a fresh server per run, current main with only the server image swapped:

server	runs	runs with a deleted person and its edge left behind
v3.2.4	12	0
v3.3.4	12	6

One-sided Fisher exact p = 0.0069. Each leftover is a different id (10000158, 10000164, 10000223, 10000191, 10000196, 10000188), delete_ops is 1000 every time and no error is raised. A reproducer with only the write/update/delete workload (no reads before it, 2 GB cap) was clean in 60 of 60 runs on each version, so the trigger needs something earlier in the cell. SurrealDB stays at v3.2.4 (pinned with this reason in runner.py since #487); this PR stays open until the trigger is narrowed down and reported upstream. Runs and script: .notes/bench/repros/surrealdb-334-delete-20261023/ (lane.sh, lane-results.txt).

[tae898]

Filed upstream as surrealdb/surrealdb#7627: on 3.3.4 the delete leftover needs the instance to have served a read workload first (2-hop traversals); a standalone script leaves rows behind in 12 of 24 runs on 3.3.4 and 0 of 24 on 3.2.4. The pin stays at 3.2.4 until a fixed release.

[tae898]

Main (f8c5f24, with #503's schedule, #487's pins and #506) is merged into this branch as aa03d55, a merge commit that keeps this PR's commits. The conflicts in COMPARATORS.md, QUANTIZATION.md, build_images.sh and runner.py took main's text everywhere except the pins this PR carries: the SurrealDB server image stays at v3.3.4 (sha256:0d25256b...) and the client image keeps python-arango==8.3.6 (main's build_images.sh still read 8.3.5 beside 8.3.6 in COMPARATORS.md and the smoke script). stage_drafts_2611.sha256 is regenerated with python3 test_stage_generator_2611.py --rewrite, not hand-merged (110 of 319 lines differ from main: the stages that bake in the SurrealDB digest, in all three generations). Checks on the merged tree, pinned to cores 0-3: make_2611_stages.py --check and --check --drafts (110 drafts) pass, version_pin_check.py reports 0 disagreements, and the CI pytest for benchmarks/experiments gives 1063 passed, 24 skipped. version_consistency_check.py needs a web_benchmarks.json payload that the repo does not carry (it exits 1 on main too); test_publish_gates_2611.py covers it and passes. This PR is still blocked on the SurrealDB v3.3.4 delete leftover (surrealdb/surrealdb#7627); the pin on main stays at v3.2.4 until a fixed release. Not merged.

🤖 Generated with Claude Code

[tae898]

Closing in favour of #508. DuckDB 1.5.6 and python-arango 8.3.6 already landed with #487, and the remaining change, SurrealDB 3.3.4, must stay out: 3.3.4 silently drops acknowledged transactional deletes after a read workload (6 of 12 runs, surrealdb/surrealdb#7627), so main keeps 3.2.4. The one useful leftover, build_images.sh still installing python-arango 8.3.5, is #508.