Skip to content

perf(zarr-indexing): group chunk points with a radix sort and stop deduplicating writes - #4649

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
d-v-b:perf/zarr-indexing-unordered-chunk-groups
Nov 18, 2026
Merged

d-v-b merged 6 commits into
zarr-developers:mainfrom
d-v-b:perf/zarr-indexing-unordered-chunk-groups

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

We stop promising that writes to the same output array cell resolve in any particular order.

🤖 AI text below 🤖

LazyArray.write no longer promises that the last of several writes to one source cell wins: the cell receives one of the values written to it, whole, but which one is unspecified, so remove repeated indices from a selection whose result matters. IndexedSet and JointSet no longer promise an order for the points within a row either. Planning now groups points by chunk with a linear-time radix sort and writes skip deduplication, so partitioning a large unordered selection is up to 5x faster and a fancy write to a NumPy array about 10x faster.

What changed
  • chunk_resolution.py: both grouping sorts (orthogonal _indexed_set, joint _joint_set) go through a new _group_by_key, which returns None when keys are already grouped and otherwise sorts them with NumPy's 16-bit radix sort: keys are shifted to start at zero (grids may have negative chunk IDs) and read as uint64, so a range above 2**63 stays exact, then sorted one 16-bit digit per pass, least significant first, with one pass per 16 bits of range (at most four). Inputs under 128 keys per pass take one stable comparison sort, which beats the passes' fixed overhead. This follows perf(indexing): radix-sort chunk indices and document repeated-index writes as unspecified #4643's argsort_chunk_indices. The joint path gains the already-sorted fast path the orthogonal path had.
  • IndexedSet / JointSet docstrings and doctests, docs/snippets/grid_partition.py and docs/guide/selection-flow.md now say the order within a row is unspecified, with each point still paired with its request position.
  • writer.py: _last_occurrences is removed. _scatter_numpy does one fancy assignment and _scatter_planned one np.put per cell, duplicates included. The agent checked that a masked NumPy source keeps data and mask from the same write when indices repeat.
  • LazyArray.write and write_into docstrings state the new rule.
  • docs/design-notes.md gains a "Repeated write targets" section: no promised value for a repeated target, internal order chosen by performance (today it happens to keep the last write), no order within a planning-table row, and no deduplication, with the reasons.
  • changes/4603.misc.md is deleted: it described the deduplication fast path this PR removes, and it was never released.
Why not keep the order

The original brief was to rank duplicates by C-order view position, so last-wins held under any within-cell order, and then drop kind="stable". The agent stopped because the within-row order was documented on exported classes. The author decided that zarr-indexing is pre-1.0, speed matters more, and repeated writes should promise nothing: callers who care remove the duplicates first. The rule matches NumPy's documented stance on advanced assignment ("if an element is set more than once, it is not possible to predict the final result"), PyTorch's index_put_ (undefined with duplicates) and JAX's .at[].set() (implementation-defined order). zarr-python #4643 now documents repeated-index writes as unspecified too, so the two packages state the same rule. Every branch of the grouping sort is stable, so rows happen to keep request order; that is an implementation detail, not a promise.

Tests
  • New test_repeated_writes_land_one_whole_value_per_cell: orthogonal, vectorized and mixed (vindex over oindex) repeated writes, through the per-cell path (a readable grid source), a NumPy source and a masked NumPy source, with the default grouping and with _group_by_key monkeypatched to np.lexsort((-np.arange(n), keys)), which reverses each row. It checks that unaddressed cells are untouched, cells addressed once get exactly their value, and repeated cells get one of their values. For the masked source, values are masked by parity, so data and mask from different writes would be caught.
  • Mutation checks (run by hand, then reverted). Each mutation fails the new test:
    • per-cell path: np.put(block, np.sort(flat), ...) (targets reordered apart from values): 5 failures
    • _indexed_set: index = np.sort(flat[positions]): 2 failures
    • _joint_set: np.sort(block.flat_index[d][positions]): 4 failures
    • NumPy path: targets sorted apart from values: 12 failures
  • New test_group_by_key_orders_keys_ascending: 100 and 2000 keys over ranges 1, 2, 28, 216, 216 + 1, 232, 232 + 1, 248, 248 + 1, 262 and the full int64 range, at offsets 0, -3, 240 + 12345 and -(240) - 777, each covering its whole range. It checks that sorted keys return None, and that otherwise the result is an intp permutation that orders the keys. It does not assert the order within a group. Each of these mutations fails it: floor instead of ceil for the pass count (16 failures), capping at 3 passes (12), taking a later digit from the unsorted keys (24), an unstable later pass (28), dropping the shift to zero (17), signed shifted keys (28), and a small-input sort on truncated keys (32). Offsets that are multiples of 2**16 let the missing-shift mutant survive, so the offsets avoid them. An unstable small-input sort cannot fail this test, because the order within a group is not promised.
  • Tests that built a sequential last-wins reference (test_write_coordinates, test_lazy_write_and_assignment, test_random_selection_chain_scatter, test_fancy_writes_touch_each_write_chunk_once[duplicates], test_write_constant_repeated_destination) now accept any written value per repeated cell. test_last_occurrences_matches_sequential_assignment is removed with the helper.
  • UV_PYTHON=3.13 just check passes: ruff, pyright (0 errors, warnings unchanged from main), 1845 passed / 15 skipped, the TensorStore parity suites (88 passed), and mkdocs build --strict. pre-commit passes on the changed files.
  • Under Python 3.15 (uv's default here), four existing *_propagates_malformed_index_protocol tests fail on main as well: CPython 3.15 rewords the __index__ error message.
Benchmarks

Apple Silicon, numpy 2.5.4, best of 3 to 5. All three columns come from one run, against snapshots of main and the earlier 32-bit radix commit. Partitioning calls plan_chunks(...).partition() on uniformly random (unordered) indices. Writes go through write_into: "planned" is a readable non-NumPy source with a 64x64 write grid on a 1024x1024 float64 array, "numpy" is a plain ndarray.

case main 32-bit radix (7626722) any-width radix (this PR)
partition oindex n=1e+06 chunks=65536 87.9 ms 17.4 ms 17.8 ms
partition oindex n=1e+06 chunks=1048576 178.9 ms 122.3 ms 115.2 ms
partition oindex n=4e+06 chunks=1048576 659.4 ms 261.7 ms 231.3 ms
partition vindex n=1e+06 chunks=65536 102.8 ms 34.5 ms 32.0 ms
partition vindex 3-D n=1e+06 chunks=2**36 567.2 ms 668.6 ms 516.2 ms
partition vindex 3-D n=4e+06 chunks=2**36 2480.9 ms 2613.6 ms 2113.9 ms
write planned vindex unique n=2e+05 31.8 ms 15.4 ms 16.0 ms
write numpy vindex unique n=2e+05 14.0 ms 1.5 ms 1.5 ms
write planned vindex repeats n=2e+05 31.7 ms 15.6 ms 15.7 ms
write numpy vindex repeats n=2e+05 14.5 ms 1.4 ms 1.4 ms
write planned vindex heavy-repeats n=2e+05 17.3 ms 5.9 ms 6.2 ms
write numpy vindex heavy-repeats n=2e+05 12.7 ms 1.4 ms 1.4 ms
write planned oindex unique rows=1024 71.7 ms 50.1 ms 52.0 ms
write planned oindex repeats rows=1024 69.6 ms 50.1 ms 50.7 ms

The 3-D cases have joint keys over 236 chunks, which the 32-bit version sorted with the default argsort. Most of their time is spent outside the sort. The sort alone on 10M random int64 keys: default argsort about 700 ms at any width, radix 202 ms (233 range, 3 passes), 231 ms (248), 329 ms (262, 4 passes). Between about 128 and 2000 keys over wide ranges, radix is a few microseconds slower than the default sort (512 keys over 2**40: 2.8 us against 7.7 us); that band keeps the radix and stable paths to match #4643.

The NumPy-path gain is mostly the removed np.unique: the C-order walk of an unordered vindex has unsorted targets, so the old helper always took its sorting branch.

Stack notes
Follow-ups noticed
  • _scatter_planned copies each cell with np.array(source[box], ...), which drops masks, so a readable non-NumPy source that returns masked blocks loses masks on write. This is pre-existing and not touched here. The new test covers masked sources only through the NumPy path.
  • The __index__ message tests need a regex that matches both the CPython 3.13 and 3.15 wording.

🤖 Generated with Claude Code

d-v-b and others added 3 commits November 18, 2026 16:36
…p deduplicating writes

Planning grouped each chunk's points with a stable sort so that every
`IndexedSet` and `JointSet` row kept request order, and the writer relied
on that order to make the last repeated write win. Both promises are
dropped: the order of points within a row is unspecified, and a source
cell a write addresses more than once receives one of its values, whole,
without saying which. Callers who need a particular value remove repeats
from the selection first.

Grouping now uses the default sort behind `_group_by_key`, which also
skips the sort for already-grouped joint keys, and the writer no longer
deduplicates targets before its single fancy assignment or `np.put`.

Tests that assumed last-wins now accept any written value, and a new test
runs orthogonal, vectorized and mixed repeated writes with the grouping
replaced by one that reverses each row.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write order

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-indexing Specific to the zarr-indexing sub-package labels Nov 18, 2026
@d-v-b d-v-b added needs release notes Automatically applied to PRs which haven't added release notes zarr-indexing Specific to the zarr-indexing sub-package labels Nov 18, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 zarr-indexing | 🛠️ Build #35994930 | 📁 Comparing 0535ec9 against latest (6eb4563)

  🔍 Preview build  

6 files changed · ± 6 modified

± Modified

@d-v-b
d-v-b marked this pull request as ready for review November 18, 2026 16:04
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.91%. Comparing base (b0020f2) to head (0535ec9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4649   +/-   ##
=======================================
  Coverage   94.90%   94.91%           
=======================================
  Files          96       96           
  Lines       14055    14070   +15     
=======================================
+ Hits        13339    13354   +15     
  Misses        716      716           

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 18, 2026
…t up to 2**32 chunks

Match zarr-indexing (zarr-developers#4649): the value an element receives when an index
array names it more than once is unspecified, as in NumPy. This removes
`last_writes` and its per-chunk resolution from the write path, and
documents the rule in the user guide and the setter docstrings.

Chunk indices are now radix-sorted 16 bits at a time up to 2**32 chunks,
which also covers arrays with more than 2**16 chunks: building an
orthogonal indexer for 10M points over 1M chunks takes 233 ms instead of
817 ms on main.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NumPy's stable sort is a linear-time radix sort for integers of at most
16 bits. `_group_by_key` shifts keys to start at zero and sorts them one
16-bit digit at a time: one pass when their range fits 16 bits, two
least-significant-first passes when it fits 32 bits, and the default
sort beyond. Grouping 1M random keys over 65,536 chunks drops from about
46 ms to 2 ms. The order within a group stays unspecified.

The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a
shift by the minimum key because grids may have negative chunk IDs.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b and others added 2 commits November 18, 2026 20:53
`_group_by_key` fell back to the default sort above a 2**32 key range,
which was slower than radix passes and, as zarr-python zarr-developers#4643 measured,
slower at every key width. It now takes one 16-bit pass per 16 bits of
the shifted key range, up to four, reading the shifted keys as uint64 so
a range above 2**63 stays exact. Inputs under 128 keys per pass take one
stable comparison sort, which beats the passes' fixed overhead. This
mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays
unspecified.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d value

Add a "Repeated write targets" design note: the library promises no
value for a cell a write addresses more than once, chooses its internal
order by performance (today a stable radix sort that happens to keep the
last write), carries no order within a planning-table row, and does not
deduplicate, because the pass would cost more than the assignment.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b d-v-b changed the title perf(zarr-indexing): group chunk points with an unstable sort and stop deduplicating writes perf(zarr-indexing): group chunk points with a radix sort and stop deduplicating writes Nov 18, 2026
@d-v-b
d-v-b merged commit caa1536 into zarr-developers:main Nov 18, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-indexing Specific to the zarr-indexing sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant