Repository navigation
perf(zarr-indexing): group chunk points with a radix sort and stop deduplicating writes - #4649
Merged
d-v-b merged 6 commits intoNov 18, 2026
Conversation
…p deduplicating writes Planning grouped each chunk's points with a stable sort so that every `IndexedSet` and `JointSet` row kept request order, and the writer relied on that order to make the last repeated write win. Both promises are dropped: the order of points within a row is unspecified, and a source cell a write addresses more than once receives one of its values, whole, without saying which. Callers who need a particular value remove repeats from the selection first. Grouping now uses the default sort behind `_group_by_key`, which also skips the sort for already-grouped joint keys, and the writer no longer deduplicates targets before its single fancy assignment or `np.put`. Tests that assumed last-wins now accept any written value, and a new test runs orthogonal, vectorized and mixed repeated writes with the grouping replaced by one that reverses each row. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write order Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documentation build overview
6 files changed ·
|
d-v-b
marked this pull request as ready for review
November 18, 2026 16:04
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4649 +/- ##
=======================================
Coverage 94.90% 94.91%
=======================================
Files 96 96
Lines 14055 14070 +15
=======================================
+ Hits 13339 13354 +15
Misses 716 716 🚀 New features to boost your workflow:
|
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 18, 2026
…t up to 2**32 chunks Match zarr-indexing (zarr-developers#4649): the value an element receives when an index array names it more than once is unspecified, as in NumPy. This removes `last_writes` and its per-chunk resolution from the write path, and documents the rule in the user guide and the setter docstrings. Chunk indices are now radix-sorted 16 bits at a time up to 2**32 chunks, which also covers arrays with more than 2**16 chunks: building an orthogonal indexer for 10M points over 1M chunks takes 233 ms instead of 817 ms on main. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NumPy's stable sort is a linear-time radix sort for integers of at most 16 bits. `_group_by_key` shifts keys to start at zero and sorts them one 16-bit digit at a time: one pass when their range fits 16 bits, two least-significant-first passes when it fits 32 bits, and the default sort beyond. Grouping 1M random keys over 65,536 chunks drops from about 46 ms to 2 ms. The order within a group stays unspecified. The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a shift by the minimum key because grids may have negative chunk IDs. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_group_by_key` fell back to the default sort above a 2**32 key range, which was slower than radix passes and, as zarr-python zarr-developers#4643 measured, slower at every key width. It now takes one 16-bit pass per 16 bits of the shifted key range, up to four, reading the shifted keys as uint64 so a range above 2**63 stays exact. Inputs under 128 keys per pass take one stable comparison sort, which beats the passes' fixed overhead. This mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays unspecified. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d value Add a "Repeated write targets" design note: the library promises no value for a cell a write addresses more than once, chooses its internal order by performance (today a stable radix sort that happens to keep the last write), carries no order within a planning-table row, and does not deduplicate, because the pass would cost more than the assignment. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We stop promising that writes to the same output array cell resolve in any particular order.
🤖 AI text below 🤖
LazyArray.writeno longer promises that the last of several writes to one source cell wins: the cell receives one of the values written to it, whole, but which one is unspecified, so remove repeated indices from a selection whose result matters.IndexedSetandJointSetno longer promise an order for the points within a row either. Planning now groups points by chunk with a linear-time radix sort and writes skip deduplication, so partitioning a large unordered selection is up to 5x faster and a fancy write to a NumPy array about 10x faster.What changed
chunk_resolution.py: both grouping sorts (orthogonal_indexed_set, joint_joint_set) go through a new_group_by_key, which returnsNonewhen keys are already grouped and otherwise sorts them with NumPy's 16-bit radix sort: keys are shifted to start at zero (grids may have negative chunk IDs) and read as uint64, so a range above 2**63 stays exact, then sorted one 16-bit digit per pass, least significant first, with one pass per 16 bits of range (at most four). Inputs under 128 keys per pass take one stable comparison sort, which beats the passes' fixed overhead. This follows perf(indexing): radix-sort chunk indices and document repeated-index writes as unspecified #4643'sargsort_chunk_indices. The joint path gains the already-sorted fast path the orthogonal path had.IndexedSet/JointSetdocstrings and doctests,docs/snippets/grid_partition.pyanddocs/guide/selection-flow.mdnow say the order within a row is unspecified, with each point still paired with its request position.writer.py:_last_occurrencesis removed._scatter_numpydoes one fancy assignment and_scatter_plannedonenp.putper cell, duplicates included. The agent checked that a masked NumPy source keeps data and mask from the same write when indices repeat.LazyArray.writeandwrite_intodocstrings state the new rule.docs/design-notes.mdgains a "Repeated write targets" section: no promised value for a repeated target, internal order chosen by performance (today it happens to keep the last write), no order within a planning-table row, and no deduplication, with the reasons.changes/4603.misc.mdis deleted: it described the deduplication fast path this PR removes, and it was never released.Why not keep the order
The original brief was to rank duplicates by C-order view position, so last-wins held under any within-cell order, and then drop
kind="stable". The agent stopped because the within-row order was documented on exported classes. The author decided that zarr-indexing is pre-1.0, speed matters more, and repeated writes should promise nothing: callers who care remove the duplicates first. The rule matches NumPy's documented stance on advanced assignment ("if an element is set more than once, it is not possible to predict the final result"), PyTorch'sindex_put_(undefined with duplicates) and JAX's.at[].set()(implementation-defined order). zarr-python #4643 now documents repeated-index writes as unspecified too, so the two packages state the same rule. Every branch of the grouping sort is stable, so rows happen to keep request order; that is an implementation detail, not a promise.Tests
test_repeated_writes_land_one_whole_value_per_cell: orthogonal, vectorized and mixed (vindex over oindex) repeated writes, through the per-cell path (a readable grid source), a NumPy source and a masked NumPy source, with the default grouping and with_group_by_keymonkeypatched tonp.lexsort((-np.arange(n), keys)), which reverses each row. It checks that unaddressed cells are untouched, cells addressed once get exactly their value, and repeated cells get one of their values. For the masked source, values are masked by parity, so data and mask from different writes would be caught.np.put(block, np.sort(flat), ...)(targets reordered apart from values): 5 failures_indexed_set:index = np.sort(flat[positions]): 2 failures_joint_set:np.sort(block.flat_index[d][positions]): 4 failurestest_group_by_key_orders_keys_ascending: 100 and 2000 keys over ranges 1, 2, 28, 216, 216 + 1, 232, 232 + 1, 248, 248 + 1, 262 and the full int64 range, at offsets 0, -3, 240 + 12345 and -(240) - 777, each covering its whole range. It checks that sorted keys returnNone, and that otherwise the result is an intp permutation that orders the keys. It does not assert the order within a group. Each of these mutations fails it: floor instead of ceil for the pass count (16 failures), capping at 3 passes (12), taking a later digit from the unsorted keys (24), an unstable later pass (28), dropping the shift to zero (17), signed shifted keys (28), and a small-input sort on truncated keys (32). Offsets that are multiples of 2**16 let the missing-shift mutant survive, so the offsets avoid them. An unstable small-input sort cannot fail this test, because the order within a group is not promised.test_write_coordinates,test_lazy_write_and_assignment,test_random_selection_chain_scatter,test_fancy_writes_touch_each_write_chunk_once[duplicates],test_write_constant_repeated_destination) now accept any written value per repeated cell.test_last_occurrences_matches_sequential_assignmentis removed with the helper.UV_PYTHON=3.13 just checkpasses: ruff, pyright (0 errors, warnings unchanged from main), 1845 passed / 15 skipped, the TensorStore parity suites (88 passed), andmkdocs build --strict. pre-commit passes on the changed files.*_propagates_malformed_index_protocoltests fail on main as well: CPython 3.15 rewords the__index__error message.Benchmarks
Apple Silicon, numpy 2.5.4, best of 3 to 5. All three columns come from one run, against snapshots of main and the earlier 32-bit radix commit. Partitioning calls
plan_chunks(...).partition()on uniformly random (unordered) indices. Writes go throughwrite_into: "planned" is a readable non-NumPy source with a 64x64 write grid on a 1024x1024 float64 array, "numpy" is a plain ndarray.The 3-D cases have joint keys over 236 chunks, which the 32-bit version sorted with the default
argsort. Most of their time is spent outside the sort. The sort alone on 10M random int64 keys: defaultargsortabout 700 ms at any width, radix 202 ms (233 range, 3 passes), 231 ms (248), 329 ms (262, 4 passes). Between about 128 and 2000 keys over wide ranges, radix is a few microseconds slower than the default sort (512 keys over 2**40: 2.8 us against 7.7 us); that band keeps the radix and stable paths to match #4643.The NumPy-path gain is mostly the removed
np.unique: the C-order walk of an unordered vindex has unsorted targets, so the old helper always took its sorting branch.Stack notes
Follow-ups noticed
_scatter_plannedcopies each cell withnp.array(source[box], ...), which drops masks, so a readable non-NumPy source that returns masked blocks loses masks on write. This is pre-existing and not touched here. The new test covers masked sources only through the NumPy path.__index__message tests need a regex that matches both the CPython 3.13 and 3.15 wording.🤖 Generated with Claude Code