Skip to content

fix(indexing): keep the last value when setitem gets repeated indices - #453

Open
d-v-b wants to merge 13 commits into
mainfrom
fix/stable-chunk-grouping
Open

d-v-b wants to merge 13 commits into
mainfrom
fix/stable-chunk-grouping

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Assigning through oindex, vindex or __setitem__ with an index array that repeats a position now stores the last value given for that position, as NumPy does. Previously the stored value depended on the platform, the chunk layout and the number of selected items.

Cause and fix

IntArrayDimIndexer and CoordinateIndexer group the selected items by chunk with np.argsort, whose default sort is not stable. Repeated indices therefore reached the in-chunk assignment in an arbitrary order. The agent changed both sorts to kind="stable", so repeats stay in the order given and NumPy's in-chunk assignment stores the last value. The sharding codec builds its inner selections with the same two classes, so it is covered by the same change.

This is not only a Linux problem: NumPy's default sort happens to be stable below 16 elements, so short examples pass on some platforms. On an arm64 Mac, a.vindex[idx] = np.arange(24) with 23 of 24 points on one cell stored 22 instead of 23 before this change.

Refs zarr-developers#2855. This is an alternative to zarr-developers#4601, which removes duplicates from the selection before writing. Compared with that approach, this one adds no pass over the selection, no second indexer build and no copy of the value, and it never flattens coordinates over the full array shape, so writes to arrays with more than 2**63 elements keep working.

Tests
  • test_setitem_repeated_index was skipped; it is now one parametrized test over oindex/vindex, four chunk layouts (including sharded) and four selections, compared against NumPy. 9 of its 32 cases fail without the fix on an arm64 Mac.
  • The property tests test_oindex and test_vindex no longer assume away repeated indices (the same change as in Keep the last value when setitem gets repeated indices zarr-developers/zarr-python#4601). Both fail without the fix and pass with it on five seeds.
  • tests/test_indexing.py, tests/test_properties.py and tests/test_codecs/test_sharding.py pass locally.
Limits
  • The fix relies on NumPy applying an advanced assignment with repeated indices in order inside a chunk. NumPy does this in practice, but its documentation does not guarantee it.
  • GPU buffers were not tested. CuPy documents the result of an assignment with repeated indices as undefined, so this change may not be enough there. Removing duplicates per chunk before the in-chunk assignment would close both gaps.

🤖 Generated with Claude Code

d-v-b and others added 13 commits November 16, 2026 17:46
The indexers group selected items by chunk with an unstable argsort, so
repeated indices reached the in-chunk assignment in an arbitrary order
and the stored value depended on the platform, the chunk layout and the
number of selected items. Sorting stably keeps repeats in the order
given, and the in-chunk NumPy assignment then stores the last value.

Refs zarr-developers#2855

Assisted-by: ClaudeCode:claude-fable-5.2
…zarr-developers#4611)

* Narrow partial reads of uncompressed chunks within a single row

When a selection touches one index along the first axis, apply the same
row window along the next axis, and so on. arr[t, y0:y1, x0:x1] then
fetches rows y0 to y1 of frame t instead of the whole frame, and a
single item is fetched for a point.

The fetched block keeps the narrowed axes with length 1, so the shifted
selection applies to it unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add release note for zarr-developers#4611

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…chunk

The stable argsort that keeps repeated indices in order was 2-3x slower than
NumPy's default sort for large unordered selections, on reads as well as
writes. NumPy sorts integers of at most 16 bits with a stable radix sort, so
chunk indices are narrowed to uint16 when there are at most 2**16 chunks.
Building an indexer for 10M unordered points then takes about 165-200 ms,
against 560-690 ms on main and 1.5-2.0 s with the plain stable sort.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r chunk

The stable sort kept repeated indices in the order given, but the in-chunk
assignment still left the stored value to the array library: NumPy does not
document the order of repeated assignments and CuPy leaves it undefined.

Following zarr-indexing, `_merge_chunk_array` now resolves a chunk selection
that names an element more than once into one that names each element once,
with the last value given for it in C order. Orthogonal selections are
resolved one axis at a time; coordinate selections jointly, in linear time
via `np.maximum.at` unless the chunk is much larger than the selection.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unk order

Repeats are now resolved before any codec sees them: `last_writes` wraps the
indexer in `_set_selection` and, for each chunk projection, keeps the entry
with the highest position in `out_selection` for every element named more
than once. This is the last value in the selection's C order, and it does
not depend on the order items are listed within a chunk, so the sort that
groups items by chunk no longer needs to be stable. Above 2**16 chunks the
indexers return to NumPy's default sort, recovering main's indexer speed.

The per-chunk merge no longer dedupes, and the sharding codec never receives
a repeated index.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t up to 2**32 chunks

Match zarr-indexing (zarr-developers#4649): the value an element receives when an index
array names it more than once is unspecified, as in NumPy. This removes
`last_writes` and its per-chunk resolution from the write path, and
documents the rule in the user guide and the setter docstrings.

Chunk indices are now radix-sorted 16 bits at a time up to 2**32 chunks,
which also covers arrays with more than 2**16 chunks: building an
orthogonal indexer for 10M points over 1M chunks takes 233 ms instead of
817 ms on main.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lue is unspecified

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sort 16 bits at a time for as many passes as the chunk count needs (up
to four), instead of falling back to NumPy's default sort above 2**32
chunks. Building an orthogonal indexer for 10M points over 2**40 chunks
takes 342 ms instead of 838 ms. Below 128 items per pass the plain
stable sort is faster and is used instead, so the sort is stable for
every input.

Assisted-by: ClaudeCode:claude-fable-5-2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…duplicating writes (zarr-developers#4649)

* perf(zarr-indexing): group chunk points with an unstable sort and stop deduplicating writes

Planning grouped each chunk's points with a stable sort so that every
`IndexedSet` and `JointSet` row kept request order, and the writer relied
on that order to make the last repeated write win. Both promises are
dropped: the order of points within a row is unspecified, and a source
cell a write addresses more than once receives one of its values, whole,
without saying which. Callers who need a particular value remove repeats
from the selection first.

Grouping now uses the default sort behind `_group_by_key`, which also
skips the sort for already-grouped joint keys, and the writer no longer
deduplicates targets before its single fancy assignment or `np.put`.

Tests that assumed last-wins now accept any written value, and a new test
runs orthogonal, vectorized and mixed repeated writes with the grouping
replaced by one that reverses each row.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): add changelog fragment for unspecified repeated-write order

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): number the changelog fragment for upstream PR 4649

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(zarr-indexing): group chunk keys with a 16-bit radix sort

NumPy's stable sort is a linear-time radix sort for integers of at most
16 bits. `_group_by_key` shifts keys to start at zero and sorts them one
16-bit digit at a time: one pass when their range fits 16 bits, two
least-significant-first passes when it fits 32 bits, and the default
sort beyond. Grouping 1M random keys over 65,536 chunks drops from about
46 ms to 2 ms. The order within a group stays unspecified.

The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a
shift by the minimum key because grids may have negative chunk IDs.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(zarr-indexing): radix-sort chunk keys of any width

`_group_by_key` fell back to the default sort above a 2**32 key range,
which was slower than radix passes and, as zarr-python zarr-developers#4643 measured,
slower at every key width. It now takes one 16-bit pass per 16 bits of
the shifted key range, up to four, reading the shifted keys as uint64 so
a range above 2**63 stays exact. Inputs under 128 keys per pass take one
stable comparison sort, which beats the passes' fixed overhead. This
mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays
unspecified.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): record why repeated write targets get no promised value

Add a "Repeated write targets" design note: the library promises no
value for a cell a write addresses more than once, chooses its internal
order by performance (today a stable radix sort that happens to keep the
last write), carries no order within a planning-table row, and does not
deduplicate, because the pass would cost more than the assignment.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants