Skip to content

feat(zarr-storage): add a Zarr-agnostic storage package with a new store API - #460

Draft
d-v-b wants to merge 6 commits into
topic/zarr-storagefrom
feat/zarr-storage-api
Draft

d-v-b wants to merge 6 commits into
topic/zarr-storagefrom
feat/zarr-storage-api

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Add zarr-storage, a new package with a Zarr-agnostic key-value storage API: Store and AsyncStore views over memory, local-filesystem and obstore backends, with the same operations as sync methods and as coroutines, conditional reads and writes (get_if_changed, put_if_absent, put_if_match, delete_if_match), byte-range reads, reads into caller buffers, and batches (get_many, get_into_many, put_many, delete_many) that run concurrently where the backend can. It ships a conformance suite in zarr_storage.testing that any backend can run.

What the API is

packages/zarr-storage/DESIGN.md is the contract; it records which source each decision came from (the roadmap, zng_storage, the zarr-python-planning store proposals, TensorStore, zarrs).

  • Store (sync methods, close/with) and AsyncStore (the same names as coroutines, aclose/async with) are views of a backend at a root, sharing one base where the key grammar, the root, read-only and closed states are enforced; backends only see backend paths. store / "a", as_async() and as_sync() return views on the same backend; the constructed store owns the backend. There is no value equality: shares_backend and overlaps answer what callers ask.
  • Keys: no leading/trailing /, no empty, . or .. segments, no NUL. Prefix arguments name directories ("a" ≡ "a/").
  • Results are closed unions of frozen types, each method returning only the outcomes it can produce: get → Found | Missing, get_if_changed → … | Unchanged, put_if_absent → Written | Rejected, put_if_match → Written | Rejected | Missing, delete_if_match, get_ranges (one generation), head, delete_dir, list, list_dir (Key | Prefix | Entry). Conditional operations are separate methods, and capabilities a backend lacks are type errors.
  • get_into is generic in the buffer type the backend reads into (host Buffer, or a device buffer for GPU direct storage); the caller allocates.
  • Batches: get_many/get_into_many (lazy, as reads finish) and put_many/delete_many (eager, in order); a failure raises BatchError with what completed and what remains. Backends may run batches natively; Store otherwise runs them serially.
  • Backends: MemoryBackend; LocalBackend (staged writes renamed into place under per-directory locks taken ancestor before descendant; unique nanosecond mtime stamps so generations don't repeat on filesystems that reuse inodes; if-match operations refused where file times are rounded); ObjectStoreBackend over obstore.
Testing
  • zarr_storage.testing.StoreSuite plus capability mixins (ConditionalPutSuite, KeyAndDirectorySuite, UniqueGenerationsSuite, ConditionalRaceSuite, ...) and refusing counterparts. Every test runs through Store and again through an AsyncStore over the same backend.
  • StoreSuite.test_state_machine: a Hypothesis state machine that drives random operations, batches included, through random views against a dict model, including retired-generation replays.
  • Race tests, a tripwire that fails any blocking obstore call made on an event-loop thread, forced-interleaving tests for the local backend's locking, and unit tests for the batch runners' windowing and failure rule.
  • 1741 tests pass on Python 3.11 and free-threaded 3.14 (macOS) and in a Linux container (python:3.12-slim); pyright strict and ruff are clean.
Stack

This PR targets the fork's topic branch topic/zarr-storage (cut from the fork's main at b0020f2e2). Follow-up PRs stack on the topic branch; the whole topic goes upstream later as one PR, with the changelog fragment renumbered then. Stacked on this: #464, shared resources (ThreadPool, Limiter) modeled on TensorStore's context resources. Planned next: a CI workflow for the package; concurrent sync batches on obstore; get_ranges_into; fsspec and zip backends; composition (retry, caching, range coalescing); transactions.

Not in this PR
  • The follow-ups listed under "Stack".
  • Moving the current store API into zarr_storage.legacy, the adapter in zarr, and the hierarchy layer; see the storage plan.
  • No Windows run (the Windows name rules are unit-tested only); no CI workflow for the package yet.

🤖 Generated with Claude Code

…ore API

Add `packages/zarr-storage`, a key-value storage layer that knows nothing
about the Zarr spec. `DESIGN.md` is the contract.

- `Store` is a view of a backend at a root: the one place that checks the
  key grammar, applies the root, and enforces read-only and closed states.
  `store / "a"` returns a view on the same backend; the constructed store
  owns the backend's lifecycle; there is no value equality
  (`shares_backend`, `overlaps` instead).
- Prefix arguments name directories; `list_dir` yields `Key`, `Prefix` or
  `Entry`; `delete` removes one key and `delete_dir` a directory.
- Reads return `None` for a missing key and take byte ranges (range,
  offset, suffix); `get_into` reads into a caller's buffer. Writes and
  deletes take `if_match` / `if_none_match` and report `applied=False`
  instead of raising. Every method has an `_async` twin.
- Backends: `MemoryBackend`, `LocalBackend` (staged writes renamed into
  place under per-directory locks, unique nanosecond mtime stamps so
  generations don't repeat on filesystems that reuse inodes), and
  `ObjectStoreBackend` over obstore.
- `zarr_storage.testing`: a conformance suite with capability mixins, run
  through both the sync and async methods, plus a Hypothesis state machine
  that checks random operations through random views against a model.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…l operations

Reads return `Found | Missing` (and `FoundRanges`, `Copied`) instead of
`ReadResult | None`, so a missing key is a value an HTTP 404 maps onto.
`get_if_changed` adds conditional reads that return `Unchanged` without
reading bytes (HTTP 304, obstore `If-None-Match`).

Conditional writes become their own methods: `put` returns `Written`,
`put_if_absent` returns `Written | Rejected`, `put_if_match` and
`delete_if_match` live in the `WritableIfMatch` and `DeletableIfMatch`
capability protocols. Each method returns only the outcomes it can
produce, and `delete_if_match` on an object store is now a type error
rather than a runtime refusal. Backends share one implementation per
operation, parameterized by a precondition function.

Assisted-by: ClaudeCode:claude-opus-5-5
…s into

`ReadableInto` and `ReadableIntoAsync` are now generic in the type of
buffer the backend writes into, and `Store.get_into` accepts exactly that
type. Host backends are `ReadableInto[Buffer]` and convert the buffer
themselves; a GPU direct storage backend can be `ReadableInto` of its
device buffer type, so passing it host memory (or the reverse) is a type
error. The caller allocates, so the store needs no allocator.

`Store.get_into` no longer falls back to `get` plus a copy for backends
without `get_into`: the method is a type error on them. The obstore
backend gains `get_into` (a read, then a copy), using the shared
`copy_into` helper.

Assisted-by: ClaudeCode:claude-opus-5-5
Store gains get_many, get_into_many, put_many and delete_many, with async
twins. A batch is a loop of the single operation run concurrently where
the backend can, with no atomicity: reads are lazy and yield
(request, result) pairs as they finish; writes are eager and return
results in the order given. Keys are checked before any operation
starts, and a write batch naming a key twice raises ValueError. After a
failure a batch starts nothing new, lets running operations finish, and
raises BatchError (an ExceptionGroup) whose completed and remaining
fields survive except*.

Store runs any backend's batches serially; the optional ReadableMany,
ReadableIntoMany[O] and WritableMany capabilities (and async twins) let
a backend run them its own way. LocalBackend runs them on its thread
pool, and ObjectStoreBackend's async batches keep `concurrency`
requests in flight. The shared runners live in zarr_storage._batch.

The state machine's new batch rules found a race in ObjectStoreBackend:
a read or delete under a file, concurrent with deleting that file,
raised the store's raw error. A refusal carrying ENOTDIR is now read as
an absent key directly, instead of by re-checking the parents.

Assisted-by: ClaudeCode:claude-opus-5-5
Store keeps the sync operations; the new AsyncStore offers the same
operations under the same names as coroutines and async iterators
(`await astore.get(k)`, `async for k in astore.list()`), so neither
class carries the other's methods. Both derive from a private _View that
holds the backend, root, read-only flag and ownership, and does key
validation, root joining and read-only checks once. `store.as_async()`
and `astore.as_sync()` are non-owning views on the same backend.

Lifecycle differs between the two: Store has close() and `with`;
AsyncStore has `await aclose()` and `async with`, which awaits a
ClosableAsync backend's aclose (new capability, for cleanup that awaits a
client session) and calls close on any other. LocalBackend gains aclose,
running its pool shutdown on a worker thread.

BREAKING CHANGE: Store's `*_async` methods are gone; use
`store.as_async()` or construct an AsyncStore.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b changed the base branch from main to topic/zarr-storage November 18, 2026 09:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant