Skip to content

feat(zarr-metadata)!: define codecs by kind and read each data type's number class - #445

Open
d-v-b wants to merge 15 commits into
mainfrom
feat/zarr-metadata-codec-data-type-support
Open

d-v-b wants to merge 15 commits into
mainfrom
feat/zarr-metadata-codec-data-type-support

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Breaking: a v3 codec is defined with the class of its kind, ArrayArrayCodecDefinition, ArrayBytesCodecDefinition or BytesBytesCodecDefinition, instead of CodecDefinition(kind=...). The kind is now the class's kind property, and CodecDefinition is their abstract base, which a scope and resolve still use to name the kind. A codec that receives an array, an ArrayCodecDefinition, has chunk_rules and pipelines, an array -> array codec also has transition, and a bytes -> bytes codec, which receives bytes, has none of them, so building one with any of them is a TypeError from the class itself.

A v3 data type definition declares what kind of numbers its values are with number_class, a NumberClass: "integral", "floating_point", "complex" or "non_numeric". cast_value and scale_offset read it to decide which data types they accept, where each kept a list of data type names, so a data type defined elsewhere is judged by what it declares: both codecs refuse one that declares complex or non-numeric values, and cast_value refuses to wrap to one that declares floating-point values. A data type that declares no number class is accepted by both, as before. A data type defined elsewhere under one of the float names the cast_value spec lists, such as bfloat16, was refused as a wrap target by its name, and is now refused once it declares "floating_point".

Breaking: cast_value and scale_offset refuse the data types their specs do not list. Each spec names the integer and floating-point types as "the following data types defined in this repository" that it supports, and that repository also defines the numpy time types and the complex types. So both codecs now refuse numpy.datetime64 and numpy.timedelta64, and cast_value refuses them as the data type it casts to, and scale_offset now refuses complex64 and complex128. All of these were accepted. The time types' own spec says they are "compatible with any codec that supports arrays of signed 64-bit integers", which the two codecs' specs do not bear out. zarr_metadata.testing.strategies no longer draws these combinations.

Agent summary

Three commits, each of which passes just check on its own.

1. One class per codec kind (3348a561d)

  • CodecDefinition was built with kind="array_array" | "array_bytes" | "bytes_bytes", and a table (_UNASKED) refused the members a codec of a kind is never asked: the chunk rules, transition and pipelines of a bytes -> bytes codec, and the transition of an array -> bytes codec.
  • Each kind is now a class. ArrayCodecDefinition holds chunk_rules and pipelines, ArrayArrayCodecDefinition adds transition, and BytesBytesCodecDefinition has none of them. CodecDefinition is an abstract base (ABC, with kind an abstract property), and it is still the kind that scopes and resolve name.
  • The pipeline and the strategies dispatch on the class instead of comparing kind strings, so pyright sees which codecs have which members.
  • A definition's repr shows its class, as in BytesBytesCodecDefinition(name='gzip'), so the error messages that include it changed.

2. number_class on data type definitions (8d8e730ec)

  • cast_value and scale_offset decided which data types they accept from three sets holding 25 data type names, 11 of them float types this package does not define, kept so that wrap could refuse them.
  • A data type now declares a property of itself, as it does storage, and the two codecs' own rules read it. Their specs state the criterion the same way ("data types that model real numbers", "an integral type").
  • Neither a codec nor a data type names the other, so an extension codec can read the class of a core data type, and an extension data type is judged by what it declares.
  • None, the default, means unknown, and both codecs accept it, as the bytes codec accepts an unknown storage.

3. Verdicts that follow the codec specs (28e0bfd0a)

  • The time types declare "non_numeric", and scale_offset refuses the same classes as cast_value, so one set of refused classes serves both codecs.
  • scale_offset's message is now "expected a chunk of an integer or floating-point data type, got ...", since "a data type with arithmetic" would be wrong for complex numbers.
Design notes
  • Rejected: pairwise opt-in, as zarrs does it. A first version gave data types a supported_codecs set of codec names and codecs a requires_data_type_support flag, modeled on zarrs's define_data_type_support!. In zarrs the per-codec trait carries behavior and properties (cast_value_is_integral()), and either crate can add a pair. As a name list it lost both: cast_value still needed a list of float names for wrap, and an extension codec could not accept core int8 without redefining it. The agent replaced it after a review of the design.
  • An inference about complex numbers. The scale_offset spec's list covers only the data types of its own repository, and it leaves out all 13 complex types defined there. The agent read that as "no complex numbers", so a data type defined elsewhere that declares "complex" is refused by scale_offset too. The alternative is to refuse only complex64 and complex128 by name.
  • zarrs. zarrs registers only the integer and float types for cast_value, so it gives the same verdict for the time types. It has no scale_offset. The agent read the zarrs source at db0f278 and did not run it, and did not check tensorstore.
  • Not raised upstream. The time types' spec in zarr-extensions still says they are compatible with any codec that supports arrays of signed 64-bit integers, which the two codec specs contradict.
Testing
  • tests/v3/test_number_class.py: a table pins the number class of each of the 24 data types this package defines. A second table runs an extension data type of each class, and one that declares none, through both codecs, as input and as cast_value's target with and without wrap, and checks where each problem is reported and its message. One error test covers a number_class that is not one of the values.
  • tests/v3/test_pipelines.py: complex and both time types joined the table of data types each arithmetic codec refuses. The two cases that pinned the old acceptance are removed.
  • tests/testing/test_strategies.py: the test that reached complex scale_offset values is removed, and the tests of complex scales and time-type cast targets lost those cases.
  • The tests of codec kinds build each kind by its class, and check that the base class refuses to be built and that a kind refuses a member it does not have.
  • Locally, just check passes on the head commit: ruff, pyright (0 errors), the suite on Python 3.11 and on the newest supported version, and mkdocs build --strict.
Stack and bookkeeping
  • This branch was built on the types-only modules branch, which merged upstream as zarr-developers/zarr-python#4627 (squash commit 901290de8). The fork's main was synced to upstream, and upstream main was merged into this branch (22e65d51b). The squash has the same tree as this branch's old base, so the merge kept this branch's side of the nine conflicting codec modules and changed no file.
  • The diff now holds only this PR's changes, 58 files. The commit list still shows the four commits of the merged branch (53d57d210, bcc20c872, c12e4e405, eb3061e05), since main holds them as one squash; they add nothing to the diff. The commits that belong to this PR are 3348a561d, 8d8e730ec, 28e0bfd0a and 199ceb595.
  • The same branch is open upstream as feat(zarr-metadata)!: define codecs by kind and read each data type's number class zarr-developers/zarr-python#4628, and the changelog fragments are numbered after it (4628.feature.md, 4628.bugfix.md).

🤖 Generated with Claude Code

Every extension module held its types (name constants, `Literal` names,
configuration and object `TypedDict`s, metadata aliases) beside its
definition. Codec modules imported data type modules at run time, so a
data type that needed a codec's types could import them only under
`TYPE_CHECKING`.

The types now live in modules that hold types only and import no
definition: one module per kind under `v3/_types/` (codec, data_type,
chunk_grid, chunk_key_encoding, consolidated) and `v2/_types/` (codec,
data_type). Their packages' `__init__`s hold no definitions, so importing
a types module never runs one. `EmptyConfiguration` moves to `_common`,
and `ZarrV2DataTypeDeclaration` to `v2/_declaration_json`.

Rules and other functions stay beside their definition, which imports
its types from the types module. Each public extension module re-exports
its types, so every public import path is unchanged. The codec rules
that name data types (`_arithmetic`, `vlen_utf8`, `vlen_bytes`,
`sharding_indexed`) import those names from the types module.

Assisted-by: ClaudeCode:claude-opus-5-5
Each types-only module is imported in a fresh interpreter in which the
`zarr_metadata`, `zarr_metadata.v2` and `zarr_metadata.v3` packages
(whose `__init__`s import every definition) are empty stand-ins. The
test asserts that no module holding a `Definition` ran. Every other
package `__init__` runs for real, so a types module placed in a package
that exports definitions fails. A control test checks that the probe
does see a definition module when an extension module is imported.

Assisted-by: ClaudeCode:claude-opus-5-5
Brings in zarr-developers#4621, which added `array_bytes_codec_default` and
`fill_value_default` to the data type definitions. Its three
`TYPE_CHECKING` imports of codec types (in `v3/_definition.py`,
`v3/data_type/string.py` and `v3/data_type/_byte.py`) become run-time
imports from `zarr_metadata.v3._types.codec`, and the codec defaults
spell their names with `BYTES_CODEC_NAME`, `VLEN_UTF8_CODEC_NAME` and
`VLEN_BYTES_CODEC_NAME`. The conflicts in the data type modules were
their import blocks: `EmptyConfiguration` comes from `_common`, and
`ConstantFillValue` from `definition._definition`.

Assisted-by: ClaudeCode:claude-opus-5-5
…lass

A codec was a `CodecDefinition(kind=...)`, and a table refused the
members a codec of a kind is never asked: the chunk rules, transition and
pipelines of a bytes -> bytes codec, and the transition of an array ->
bytes codec. Each kind is now a class: `ArrayArrayCodecDefinition`,
`ArrayBytesCodecDefinition` and `BytesBytesCodecDefinition`.
`ArrayCodecDefinition`, the base of the first two, holds `chunk_rules`
and `pipelines`, and only an array -> array codec has `transition`, so a
bytes -> bytes codec has no member that would be about a chunk.
`CodecDefinition` is their abstract base, whose `kind` property each
class gives, and it stays the kind that a scope and `resolve` name.

The pipeline and the strategies dispatch on the class instead of
comparing kind strings, so the type checker sees which codecs have which
members.

Assisted-by: ClaudeCode:claude-opus-5-5
`cast_value` and `scale_offset` decided which data types they accept
from lists of data type names: 25 names across three sets, 11 of them
float types this package does not define, kept only so `wrap` could
refuse them. A data type defined elsewhere was judged by whether its
name happened to be listed.

A `DataTypeDefinition` now declares what kind of numbers its values are
with `number_class`, a `NumberClass`: integral, floating-point, complex
or non-numeric. The two codecs' rules read it, as the `bytes` codec
reads `storage`: `cast_value` refuses complex and non-numeric data types
on either side of the cast and wraps only to one that is not
floating-point, and `scale_offset` refuses non-numeric ones. This is how
their specs state it ("data types that model real numbers", "an integral
type"), and neither a codec nor a data type has to name the other, so an
extension codec can read the class of a core data type and an extension
data type is judged by what it declares.

A data type that declares no number class is accepted, as an unknown
`storage` is, so nothing that was accepted is refused. Each of the 24
data types this package defines declares its class and is accepted or
refused as before, with the same messages.

Assisted-by: ClaudeCode:claude-fable-5-2
…cs do not list

The `cast_value` and `scale_offset` specs each name the integer, unsigned
integer and floating-point types as "the following data types defined in
this repository" that the codec supports. That repository also defines
the numpy time types and the complex types, and neither list names them.
The package accepted the time types for both codecs, on the strength of
the time types' own spec ("compatible with any codec that supports
arrays of signed 64-bit integers"), and accepted the complex types for
`scale_offset`, which its spec was read as leaving open.

The codec's spec says what the codec is defined for, so both codecs now
refuse `numpy.datetime64`, `numpy.timedelta64` and the complex types.
The time types declare the non-numeric number class, and `scale_offset`
refuses the same classes as `cast_value`, so one set serves both. zarrs
gives the same verdict for `cast_value`.

The strategies no longer draw these combinations, and the tests that
reached them are removed or moved to the table of refused data types.

Assisted-by: ClaudeCode:claude-fable-5-2
… after fork PR 445

The fragments take the number of the fork PR for now, and are renumbered
to the upstream PR that merges them.

Assisted-by: ClaudeCode:claude-fable-5-2
Upstream main gained zarr-developers#4627 as squash commit 901290d, whose tree is the
same as this branch's base (eb3061e). The nine conflicts were codec
modules that this branch changed after that base, so each keeps this
branch's version, and the merge leaves the tree unchanged.

Assisted-by: ClaudeCode:claude-opus-5-5
…elopers#4626)

* chore(deps-dev): bump the python-dependencies group across 1 directory with 13 updates

Bumps the python-dependencies group with 13 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [obstore](https://github.com/geospatial-jeff/pyasyncio-benchmark) | `0.12.0` | `0.12.1` |
| [coverage](https://github.com/coveragepy/coveragepy) | `7.16.2` | `7.17.2` |
| [pytest](https://github.com/pytest-dev/pytest) | `9.1.1` | `9.2.0` |
| [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) | `1.4.0` | `1.4.1.post1` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.170.5` | `6.170.14` |
| [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) | `5.3.0` | `5.3.1` |
| [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed) | `5.0.3` | `5.0.4` |
| [uv](https://github.com/astral-sh/uv) | `0.13.5` | `0.13.7` |
| [moto[s3,server]](https://github.com/getmoto/moto) | `5.2.4` | `5.2.5` |
| [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.7` | `9.8.0` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.5` | `1.12.6` |
| [ruff](https://github.com/astral-sh/ruff) | `0.17.1` | `0.17.3` |
| [mypy](https://github.com/python/mypy) | `2.4.0` | `2.4.1` |



Updates `obstore` from 0.12.0 to 0.12.1
- [Commits](https://github.com/geospatial-jeff/pyasyncio-benchmark/commits)

Updates `coverage` from 7.16.2 to 7.17.2
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.16.2...7.17.2)

Updates `pytest` from 9.1.1 to 9.2.0
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@9.1.1...9.2.0)

Updates `pytest-asyncio` from 1.4.0 to 1.4.1.post1
- [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases)
- [Commits](pytest-dev/pytest-asyncio@v1.4.0...v1.4.1.post1)

Updates `hypothesis` from 6.170.5 to 6.170.14
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.170.5...v6.170.14)

Updates `pytest-benchmark` from 5.3.0 to 5.3.1
- [Release notes](https://github.com/ionelmc/pytest-benchmark/releases)
- [Changelog](https://github.com/ionelmc/pytest-benchmark/blob/master/CHANGELOG.rst)
- [Commits](ionelmc/pytest-benchmark@v5.3.0...v5.3.1)

Updates `pytest-codspeed` from 5.0.3 to 5.0.4
- [Release notes](https://github.com/CodSpeedHQ/pytest-codspeed/releases)
- [Changelog](https://github.com/CodSpeedHQ/pytest-codspeed/blob/master/CHANGELOG.md)
- [Commits](CodSpeedHQ/pytest-codspeed@v5.0.3...v5.0.4)

Updates `uv` from 0.13.5 to 0.13.7
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](astral-sh/uv@0.13.5...0.13.7)

Updates `moto[s3,server]` from 5.2.4 to 5.2.5
- [Release notes](https://github.com/getmoto/moto/releases)
- [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md)
- [Commits](getmoto/moto@5.2.4...5.2.5)

Updates `mkdocs-material[imaging]` from 9.7.7 to 9.8.0
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.7.7...9.8.0)

Updates `markdown-exec[ansi]` from 1.12.5 to 1.12.6
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.5...1.12.6)

Updates `ruff` from 0.17.1 to 0.17.3
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.17.1...0.17.3)

Updates `mypy` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v2.4.0...v2.4.1)

---
updated-dependencies:
- dependency-name: obstore
  dependency-version: 0.12.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: coverage
  dependency-version: 7.17.2
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: pytest
  dependency-version: 9.2.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: pytest-asyncio
  dependency-version: 1.4.1.post1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.170.14
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytest-benchmark
  dependency-version: 5.3.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytest-codspeed
  dependency-version: 5.0.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.13.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: moto[s3,server]
  dependency-version: 5.2.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocs-material[imaging]
  dependency-version: 9.8.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.12.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.17.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mypy
  dependency-version: 2.4.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* test: keep the suite running with pytest 9.2 and hypothesis 6.170.7+

pytest 9.2 applies the configured filterwarnings while importing plugins
(pytest#12697), so the "error" filter turns the ast.Num DeprecationWarning
that astor (a dependency of pytest-accept) triggers at import on Python
< 3.14 into a startup failure (exit code 3). Ignore that one astor-scoped
deprecation, which pytest <= 9.1 never routed through the filters.

pytest 9.2 also requires packaging>=24, which conflicts with the min_deps
pin packaging==22.* (the declared runtime floor packaging>=22.0). Keep that
floor tested by overriding pytest to <9.2 in the min_deps env only, via
UV_OVERRIDE and ci/min_deps-overrides.txt; every other env uses pytest 9.2.0.

hypothesis 6.170.7 annotates run_state_machine_as_test as taking
type[RuleBasedStateMachine] (its docstring still accepts any zero-arg
factory), so mypy now reports arg-type instead of no-untyped-call at the
four call sites that pass a factory function. Update the ignore codes;
no test logic changes.

* test: drop the unused pytest-accept plugin

pytest-accept was added in zarr-developers#2589 for a `hatch run doctest:fix` script that
ran `pytest --accept` over the RST doctests. That script and those doctests
are gone, and nothing in the justfile or CI passes `--accept`. The plugin's
only remaining effect was importing astor (last released in 2019) at every
pytest startup, whose `ast.Num` lookup pytest 9.2 turns into a startup error
under our "error" warning filter. Removing the plugin removes astor, so the
astor-scoped filter added for pytest 9.2 is no longer needed.

Assisted-by: ClaudeCode:claude-opus-5-5

* build(deps): require packaging>=24.0

pytest 9.2 requires packaging>=24, so the min_deps env could no longer pin
packaging==22.*. Instead of holding min_deps on pytest<9.2 through a
UV_OVERRIDE file, raise the runtime floor to packaging>=24.0 (released
March 2024) and test that floor in min_deps. zarr uses only
packaging.version.Version and parse, which behave the same on 22 and 24.
min_deps now runs the same pytest as every other env, and the override
file and env var are gone.

Assisted-by: ClaudeCode:claude-opus-5-5

* docs: add a changelog entry for the packaging>=24.0 floor

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: invisiblefix-bot <355452247+invisiblefix-bot@users.noreply.github.com>
PDGGK and others added 3 commits November 12, 2026 21:07
* fix: migrate v2 arrays whose fill_value is null

Zarr v2 allows "fill_value": null, and xarray writes it for variables without a _FillValue. Zarr v3 requires a fill value, so `zarr migrate v3` raised a TypeError on such arrays. Use the data type's default scalar, which is what zarr-python already returns for unwritten chunks of the v2 array.

* docs: add changelog entry for zarr-developers#4616

* test(migrate): cover a partly written chunk in the null fill value test

Write only part of the first chunk, so the test checks that the padding
stored in a partial chunk reads back the same after migration, and compare
the v2 read against the same expected values as the v3 read.

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com>
…not plain numbers

The open zarr-extensions proposals include an optional (nullable) data
type, a fixed-size list and a ragged ndarray. Their values hold numbers
without being numbers, and neither arithmetic codec's spec lists them, so
they belong in `non_numeric`, which said "not numbers at all". It now
says every value that is not a plain number, naming records, lists and
values that may be missing.

Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants