Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changes/4616.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
`zarr migrate v3` now migrates Zarr format 2 arrays whose `fill_value` is
`null`, such as the integer variables and coordinates xarray writes. The v3
metadata uses the data type's default fill value, which is what reading an
unwritten chunk of the v2 array already returned. Before, the migration failed
with `TypeError: Cannot convert object None ... to a scalar`.
1 change: 1 addition & 0 deletions changes/4626.removal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The minimum supported packaging version is now 24.0.
2 changes: 1 addition & 1 deletion ci/big-endian/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ RUN git clone -q --depth 1 --branch 1.1.2 https://github.com/google/crc32c /tmp/

RUN python3 -m venv --system-site-packages /venv
ENV PATH=/venv/bin:$PATH
RUN pip install -q --no-binary google-crc32c google-crc32c msgspec pytest-accept
RUN pip install -q --no-binary google-crc32c google-crc32c msgspec
# Debian sid's python3-dateutil 3.0.0 declares a dependency on the `tzdata` distribution,
# which no Debian package installs, so `pip check` fails on the image as built. tzdata is
# pure Python; install it from PyPI.
Expand Down
3 changes: 0 additions & 3 deletions ci/min_deps-overrides.txt

This file was deleted.

2 changes: 1 addition & 1 deletion docs/user-guide/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Required dependencies include:

- [Python](https://docs.python.org/3/) (3.12 or later)
- [packaging](https://packaging.pypa.io/en/stable/) (22.0 or later)
- [packaging](https://packaging.pypa.io/en/stable/) (24.0 or later)
- [numpy](https://numpy.org) (2.0 or later)
- [numcodecs](https://numcodecs.readthedocs.io/en/stable/) (0.14 or later)
- [google-crc32c](https://github.com/googleapis/python-crc32c) (1.5 or later)
Expand Down
10 changes: 10 additions & 0 deletions packages/zarr-metadata/changes/4628.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
**Breaking:** `cast_value` and `scale_offset` refuse the data types their
specs do not list. Each spec names the integer and floating-point types as
"the following data types defined in this repository" that it supports, and
that repository also defines the numpy time types and the complex types. So
both codecs now refuse `numpy.datetime64` and `numpy.timedelta64`, and
`cast_value` refuses them as the data type it casts to, and `scale_offset`
now refuses `complex64` and `complex128`. All of these were accepted. The time
types' own spec says they are "compatible with any codec that supports arrays
of signed 64-bit integers", which the two codecs' specs do not bear out.
`zarr_metadata.testing.strategies` no longer draws these combinations.
21 changes: 21 additions & 0 deletions packages/zarr-metadata/changes/4628.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
**Breaking:** a v3 codec is defined with the class of its kind,
`ArrayArrayCodecDefinition`, `ArrayBytesCodecDefinition` or
`BytesBytesCodecDefinition`, instead of `CodecDefinition(kind=...)`. The kind
is now the class's `kind` property, and `CodecDefinition` is their abstract
base, which a scope and `resolve` still use to name the kind. A codec that
receives an array, an `ArrayCodecDefinition`, has `chunk_rules` and
`pipelines`, an array -> array codec also has `transition`, and a bytes ->
bytes codec, which receives bytes, has none of them, so building one with any
of them is a `TypeError` from the class itself.

A v3 data type definition declares what kind of numbers its values are with
`number_class`, a `NumberClass`: `"integral"`, `"floating_point"`,
`"complex"` or `"non_numeric"`. `cast_value` and `scale_offset` read it to
decide which data types they accept, where each kept a list of data type
names, so a data type defined elsewhere is judged by what it declares: both
codecs refuse one that declares complex or non-numeric values, and
`cast_value` refuses to `wrap` to one that declares floating-point values. A
data type that declares no number class is accepted by both, as before. A data
type defined elsewhere under one of the float names the `cast_value` spec
lists, such as `bfloat16`, was refused as a `wrap` target by its name, and is
now refused once it declares `"floating_point"`.
22 changes: 14 additions & 8 deletions packages/zarr-metadata/docs/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ from typing_extensions import TypedDict

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context


class AcmeLz4Configuration(TypedDict, closed=True):
Expand All @@ -51,10 +51,9 @@ def acme_lz4_rules(
)


ACME_LZ4 = CodecDefinition(
ACME_LZ4 = BytesBytesCodecDefinition(
name="acme.lz4",
configuration=AcmeLz4Configuration,
kind="bytes_bytes",
size="dynamic",
rules=acme_lz4_rules,
)
Expand Down Expand Up @@ -110,13 +109,20 @@ Other kinds declare more:
`create_default` is given no codecs (`array_bytes_codec_default`, such
as `vlen-utf8` for `string`). A data type whose values vary in size
declares its own, since the default, `bytes`, encodes values of a fixed
size.
size. It declares what kind of numbers its values are (`number_class`:
`"integral"`, `"floating_point"`, `"complex"` or `"non_numeric"`), which
the codecs that are defined only for some kinds of numbers read:
`cast_value` and `scale_offset` refuse a data type that declares
complex or non-numeric values. A data type that does not declare it is
accepted by both.
- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk
lengths (`chunk_lengths`).
- A codec declares:
- what it checks about the chunk it receives (`chunk_rules`),
- what it hands on (`transition`),
- any pipelines it contains (`pipelines`).
- A codec is defined by the class of its kind, `ArrayArrayCodecDefinition`,
`ArrayBytesCodecDefinition` or `BytesBytesCodecDefinition`. A codec that
receives an array declares what it checks about the chunk it receives
(`chunk_rules`) and any pipelines it contains (`pipelines`), and an
array -> array codec also declares what it hands on (`transition`). A
bytes -> bytes codec receives bytes, so it declares none of these.

The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring
describes each of these.
6 changes: 3 additions & 3 deletions packages/zarr-metadata/docs/parsing-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,12 +258,12 @@ To make an unknown data type or codec known, add its definition to the context's

```python test="true" session="parsing-validation"
from zarr_metadata.definition import EmptyConfiguration
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context

# The definition of the codec that was unknown: here, a codec named
# `acme.codec` that takes no configuration.
MY_CODEC = CodecDefinition(
name="acme.codec", configuration=EmptyConfiguration, kind="bytes_bytes", size="dynamic"
MY_CODEC = BytesBytesCodecDefinition(
name="acme.codec", configuration=EmptyConfiguration, size="dynamic"
)
context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[MY_CODEC]))
reading = read_array_metadata_v3(raw, context=context)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@

resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}},
CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)
resolved # RefusedDeclaration(..., definition=CodecDefinition(name='gzip'), ...)
resolved # RefusedDeclaration(..., definition=BytesBytesCodecDefinition(name='gzip'), ...)
problems[0].loc # ('configuration', 'level')
problems[0].input # 12
dict(problems[0].ctx) # {'ge': 0, 'le': 9}
Expand Down Expand Up @@ -125,7 +125,7 @@

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context


class AcmeLz4Configuration(TypedDict, closed=True):
Expand All @@ -143,10 +143,9 @@ def acme_lz4_rules(
)


ACME_LZ4 = CodecDefinition(
ACME_LZ4 = BytesBytesCodecDefinition(
name="acme.lz4",
configuration=AcmeLz4Configuration,
kind="bytes_bytes",
size="dynamic",
rules=acme_lz4_rules,
)
Expand Down
66 changes: 30 additions & 36 deletions packages/zarr-metadata/src/zarr_metadata/testing/_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
from zarr_metadata._common import EmptyConfiguration
from zarr_metadata.definition._definition import AcceptedDeclaration
from zarr_metadata.testing._data_type import (
COMPLEX_WIDTHS,
FLOAT_WIDTHS,
INTEGER_RANGES,
float_fill_values,
Expand All @@ -35,10 +34,11 @@
fill_values_of,
pipelines_for,
read,
scope_of,
)
from zarr_metadata.testing._strategy import EMPTY_CONFIGURATIONS, MAX_RANK, CodecStrategy
from zarr_metadata.v3._definition import Chunk, DataTypeDefinition, storage_of
from zarr_metadata.v3.codec._arithmetic import FLOATING_POINT, NO_REAL_NUMBERS, NOT_NUMBERS
from zarr_metadata.v3.codec._arithmetic import NO_REAL_NUMBERS
from zarr_metadata.v3.codec.blosc import (
BLOSC_CNAME,
BLOSC_CODEC,
Expand Down Expand Up @@ -72,16 +72,20 @@
ZSTD_MIN_LEVEL,
ZstdCodecConfiguration,
)
from zarr_metadata.v3.data_type.numpy_datetime64 import NUMPY_DATETIME64_DATA_TYPE
from zarr_metadata.v3.data_type.numpy_timedelta64 import NUMPY_TIMEDELTA64_DATA_TYPE

if TYPE_CHECKING:
from collections.abc import Mapping

from zarr_metadata._common import JSONValue
from zarr_metadata.definition._definition import ResolvedDeclaration
from zarr_metadata.testing._strategy import Configurations, Strategies
from zarr_metadata.v3._definition import CodecDefinition, CodecSize, Lengths
from zarr_metadata.v3._definition import (
ArrayBytesCodecDefinition,
CodecDefinition,
CodecSize,
Lengths,
NumberClass,
)

_BLOSC_MAX_TYPESIZE: Final = 255
"""The widest element c-blosc takes: `BLOSC_MAX_TYPESIZE`."""
Expand Down Expand Up @@ -109,12 +113,6 @@
_SMALL_NUMBERS: Final = st.integers(0, 127)
"""Values every integral and floating-point data type holds: for a chunk whose data type is not known."""

_NUMPY_TIMES: Final = frozenset({NUMPY_DATETIME64_DATA_TYPE, NUMPY_TIMEDELTA64_DATA_TYPE})
"""The numpy time types, each stored as a signed 64-bit integer."""

_NUMPY_TIME_RANGE: Final = (-(2**63) + 1, 2**63 - 1)
"""The signed 64-bit integers but the least, which numpy reads as `NaT`: the finite numpy times."""


@st.composite
def _endians(draw: st.DrawFn, *, required: bool) -> BytesCodecConfiguration:
Expand Down Expand Up @@ -183,18 +181,14 @@ def _offsets_and_scales(
def _scale_offset(
chunk: Chunk, _strategies: Strategies
) -> st.SearchStrategy[ScaleOffsetCodecConfiguration] | None:
"""An `offset` and a `scale`, each a value of the data type the codec is handed, the scale not zero; None for a data type whose values are not numbers.
"""An `offset` and a `scale`, each a value of the data type the codec is handed, the scale not zero; None for a data type whose values are not integers or floating-point numbers.

Each is a fill value of the chunk's data type
(https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/codecs/scale_offset/README.md?plain=1#L39-L43),
so a float's is in every spelling a fill value takes, a `NaN` and an
infinity among them, and a complex number's is a pair of them. The
scale is drawn non-zero in any spelling, so that decoding never divides
by zero; a scale small enough that decoding overflows is still drawn. A
numpy time is drawn as the signed 64-bit integer it is stored as, never
`NaT`, since it "is compatible with any codec that supports arrays of
signed 64-bit integers"
(https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/numpy.datetime64/README.md?plain=1#L120).
infinity among them. The scale is drawn non-zero in any spelling, so
that decoding never divides by zero; a scale small enough that decoding
overflows is still drawn.
A data type this package does not define, whose values it cannot
draw, takes only the configuration with neither, the identity.
Handed a chunk whose data type is unknown, the values are small
Expand All @@ -205,22 +199,15 @@ def _scale_offset(
if definition is None:
offset: st.SearchStrategy[JSONValue] = _SMALL_NUMBERS
scale: st.SearchStrategy[JSONValue] = st.integers(1, 127)
elif definition.name in NOT_NUMBERS:
elif definition.number_class in NO_REAL_NUMBERS:
return None
elif definition in INTEGER_RANGES or definition in _NUMPY_TIMES:
low, high = INTEGER_RANGES.get(definition, _NUMPY_TIME_RANGE)
elif definition in INTEGER_RANGES:
low, high = INTEGER_RANGES[definition]
offset = st.integers(low, high)
scale = st.integers(low, high).filter(lambda value: value != 0)
elif definition in FLOAT_WIDTHS:
offset = float_fill_values(FLOAT_WIDTHS[definition])
scale = offset.filter(lambda value: not is_zero_float(value))
elif definition in COMPLEX_WIDTHS:
part = float_fill_values(COMPLEX_WIDTHS[definition])
offset = st.tuples(part, part)
# Non-zero in either part is a non-zero complex number.
scale = st.tuples(part, part).filter(
lambda pair: not (is_zero_float(pair[0]) and is_zero_float(pair[1]))
)
else:
return st.builds(ScaleOffsetCodecConfiguration)
return _offsets_and_scales(offset, scale)
Expand All @@ -235,7 +222,7 @@ def _zstd(draw: st.DrawFn) -> ZstdCodecConfiguration:
return configuration


def _vlen(definition: CodecDefinition[Any]) -> CodecStrategy:
def _vlen(definition: ArrayBytesCodecDefinition[Any]) -> CodecStrategy:
"""A codec with nothing to configure that takes only some data types: none for a chunk of another, which its own chunk rules say."""
return CodecStrategy(
configurations=lambda chunk, _strategies: (
Expand All @@ -251,9 +238,11 @@ def _unchunked(configurations: Configurations) -> CodecStrategy:
return CodecStrategy(configurations=lambda _chunk, _strategies: configurations)


def _name_read(data_type: ResolvedDeclaration[DataTypeDefinition[Any]] | None) -> str | None:
"""The name of the definition that accepted `data_type`; None when none did."""
return data_type.definition.name if isinstance(data_type, AcceptedDeclaration) else None
def _number_class_read(
data_type: ResolvedDeclaration[DataTypeDefinition[Any]] | None,
) -> NumberClass | None:
"""The number class the definition that accepted `data_type` declares; None when none accepted it, or it declares none."""
return data_type.definition.number_class if isinstance(data_type, AcceptedDeclaration) else None


def _source_scalars(chunk: Chunk, strategies: Strategies) -> st.SearchStrategy[JSONValue]:
Expand Down Expand Up @@ -294,7 +283,7 @@ def _cast_value_configuration(
declaration, DataTypeDefinition, strategies
)
modes: tuple[CastOutOfRangeMode, ...] = CAST_OUT_OF_RANGE_MODE
if _name_read(target) in FLOATING_POINT:
if _number_class_read(target) == "floating_point":
modes = tuple(mode for mode in modes if mode != "wrap")
configuration = CastValueCodecConfiguration(data_type=declaration)
if draw(st.booleans()):
Expand All @@ -320,9 +309,14 @@ def _cast_value(
of its direction; `wrap` is drawn only for a type that is not
floating-point.
"""
if _name_read(chunk.data_type) in NO_REAL_NUMBERS:
if _number_class_read(chunk.data_type) in NO_REAL_NUMBERS:
return None
targets = [name for name in data_type_names(strategies) if name not in NO_REAL_NUMBERS]
data_types = scope_of(strategies).data_types
targets = [
name
for name in data_type_names(strategies)
if data_types[name].number_class not in NO_REAL_NUMBERS
]
if len(targets) == 0:
return None
return _cast_value_configuration(targets, _source_scalars(chunk, strategies), strategies)
Expand Down
Loading
Loading