resolve

blobmap.resolve

Reading the format: object key to blob id.

This lives in blobmap rather than in the consuming service because it is the semantics of the manifest. If a consumer reimplements the parse rules, the format and its interpretation drift, and the failure mode is silent misattribution of chunks to blobs.

The trie holds one node per cut, never per object. A store with 300,000 objects contributes a handful of nodes, and a bucketed array is a single node however many blobs it spans, because the id is arithmetic rather than an entry. A petabyte at 100 GB blobs is single digit megabytes of Python dicts.

Example

from blobmap.model import Blob, Bucket, Manifest manifest = Manifest("cordex/a.zarr", ... (Blob("b_tas", ("tas/c",), ... Bucket(0, 2048, "v3_slash")),), ... ("**/zarr.json",)) trie = Trie() trie.add(manifest) trie.lookup("cordex/a.zarr/tas/c/5000/0/0").blob_id 'b_tas_2' trie.lookup("cordex/a.zarr/zarr.json").kind 'hot' trie.lookup("somewhere/else/entirely").kind 'unmanaged'

Resolution dataclass

Resolution(
    kind: str,
    blob_id: str | None = None,
    blob: Blob | None = None,
    bucket_index: int | None = None,
)

What a key resolves to.

Attributes:

Name	Type	Description
kind	str	One of blob, hot, pinned or unmanaged. hot means a metadata object or coordinate, held back because of what it is. pinned means someone deliberately asked for it to stay on disk; it is reported separately so an operator can tell a structural decision from a human one. unmanaged means nothing claims this key, which is normal and safe rather than an error.
blob_id	str | None	The concrete id, such as b_tas_2, or None unless kind is blob. This is the key against blobtier's state table.
blob	Blob | None	The definition that matched, for callers that need its prefixes or bucket.
bucket_index	int | None	Which bucket the key fell into, 0 for an unbucketed blob.

Example

Resolution("hot").archivable False

archivable property

archivable: bool

Whether this key may be moved to tape.

Returns:

Type	Description
bool	True only for kind == "blob". Metadata, coordinates and
bool	unmanaged keys all stay hot.

Trie

Trie()

Segment-wise longest prefix match over all known manifests.

Deepest declaration wins, so a store that outgrows a parent scope manifest and gets its own simply overrides it for that subtree. There is no special case in the lookup and nesting needs no validation.

Build once at startup from ManifestStore.load_all, then rebind atomically on reload. In-flight lookups finish against the old trie and the next one sees the new, so no lock is needed.

Example

from blobmap.model import Blob, Manifest trie = Trie() trie.add_all([Manifest("a", (Blob("b_a", ("x",)),), ()), ... Manifest("a/x", (Blob("b_deep", ("y",)),), ())]) trie.lookup("a/x/y/0").blob_id # deepest declaration wins 'b_deep_0'

Source code in src/blobmap/resolve.py

def __init__(self) -> None:
    self._root: dict[str, Any] = {}
    # pins live in their own trie because they must win regardless of
    # depth. A longest-prefix match over a single trie would let a blob at
    # `tas/c` beat a pin on `tas`, which is exactly the case pinning
    # exists for.
    self._pins: dict[str, Any] = {}
    self._hot_basenames: set[str] = set(METADATA_BASENAMES)
    self._epochs: dict[str, int] = {}

epochs property

epochs: dict[str, int]

Epoch of every indexed scope.

Returns:

Type	Description
dict[str, int]	Mapping of scope to epoch, for deciding whether a reload is
dict[str, int]	needed without diffing the definitions.

add

add(manifest: Manifest) -> None

Insert one manifest's declarations.

Parameters:

Name	Type	Description	Default
manifest	Manifest	The manifest to index. Its scope is prepended to every prefix, so keys are matched absolutely.	required

Source code in src/blobmap/resolve.py

def add(self, manifest: Manifest) -> None:
    """Insert one manifest's declarations.

    Args:
        manifest: The manifest to index. Its scope is prepended to every
            prefix, so keys are matched absolutely.
    """
    scope = manifest.scope.strip("/")
    self._epochs[scope] = manifest.epoch

    for pattern in manifest.hot_always:
        if pattern.startswith("**/"):
            # a bare basename: cheaper to check than to walk
            self._hot_basenames.add(pattern[3:])
        else:
            self._insert(_join(scope, pattern.rstrip("*/")), HOT)

    for blob in manifest.blobs:
        for prefix in blob.prefixes:
            self._insert(_join(scope, prefix), blob)

    for pin in manifest.pinned:
        self._insert(_join(scope, pin.prefix), PINNED, self._pins)

add_all

add_all(manifests: Iterable[Manifest]) -> None

Insert many manifests.

Parameters:

Name	Type	Description	Default
manifests	Iterable[Manifest]	Usually the result of ManifestStore.load_all().	required

Source code in src/blobmap/resolve.py

def add_all(self, manifests: Iterable[Manifest]) -> None:
    """Insert many manifests.

    Args:
        manifests: Usually the result of `ManifestStore.load_all()`.
    """
    for m in manifests:
        self.add(m)

lookup

lookup(key: str) -> Resolution

Map an absolute object key to a blob.

Misses are normal and safe. An unpartitioned store, foreign data or a brand new upload resolves to unmanaged, which means hot. No database call and no exception. Only registered blobs are archivable, so a miss is the conservative default rather than something to handle.

Parameters:

Name	Type	Description	Default
key	str	Full object key, including the scope prefix.	required

Returns:

Type	Description
Resolution	A Resolution. Never None.

Example

from blobmap.model import Blob, Bucket, Manifest trie = Trie() trie.add(Manifest("s", (Blob("b_tas", ("tas/c",), ... Bucket(0, 100, "v3_slash")),), ())) trie.lookup("s/tas/c/250/0/0").bucket_index 2 trie.lookup("s/tas/c/notanumber/0").kind 'unmanaged'

Source code in src/blobmap/resolve.py

def lookup(self, key: str) -> Resolution:
    """Map an absolute object key to a blob.

    Misses are normal and safe. An unpartitioned store, foreign data or a
    brand new upload resolves to unmanaged, which means hot. No database
    call and no exception. Only registered blobs are archivable, so a
    miss is the conservative default rather than something to handle.

    Args:
        key: Full object key, including the scope prefix.

    Returns:
        A `Resolution`. Never `None`.

    Example:
        >>> from blobmap.model import Blob, Bucket, Manifest
        >>> trie = Trie()
        >>> trie.add(Manifest("s", (Blob("b_tas", ("tas/c",),
        ...                              Bucket(0, 100, "v3_slash")),), ()))
        >>> trie.lookup("s/tas/c/250/0/0").bucket_index
        2
        >>> trie.lookup("s/tas/c/notanumber/0").kind
        'unmanaged'
    """
    parts = _split(key)
    if not parts:
        return UNMANAGED
    if parts[-1] in self._hot_basenames:
        return HOT
    if self._pins and _covered(self._pins, parts):
        return PINNED

    node: dict[str, Any] | None = self._root
    best: Any = None
    best_depth = 0
    for depth, part in enumerate(parts, start=1):
        assert node is not None
        node = node.get(part)
        if node is None:
            break
        if _MARK in node:
            best, best_depth = node[_MARK], depth

    if best is None:
        return UNMANAGED
    if isinstance(best, Resolution):
        return best

    blob: Blob = best
    n = _bucket_index(blob.bucket, parts, best_depth)
    if n is None:
        return UNMANAGED
    return Resolution("blob", blob.instance(n), blob, n)

resolve

resolve(manifest: Manifest, key: str) -> Resolution

Resolve one key against one manifest.

Builds a throwaway trie, so this is for tests and one-off inspection. A service should build a Trie once and reuse it.

Parameters:

Name	Type	Description	Default
manifest	Manifest	The manifest to resolve against.	required
key	str	Key relative to the manifest scope, unlike Trie.lookup which takes an absolute key.	required

Returns:

Type	Description
Resolution	A Resolution.

Example

from blobmap.model import Blob, Manifest resolve(Manifest("s", (Blob("b_pr", ("pr/c",)),), ()), ... "pr/c/7/0/0").blob_id 'b_pr_0'

Source code in src/blobmap/resolve.py

def resolve(manifest: Manifest, key: str) -> Resolution:
    """Resolve one key against one manifest.

    Builds a throwaway trie, so this is for tests and one-off inspection. A
    service should build a [`Trie`][blobmap.resolve.Trie] once and reuse it.

    Args:
        manifest: The manifest to resolve against.
        key: Key *relative to the manifest scope*, unlike
            [`Trie.lookup`][blobmap.resolve.Trie.lookup] which takes an
            absolute key.

    Returns:
        A `Resolution`.

    Example:
        >>> from blobmap.model import Blob, Manifest
        >>> resolve(Manifest("s", (Blob("b_pr", ("pr/c",)),), ()),
        ...         "pr/c/7/0/0").blob_id
        'b_pr_0'
    """
    trie = Trie()
    trie.add(manifest)
    return trie.lookup(_join(manifest.scope.strip("/"), key))