Skip to content

abilian_cdn.storage

Where bytes live and how they get there.

storage.objects

abilian_cdn.storage.objects

Putting an object into the store, and finding what a zone holds.

Both faces and the browser land here rather than ordering the store, the cache and the index for themselves: the order is what keeps a crash from leaving a row that points at nothing, and one decided in three places would eventually be decided differently in three places.

PayloadTooLargeError

Bases: ClientException

An upload past the configured ceiling.

IncomingBody

Bases: Protocol

What spooling needs from a request: a declared size, and the bytes.

A protocol rather than Request because that is the whole of it — which is also what makes the size ceiling testable without a web server.

Declared dataclass

What the uploader said about an object, as opposed to its bytes.

One argument rather than three, because they arrive together, are stored together, and every upload path has to pass all of them — a signature that grows a parameter per header is one that quietly loses one.

spool_to_disk async 

spool_to_disk(
    request: IncomingBody, destination: Path, limit: int
) -> int

Write the request body to disk, refusing anything past limit.

Declared length is checked first so an oversized upload is refused before it is transferred, and the running total is checked too, because the declared length is the client's claim rather than a fact.

store_object async 

store_object(
    session: AsyncSession,
    blobs: Blobs,
    *,
    zone: Zone,
    path: str,
    body: IncomingBody,
    declared: Declared,
    limit: int,
    actor: Actor,
) -> tuple[StoredObject, bool]

Put one object in the store and in the index, in that order.

The API and the web UI both land here — two upload paths that each ordered these steps for themselves would eventually order them differently, and the difference only shows up as an orphan after a crash.

Returns the object and whether it was created rather than replaced.

paths_under async 

paths_under(
    session: AsyncSession,
    zone: Zone,
    blobs: Blobs,
    prefix: str,
) -> list[str]

Every path of this zone under a prefix, from wherever the zone lists them.

A storage zone is listed by the index, which knows its paths whether they are cached or not. A pull zone has no index — what it mirrors is somebody else's — so the only paths it can name are the ones it has cached.

storage.blobs

abilian_cdn.storage.blobs

Where bytes go, and in what order.

The service keeps every object in two places: S3, which is durable, and the local cache, which is fast and disposable. Both the ingest path and the read path have to touch both, and the order they touch them in is what keeps a crash from producing a database row that points at bytes nobody has. That order is decided once, here, rather than in each handler.

Blobs

The cache and the object store, kept in step.

scratch_file 

scratch_file(name: str) -> Path

Somewhere to spool an upload before it is published anywhere.

publish async 

publish(
    zone_id: UUID,
    path: str,
    source: Path,
    content_type: str,
) -> StoredObject

Put an uploaded file into S3, then into the cache.

S3 first, deliberately: if the process dies between the two, the worst outcome is bytes in S3 that no row references — an orphan a sweeper can find. The other order produces a row pointing at nothing, which every reader trips over and no sweeper can repair.

The caller writes the database row after this returns, for the same reason.

copy async 

copy(
    zone_id: UUID,
    source_key: str,
    path: str,
    content_type: str,
) -> StoredObject

Copy one object onto another path, inside the store.

The same order as a write: the store first, then the local copy of wherever it landed is dropped, and the caller writes the row last. A cached copy of the destination is the one thing that would otherwise outlive what it describes.

ensure_local async 

ensure_local(
    zone_id: UUID, obj: StoredObject
) -> tuple[Path, str]

The object's bytes on local disk, with where they came from.

Returns the cache status the response reports: HIT when the cache already had it, MISS when S3 had to be asked.

remove async 

remove(zone_id: UUID, path: str, storage_key: str) -> None

Drop an object from both places.

Cache first: a stale cached copy served after the object is gone is a correctness bug, while a moment's extra miss is not.

purge 

purge(zone_id: UUID, paths: Sequence[str]) -> int

Drop local copies of these paths, and nothing else.

A purge is about the cache: a storage zone keeps its objects in S3 and the next request fetches them again, and a pull zone goes back to the origin. Neither loses anything by being purged, which is what makes it safe to offer as a button.

purge_everything 

purge_everything(zone_id: UUID) -> int

Drop everything this zone has cached.

cached_paths 

cached_paths(zone_id: UUID, prefix: str) -> list[str]

Which cached paths of a zone start with prefix.

cached_entries 

cached_entries(zone_id: UUID) -> Iterator[Entry]

What a zone is holding, as the cache describes it.

storage_key staticmethod 

storage_key(zone_id: UUID, path: str) -> str

Where an object lives in the bucket.

Keyed by zone id rather than slug so renaming a zone does not mean moving every object it holds.

storage.bucket

abilian_cdn.storage.bucket

S3 object storage — the durable half of a storage zone.

The bytes live here; PostgreSQL holds the index. This module knows about keys and blobs and nothing about zones, paths, or who is allowed to read them.

ObjectNotFoundError

Bases: LookupError

No object at that key. Raised rather than returning None.

StoredObject dataclass

What the store knows about one blob.

ObjectStore

The S3 operations this service performs, and no others.

put_file async 

put_file(
    key: str, path: Path, content_type: str
) -> StoredObject

Store a local file under key and report what the store now holds.

One PUT, no multipart: the service caps objects well below the 5 GiB single-request limit. Objects larger than that need multipart, which is a different method, not a bigger version of this one.

download async 

download(key: str, path: Path) -> StoredObject

Stream the object to a local file — how the cache is filled.

copy async 

copy(
    source: str, key: str, content_type: str
) -> StoredObject

Copy one object to another key without the bytes leaving the store.

An S3 client updating metadata does it by copying an object onto itself, so this runs on an ordinary sync of an unchanged tree. Pulling the bytes here and pushing them back would turn that into two transfers of every file.

head async 

head(key: str) -> StoredObject

Metadata for one object, or :exc:ObjectNotFoundError.

delete async 

delete(key: str) -> None

Remove the object. Deleting what is not there is not an error.

list_prefix async 

list_prefix(
    prefix: str = "",
) -> AsyncIterator[StoredObject]

Every object under a prefix, a page at a time.

For reconciliation, which walks the whole store — so it pages rather than materialising a listing that can be arbitrarily large.

ping async 

ping() -> None

Raise unless the bucket is reachable. For the deep health check.

connect async 

connect(settings: Settings) -> AsyncIterator[ObjectStore]

Open the object store, with one client for the life of the caller.

A client per request would open a connection pool per request, which is how a service exhausts file descriptors under the load it was built for. Tests open it the same way production does, so they exercise the same retry and timeout budget.

storage.cache

abilian_cdn.storage.cache

The local disk cache.

Disposable by design: deleting the cache directory costs latency and nothing else. S3 holds the bytes, PostgreSQL holds the index, and this holds whatever has been asked for recently.

A pull zone has no S3 copy and no index row — what it mirrors belongs to somebody else. Its entries therefore carry a sidecar next to the body, saying what the origin said and until when we may repeat it. Storage zones need no sidecar: their metadata is in the database, which is the index of record.

Entry dataclass

One remembered answer: what it was, and until when it may be repeated.

A status other than 200 is a negative entry — the origin said the path is not there — and has no body on disk.

fresh 

fresh(now: float) -> bool

Whether this may still be served without asking the origin.

age 

age(now: float) -> int

Seconds since the origin gave us this, for X-Cache-Age.

SweepReport dataclass

What one sweep did, for the log line and for the tests.

Cache

Files on disk, addressed by (zone, object path).

locate 

locate(zone_id: UUID, path: str) -> Path

Where an object's bytes live, cached or not.

The name is a hash rather than the object path: paths run to 1024 bytes of arbitrary UTF-8, which is longer than filesystems accept and includes forms two different paths share on a case-insensitive one. A hash has none of those problems and needs no escaping rules.

lookup 

lookup(zone_id: UUID, path: str) -> Path | None

The cached file, or None when it is not there — a miss, not a fault.

store 

store(zone_id: UUID, path: str, source: Path) -> Path

Move an already-written file into the cache, atomically.

Atomic because a reader that finds a half-written file has no way to know it is half-written: the rename either publishes the whole file or nothing.

evict 

evict(zone_id: UUID, path: str) -> None

Drop one object, body and sidecar.

Both, always: a sidecar outliving its body claims a size and a type for bytes that are gone, and the next reader would believe it. Evicting what is not cached is not an error.

describe 

describe(zone_id: UUID, path: str) -> Entry | None

What the sidecar says about this path, if anything readable.

An unreadable sidecar answers None rather than raising: a cache is allowed to forget, and the caller's next move — ask the origin — is the right one either way.

remember 

remember(zone_id: UUID, entry: Entry) -> None

Write the sidecar, atomically, so no reader sees half of one.

paths_under 

paths_under(zone_id: UUID, prefix: str) -> list[str]

Every cached path of this zone that starts with prefix.

Read out of the sidecars, because the file names are hashes: nothing about a cache directory can be matched against a prefix without asking each entry what it is. Only a zone with sidecars — a pull zone — can answer; a storage zone's paths are in the database, which knows them whether they are cached or not.

entries 

entries(zone_id: UUID) -> Iterator[Entry]

Every readable sidecar of one zone.

purge 

purge(zone_id: UUID, paths: Sequence[str]) -> int

Drop these paths from the cache. Returns how many were named.

purge_zone 

purge_zone(zone_id: UUID) -> int

Drop everything this zone has cached.

By directory rather than entry by entry: a zone can hold more entries than we want to enumerate, and the whole point is that none of them survive.

scratch_file 

scratch_file(name: str) -> Path

A path to write to before publishing into the cache.

On the same filesystem as the cache, so the move that publishes it is a rename rather than a copy of the whole object.

sweep 

sweep(
    cache: Cache, high_water: int, low_water: int
) -> SweepReport

Evict least-recently-used files until the cache is under low_water.

Least-recently-read, by access time, which is the question that matters for a cache: the file written an hour ago and read every minute since is worth more than the one written a minute ago and never read.

Two marks, not one: evicting down to the ceiling on every write would mean a sweep per upload once the cache is full, each freeing one file.

Filesystems mounted relatime update atime lazily, so the ordering is approximate. An approximate LRU evicts a slightly wrong file; an exact one would need its own bookkeeping on the read path, which costs more than the mistake does.

storage.origin

abilian_cdn.storage.origin

Pull zones: mirroring an origin nobody here controls.

A pull zone owns no bytes. It answers from the local cache, and when the cache has nothing fresh it asks the origin and remembers the answer for as long as the origin's own headers allow — clamped to the window the zone configures, because an origin that says max-age=31536000 has not been told how long we are willing to be wrong.

Everything an origin sends that is not content is dropped on the way through. Its cookies in particular: a delivery host that ever sets one stops being cacheable by every proxy between here and the reader.

OriginUnreachableError

Bases: ClientException

The origin could not be asked, and nothing here could answer for it.

LoopDetectedError

Bases: ClientException

A request that has already been through this service.

Fetched dataclass

What the cache holds for a path now, and how it came to hold it.

body is None exactly when the entry is a remembered absence — the origin said the path is not there, and there are no bytes to point at.

InvalidOriginError

Bases: ValueError

An origin URL a zone may not have, with the rule it broke.

SingleFlight

One fetch at a time per path, in this process.

A cold popular object is asked for by a hundred clients at once, and without this the origin gets a hundred identical requests — the thundering herd a CDN exists to absorb. Another process may still duplicate the fetch; that is a factor of the worker count, not of the traffic.

only_one async 

only_one(key: tuple[UUID, str]) -> AsyncIterator[None]

Hold the lock for this key, counting who is waiting for it.

Origin

Fetches, caches, revalidates and — when it must — survives an origin.

fetch async 

fetch(zone: Zone, path: str) -> Fetched

What this zone should answer for path, fetching it if need be.

Checked twice against the cache: once before queueing behind whoever is already fetching this path, and once after — by which time they have usually filled it, which is the entire point of queueing.

validate_origin 

validate_origin(url: str) -> str

Return the origin a pull zone will mirror, or say why it cannot.

Checked once, here, because everything downstream builds a URL out of it by concatenation: an origin with a query string or a missing scheme would produce requests that fail in ways nobody would trace back to this field.

refuse_loops 

refuse_loops(via: str | None) -> None

Refuse a request that has already passed through this service.

A pull zone whose origin resolves back here — by mistake or by a misconfigured pair of zones — would otherwise fetch itself until something ran out of sockets.

storage.reconcile

abilian_cdn.storage.reconcile

The index and the object store, checked against each other.

An upload writes the bytes first and the index row second, so a crash in between leaves a blob nothing points at. A delete removes the bytes first for the same reason, so a crash there leaves a row pointing at nothing. Both are rare and neither is visible: the first quietly costs storage, the second answers 500 to the one person who asks for that file.

Both sides are walked in key order rather than read into memory. S3 lists lexicographically and Postgres can sort the same way, so one pass over two streams compares a zone of any size in constant memory.

Report dataclass

What one zone's two sides disagree about.

check async 

check(
    session: AsyncSession, store: ObjectStore, zone: Zone
) -> Report

Compare one zone's index against the bucket, changing nothing.

repair async 

repair(
    session: AsyncSession,
    store: ObjectStore,
    zone: Zone,
    report: Report,
) -> Report

Act on what check found: the bytes are the record, the index follows.

Orphaned blobs are deleted, because nothing can reach them. Dangling rows are deleted too — the bytes they describe are gone, and a row that only produces a 500 is worse than an honest 404. A size mismatch is corrected from the store rather than the other way round.

zones async 

zones(
    session: AsyncSession, slug: str | None = None
) -> list[Zone]

The zones to check: one by name, or every storage zone there is.