Running it¶

One process serves everything. Two small workers keep the disk and the database from growing without bound.

Deployment¶

The service is one Hop3 app, described by hop3.toml: a web process, a PostgreSQL addon, a persistent cache directory, and the workers.

cdn migrate                    # before the new code serves traffic
cdn sweep  --interval 300      # evict from the cache past its high-water mark
cdn rollup --interval 3600     # fold statistics, prune the audit log

The web process must bind 0.0.0.0:$PORT. An ASGI app also needs an explicit start command: uvicorn abilian_cdn.app:serve --factory --host 0.0.0.0 --port $PORT. It names serve, which wraps the application in the host router that CDN_S3_HOST needs.

The proxy in front¶

Two things have to be set on whatever terminates HTTP. On Hop3 the app declares both itself in hop3.toml; anywhere else they belong to the operator's nginx.

• a body limit above CDN_MAX_UPLOAD_BYTES ([proxy] max-body-size, which renders to nginx's client_max_body_size). Below it the proxy refuses a large upload before this service sees the request, and answers in HTML naming neither the zone nor the object. Hop3's default is 50m, smaller than most things a CDN carries.
• request buffering off ([proxy] request-buffering = false, or proxy_request_buffering off), so a large upload streams through. Buffered, the proxy writes the whole body to its own disk before the app sees a byte, so a 2 GB upload needs 2 GB of proxy scratch space.

The first account¶

A fresh deployment has no organisation and nobody who can sign in. cdn provision makes both, and is idempotent, so every deploy can run it:

cdn provision --org acme --org-name "Acme Corp" --zone assets --email you@example.com

On Hop3 it is declared in hop3.toml. The platform generates the password, hands it to the command in the environment, and surfaces it to the operator:

[admin]
email = "operator"
login = "email"
create = "cdn provision --org acme --org-name 'Acme' --zone assets"
verify = "cdn provision --check --org acme --zone assets"

[admin.password]
generate = "password"
length = 24

verify runs after create and has to exit 0 before the account is treated as made. It exists because a bootstrap command can print an error and still exit 0. That has meant handing an operator the password to an account that was never created. cdn provision --check changes nothing, names what is missing, and fails when anything is.

The email, name and password come from HOP3_ADMIN_EMAIL, HOP3_ADMIN_USER and HOP3_ADMIN_PASSWORD when the flags are absent. When the platform supplies the password, provision does not print it: the platform already knows it, and the deploy log is the one place a generated credential should not be.

The cache volume¶

CDN_CACHE_DIR holds the local copies and their metadata. It is persistent so a restart does not re-fetch everything, and disposable: losing it costs latency, never data.

The sweeper evicts least-recently-read files once the directory passes CDN_CACHE_HIGH_WATER and stops at CDN_CACHE_LOW_WATER. Two marks, so a full cache does not evict on every single write.

Exclude it from backups.

What to back up¶

PostgreSQL holds the index, the accounts, the tokens, the statistics and the audit log. S3 holds the bytes of every storage zone. Back up the first; the second is already durable.

You also need CDN_SECRET_KEY. Losing it signs everyone out and invalidates every derived S3 credential; the objects are unaffected.

Checking the two halves agree¶

An upload writes the bytes before the index row, and a delete removes them after it, so a crash in between can leave a blob nothing points at or a row pointing at nothing. Neither is visible in normal use: the first costs storage nobody is using, the second answers 500 to the one person who asks for that file.

cdn reconcile                  # report
cdn reconcile assets --repair  # and fix

It walks the index and the bucket as two sorted streams, so a zone of any size is compared in constant memory. It reports by default and exits non-zero when something disagrees, which makes it a reasonable nightly cron; --repair deletes orphaned blobs, drops dangling rows and corrects sizes from the store.

Health checks¶

The shallow check queries the database on every call, because a check that only proved the process was running would report a service answering every request with a 500 as healthy. The deep one is not for the load balancer: taking the service out of rotation because a third party is slow turns their bad minute into your outage, while cached objects would have kept serving.

A local object store¶

For development, any S3 works. The test suite and the examples here use Garage:

scripts/garage-dev.sh start

MinIO, Scaleway, OVH and AWS all work the same way. The service holds no assumption beyond S3 itself.