S3 API¶

Mounted at /_/s3, or at the root of CDN_S3_HOST when one is set. A bucket is a zone; a key is an object path. For the client side of this, see S3 clients.

Authentication¶

AWS Signature Version 4, verified against the request as it arrived — the path, the query and the signed headers are recomputed rather than trusted, so nothing downstream has to wonder whether what was signed is what turned up.

The access key id is a token's key id. The secret is derived from that id and CDN_SECRET_KEY with HMAC, so it is never stored: rotating the service key invalidates every S3 credential at once.

Presigned query authentication is not implemented. Nothing on the target list needs it to sync files, and this service has signed URLs of its own that are simpler for the case presigning would serve.

Operations¶

Everything else answers 501 NotImplemented.

Details that matter to a real client¶

A PUT carrying x-amz-copy-source is a copy, not an upload, and its body is empty on purpose. Treating it as an upload would store nothing over the destination — which is how a client rewriting a timestamp ends up deleting the file it was describing. rclone does this on every sync of a tree it has seen before.

aws-chunked bodies are decoded as they stream. A client that signs a streaming payload sends the length of the decoded content in x-amz-decoded-content-length and frames the body itself; the frames are unwrapped and the digest checked against what was signed.

x-amz-meta-* is stored and returned, bounded at S3's own 2 KB. rclone keeps each file's modification time there; without it, rclone decides on every sync that every file still needs its timestamp written, and issues one copy per file for ever. x-amz-metadata-directive decides whether a copy keeps the source's metadata or takes the request's.

Multipart upload is not implemented. A single request is bounded by CDN_MAX_UPLOAD_BYTES.

Errors¶

XML, with the code and status S3 pairs together — a client that gets HTML, or an empty body with a status, reports something that has nothing to do with what went wrong.

SlowDown is what botocore's retry logic already backs off from. QuotaExceeded is Ceph RGW's spelling: S3 proper has no notion of a quota, so there is no more canonical code to borrow.

The hostname, and why it exists¶

s3cmd folds an endpoint's path into the Host header it signs — host:cdn.example.com/_/s3/assets — so no signature it makes can match a service mounted under a path. Setting CDN_S3_HOST gives the S3 face a hostname whose root is the API; the bucket is then the first path segment and every client agrees about what was signed.

The rewrite happens outside the application, before routing, and the path the client signed travels with the request — checking a signature against the rewritten path would refuse every request. A signature made for /_/s3/{bucket} and sent to the S3 host is refused, so the rewrite cannot be used to pass one signature off as another.