keystone.dunn.dev

architecture

How keystone works

keystone speaks the protocol of stock tangd and serves many named key sets from one Cloudflare Worker. Deleting a set's secret revokes it. keyset mints, checks, deploys and revokes the sets.

Overview

clevis in the host's initramfs talks to keystone through Cloudflare's edge, where a WAF rate limit can sit ahead of the Worker to slow a flood that would otherwise spend the Free plan's daily request quota. A key set is one signing key and one exchange key, held in a Worker secret named TANG_KEY_<NAME>. keystone answers two requests per set: adv returns its signed public keys, and rec runs the key recovery a booting host needs.

CLOUDFLARE ACCOUNT HTTPS reads DISK HOST initramfs clevis · LUKS EDGE Cloudflare WAF rate limit WORKER keystone adv · rec WORKER SECRETS key sets TANG_KEY_<NAME> CLOUDFLARE ACCOUNT HTTPS reads DISK HOST initramfs clevis · LUKS EDGE Cloudflare WAF rate limit WORKER keystone adv · rec WORKER SECRETS key sets TANG_KEY_<NAME>

Binding

clevis checks the advertisement's signature against the pinned thp, the thumbprint of the set's signing key, so a forged advertisement fails the bind.

It picks a random c and computes K = c·S from S, the exchange public key; K wraps a new slot passphrase.

The header keeps C = c·G (G is the curve's base point), the kid (the exchange key's thumbprint) and the advertisement, and c and K are discarded.

CLEVIS LUKS BIND adv/<thp> signed adv WORKER keystone serves adv only VERIFY signature against pinned thp DERIVE K = c·S fresh c, C = c·G LUKS HEADER public only C · kid · adv S: exchange public key c and K discarded + slot passphrasewrapped under K CLEVIS LUKS BIND adv/<thp> signed adv WORKER keystone adv: exchange key S VERIFY signature against pinned thp DERIVE K = c·S C = c·G kept; c, K dropped LUKS HEADER public only C, kid, adv, passphrase under K

Unlock

At boot clevis reads C from the header, picks a fresh random e and sends X = C + e·G to rec.

The Worker returns Y = s·X, where s is the exchange private key.

With S from the advertisement stored in the header, the host computes K = Y − e·S = s·C, which equals the bind's c·S because both are s·c·G, and K unwraps the slot passphrase. The Worker sees only the blinded X, never C or K.

POST /<name>/rec/<kid> X Y DISK HOST initramfs clevis · LUKS WORKER keystone exchange key s X = C + e·G Y = s·X K = Y − e·S LUKS slot opens C from the LUKS header e fresh at this boot s: exchange private key = s·C, the unwrapping key S from the stored adv K unwraps the slot passphrase POST /<name>/rec/<kid> Y DISK HOST X = C + e·G C stored, e fresh WORKER Y = s·X s: exchange private key DISK HOST K = Y − e·S = s·C, unwraps the passphrase

On the wire

The advertisement, X, Y, and the set name, kid and client address cross the network; the Worker logs one line per successful rec with name, address, ASN and country. C, e, K and the passphrase never leave the host. Since e is fresh each boot, X differs every time. A recording gives up neither C nor K, even to someone holding s: e hides C, and K needs s together with C from the disk's own header.

CROSSES THE WIRE NEVER LEAVES THE HOST signed adv public signing and exchange keys X = C + e·G blinded afresh at every boot Y = s·X s stays in the Worker name · kid · address log: name, address, ASN, country K = s·C unwraps the slot passphrase C the stored point, unblinded e this boot's ephemeral key slot passphrase unwrapped by K on the host CROSSES THE WIRE NEVER LEAVES THE HOST signed adv public signing and exchange keys X = C + e·G blinded afresh at every boot Y = s·X s stays in the Worker name · kid · address log: name, address, ASN, country K = s·C unwraps the slot passphrase C the stored point, unblinded e this boot's ephemeral key slot passphrase unwrapped by K on the host

A key set

The set server-1 lives in the Worker secret TANG_KEY_SERVER_1: the name upper-cased, with - turned into _. Its routes are /server-1/adv, /server-1/adv/<thp>, which answers only for the matching signing thumbprint, and POST /server-1/rec/<kid>. The signing key (ES512 on P-521, ES256 on P-256) signs the advertisement; the exchange key, labelled ECMR for the McCallum-Relyea exchange above, answers rec. Sets share no keys, so revoking one leaves the others serving.

ROUTES TANG_KEY_SERVER_1 GET /server-1/adv JWS over both public keys GET /server-1/adv/<thp> thp: signing key thumbprint POST /server-1/rec/<kid> kid: exchange key thumbprint SIGNING KEY ES512 · P-521 signs the adv EXCHANGE KEY ECMR · P-521 in the adv · answers rec server-1 → TANG_KEY_SERVER_1 one curve: P-521 or P-256 ROUTES TANG_KEY_SERVER_1 server-1 → TANG_KEY_SERVER_1 GET /server-1/adv JWS over both public keys GET /server-1/adv/<thp> thp: signing key thumbprint POST /server-1/rec/<kid> kid: exchange key thumbprint SIGNING KEY ES512 · P-521 signs the adv EXCHANGE KEY ECMR · P-521 in the adv · answers rec one curve: P-521 or P-256

Lifecycle

keyset new mints a set for the offline key store, deploy --new makes it live, and revoke deletes its secret, so its adv answers 404 and its disks stop unlocking.

The store then keeps null under that name, a tombstone, and every later deploy checks it still answers 404.

keyset refuses new keys under a live name, which would strand every disk bound to it, so rotation uses a new name.

deploy --new revoke null entry MINTED server-1 keyset new LIVE server-1 adv 200 REVOKED no secret adv 404 TOMBSTONE null deploy input new keys, same name REFUSE · rotate to server-2 private keys in your key store its disks stop unlocking each deploy proves 404 deploy --new revoke null entry MINTED server-1 keyset new · key store LIVE server-1 adv 200 REVOKED no secret adv 404 · disks stop unlocking TOMBSTONE null each deploy proves 404 new keys, same name REFUSE · rotate to server-2

Rotation

The disk's LUKS header holds a passphrase slot and one clevis slot per bound set.

Rotation binds server-2 in a second slot, reboots, then unbinds and revokes server-1. Rebooting before the unbind proves the new slot while the old one still opens the disk.

A copy taken before the unbind still opens with server-1's keys, so the store swaps them for the tombstone.

BOUND server-1 BIND server-2 REBOOT unlocks UNBIND server-1 REVOKE server-1 passphrase LUKS slot server-1 clevis slot server-2 clevis slot set server-1 at keystone 404 a copy taken before the unbind opens with server-1's keys SLOT server-1 SLOT server-2 SET server-1 bound server-1 bind server-2 reboot unlocks unbind server-1 revoke server-1 404 the passphrase slot stays throughout a copy taken before the unbind opens with server-1's keys

Rollback

Each Worker version carries the secrets it was deployed with. After the revoke in version 3, rolling back to version 2 serves server-1 again, though the Worker's secret list no longer shows it. keyset deploy flags that through the tombstone as REVOKED-LIVE, and --apply builds version 4 from the current secrets, which drops it. So never roll back across a revoke.

DEPLOY version 1 DEPLOY --NEW version 2 REVOKE version 3 ROLLBACK to version 2 DEPLOY --APPLY version 4 server-1 served 404 404 server-1 in secret list server-2 served server-2 in secret list served, not listed: keyset deploy reports REVOKED-LIVE server-1 SERVED LISTED server-2 SERVED LISTED deploy version 1 deploy --new version 2 revoke version 3 rollback to version 2 deploy --apply version 4 404 404 server-1 served, not listed: keyset deploy reports REVOKED-LIVE --apply redeploys from current secrets

keyset deploy

keyset deploy reads the key store on stdin as one JSON object, a set or null per name, and gives each entry a verdict against the live Worker.

Any REFUSE stops the run with exit 1; without --apply it ends at the plan.

With --apply, wrangler uploads only the --new sets, then keyset checks each set's signed adv and a real rec, and each tombstone for a 404. The commands are in Install.

STDIN key sets set | null PLAN each set vs live Worker GATE no REFUSE and --apply WRANGLER deploy --new secrets CHECK every set adv · rec VERDICTS ok live and unchanged new absent, named by --new revoked null, answers 404 REVOKED-LIVE null, yet served drift live, not in stdin REFUSE --url ≠ secret list, … STOP any REFUSE exit 1 no --apply exit 0, plan CHECKED adv signed by the set's key keys exactly the set's rec a real exchange answers null answers 404 STDIN key sets set | null PLAN each set vs live Worker GATE no REFUSE and --apply WRANGLER deploy --new secrets CHECK every set adv · rec STOP any REFUSE exit 1 no --apply exit 0, plan VERDICTS ok live and unchanged new absent, named by --new revoked null, answers 404 REVOKED-LIVE null, yet served drift live, not in stdin REFUSE --url ≠ secret list, … CHECKED adv signed by the set's key keys exactly the set's rec a real exchange answers null answers 404

Curves

Each bar is the measured CPU time for one rec, against the Free plan's 10 ms of CPU per request. P-521, the default and the curve tangd-keygen uses, takes 7 ms at the median and up to 12 ms cold, so it needs Workers Paid. P-256 stays at 6 ms or under and fits the Free plan; keyset new --curve P-256 mints one.

median up to, cold P-521 default · ES512 P-521 median: 7 ms P-521 cold: up to 12 ms 7 ms median 12 ms cold P-256 ES256 P-256 cold: up to 6 ms 6 ms cold Free plan: 10 ms 0 2 4 6 8 10 12 14 ms median up to, cold P-521 default · ES512 P-521 median: 7 ms P-521 cold: up to 12 ms 7 ms median 12 ms P-256 ES256 P-256 cold: up to 6 ms 6 ms Free plan: 10 ms 0 2 4 6 8 10 12 14 ms

Trust

Cloudflare, and whoever controls the account, can read the private keys, as can any user or token allowed to deploy a new version, since that code reads the secrets. The API lists secret names only, and disk hosts hold only public keys. To split trust, bind with clevis sss at a threshold of two across keystone and a second Tang server, so neither alone unlocks the disk.

CAN READ CANNOT READ reads write-only adv only CLOUDFLAREaccountwhoever controls DEPLOY RIGHTSnew versionany user or token WORKER VERSIONSevery versionown snapshot WORKER SECRETS key sets TANG_KEY_<NAME> CLOUDFLARE API secret list names only DISK HOSTS LUKS header public keys only CAN READ CANNOT READ reads adv only write-only CLOUDFLAREaccountwhoever controls DEPLOY RIGHTSnew versionany user or token WORKER VERSIONSevery versionown snapshot WORKER SECRETS key sets TANG_KEY_<NAME> DISK HOSTS LUKS header public keys only CLOUDFLARE API secret list names only

Alerts

When alerts are set up, each successful rec asks whether its source, a /24 or /48 network plus ASN and country, is new for the set. The first new source in an hour sends one alert; up to five more wait in KV for the hourly digest, and the host never waits on the mail. The kid is public, so an alert shows a request was made; it does not prove an unlock. The Workers log stays the full record.

after first this hour meanwhile send fails answer seen WORKER rec succeeds logs the source NEW SOURCE? /24 or /48 + ASN, country SEND_EMAIL one alert per set per hour DISK HOST unlocks never waits SEEN BEFORE no mail nothing held KV held up to 5 an hour HOURLY CRON one digest send_email answer after seen first meanwhile fails WORKER rec succeeds logs the source DISK HOST unlocks never waits NEW SOURCE? /24 or /48 + ASN, country SEEN BEFORE no mail nothing held SEND_EMAIL one alert per set per hour KV held up to 5 an hour HOURLY CRON one digest send_email

Failure

If keystone is unreachable or the day's request quota is spent, clevis gets no key and LUKS asks for the passphrase, so losing keystone costs a typed passphrase per disk; the data is untouched as long as each disk keeps its passphrase slot. If a disk is stolen, revoking its set makes rec answer 404, and the disk stays locked from then on.

KEYSTONE UNREACHABLE OR QUOTA SPENT DISK STOLEN, SET REVOKED rec DISK HOSTbootsclevis tangKEYSTONEunreachableor quota spentCLEVISfailsno key for LUKSLUKSpassphrasedata intact error rec STOLEN DISKbootsclevis tangKEYSTONE404set revokedCLEVISfailsno key for LUKSLUKSstays lockedfrom then on KEYSTONE UNREACHABLE OR QUOTA SPENT DISK STOLEN, SET REVOKED rec DISK HOSTbootsclevis tang KEYSTONEunreachableor quota spent CLEVISfailsno key for LUKS LUKSpassphrasedata intact error rec STOLEN DISKbootsclevis tang KEYSTONE404set revoked CLEVISfailsno key for LUKS LUKSstays lockedfrom then on