aidion: the operation engine

Richard Heycock

aidion keeps a running application true to its package. It installs only artefacts that match the digests their manifest declares. It carries authority with every request as a capability token that any holder can narrow. It records each consequential action in an audit chain whose links a separate key server computes, and it resumes interrupted work from the last completed step.

1 What aidion is

A tenant is one organisation served from a shared installation. A package is a versioned archive of artefacts, with a manifest listing the services it provides and the packages it depends on. A binding records the package version bound to a logical service name.

aidion consists of these parts:

It relies on four external systems: Nomad places and runs workloads, Restate executes workflows durably, Postgres holds state and an S3-compatible store holds artefacts.

2 Packages

Upload

A package arrives as an archive holding a manifest and its artefacts. aidion decodes the manifest and checks each artefact's SHA-256 digest and byte size against it. aidion validates a job template in the manifest with Nomad's parser. It stores artefacts under their content address, sha256:<digest>, and the archive under the package's identifier. A package is unique by name, version and tenant.

Dependencies

The manifest pins each dependency by name, version and digest, resolved within the uploader's tenant. aidion records a dependency that has yet to arrive as waiting, and links it when it arrives. aidion computes the dependency closure of a package and reports what is missing from it.

Install

Install checks the dependency closure first. A package with an incomplete closure installs only when the caller's token permits an incomplete install. aidion records such an install as incomplete. aidion then renders a job for every service in the package before submitting any. A package starts only when every one of its services renders. Each job fetches its artefact from aidion by digest, and Nomad verifies the download against that digest.

Retirement

aidion retires a package only once the bindings that refer to it are deleted.

3 Services and bindings

A binding records a logical service name, the version and artefact digest bound to it, the endpoint the service listens on and the binding's tier: running, stopped or archived. When aidion sets a binding, aidion finds the binding's package by artefact digest within the caller's tenant. aidion deletes a binding only once it is archived.

aidion applies a tier change to the scheduler first, and records the new tier once the scheduler accepts it. running scales the service's job up, and resubmits the package's jobs if the job is absent. stopped scales the job to zero. archived stops it.

The sync agent

The sync agent follows the scheduler's allocation event stream. When an allocation starts running, the agent writes the allocation's address to the binding as the endpoint and registers the deployment with Restate. When an allocation completes, fails or is lost, the agent clears the endpoint. The agent stores its position in the event stream. On restart it reconciles every current allocation, then resumes the stream from the stored position. In production the sync agent refuses an endpoint at a private address.

4 Monitoring

aidion keeps the current state of every bound service, its allocations and the nodes they run on. It builds that state from the scheduler's allocation, deployment, job, evaluation and node events. From these it derives one status per service, by fixed precedence: failing, pending, degraded, unknown or healthy. aidion records each scheduling failure with the scheduler's reason.

A health agent polls each running service's health endpoint every ten seconds and records the result as passing, warning or critical.

5 Durable workflows

Workflows run on Restate, which journals each step. A workflow interrupted by a failure resumes from its last completed step.

The identity layer is a Restate virtual object for each logical name. When aidion sets a binding, it records the binding's version and artefact digest there. The invocation workflow resolves a logical name through the identity layer and invokes the service at the version it resolved. Restate journals the resolution, so a replayed invocation uses the same version even if the binding changes in the meantime.

A schedule invokes a service once after a delay, or repeatedly on a cron expression. Each wait is a durable timer, so it survives a restart.

6 Authority

Sign-in

A password is used only to sign in. The API returns a macaroon in exchange. The macaroon is the only credential the API issues.

Macaroons

A macaroon is a bearer token carrying caveats, each a condition on what its holder may do. aidion mints each token with four caveats: the subject, the tenant, the actions permitted and an expiry, set by default to eight hours after minting. aidion derives the permitted actions from the user's role (admin, operator or viewer) as capabilities for each domain.

The token's signature is an HMAC-SHA256 chain. The first link is an HMAC under the tenant's root key. Each caveat adds a link. Adding a caveat needs only the token, so any holder can narrow it. Removing a caveat breaks the chain. Verification recomputes the chain, compares signatures in constant time and accepts only caveats of the types it recognises.

Enforcement

Every API route except sign-in, sign-out and health requires a macaroon. aidion checks the macaroon's signature when the request arrives. An authoriser then checks the macaroon's caveats against the resource action the request performs, and refuses an expired token. aidion's workflow services verify the token on each request they receive. aidion's background agents act under a service token limited to the registry, monitoring and reading packages, minted again before it expires.

7 Keys

The key server holds each tenant's keys and performs cryptographic operations with them for its callers. Key material stays inside it. Each key belongs to a tenant and a purpose: HMAC-SHA256, Ed25519 signing or X25519 key agreement.

Envelope encryption

The key server derives a key-encryption key from a site master key with HKDF-SHA256 each time it needs one. Each tenant has a data key of its own, encrypted under the key-encryption key. Each of the tenant's keys is encrypted under the tenant's data key. Every layer uses AES-256-GCM with the tenant, purpose and version bound into the authenticated data, so a ciphertext decrypts only for the tenant and purpose it was made for. The key server holds decrypted keys only in memory. The database stores keys only in encrypted form.

Lifecycle

Each tenant and purpose has a versioned chain of keys, with one version active. Rotation creates a new active version. A rotated key keeps verifying and stops signing, so records made under it remain checkable. Revocation withdraws a key from every operation.

Callers

In production the key server identifies each caller by its mutual-TLS client certificate. An explicit list states which caller may perform which operation for which purpose. Only administrators can revoke a key.

8 Audit

The chain

Each tenant's audit events form a hash chain. An event records the actor, the resource, the event type, its context and the state before and after. Its hash is an HMAC-SHA256 over a canonical encoding of the event, its sequence number and the previous event's hash. The first event chains from a genesis value for the tenant. The key server computes each HMAC and is the only holder of the chaining key. The audit service assigns sequence numbers under a lock for each tenant, which gives the chain a single order.

Verification

The key server verifies each link in constant time, so it detects a changed or missing event at the first link after it. At intervals the audit service gathers a period's event hashes into a Merkle tree. The key server signs the tree's root with the tenant's Ed25519 key, and the audit service keeps the root in write-once storage, so a third party can check the trail with the tenant's public key alone.

The store

The audit service stores events in Postgres, keyed by tenant and sequence number. The database grants the application role insert and read rights only, and revokes update and delete from every role. The audit service recognises a repeated event by its identifier and leaves the stored row as it is.

Delivery

The audit client writes each event first to a durable outbox on the machine that produced it, under a key that prevents a retry from recording it twice. A forwarder delivers it through Restate and retries with backoff. The forwarder sets aside an event whose delivery fails, for an operator to requeue. When the outbox is near its limit it accepts only events marked critical.

What aidion records

Sign-ins, package uploads, installs and retirements, binding changes, tier transitions, invocations, schedules and artefact fetches.

9 Interfaces

HTTP API

JSON over HTTP, authorised by macaroon.

A separate node listener serves artefacts by digest to the scheduler's clients. A metrics listener serves Prometheus metrics.

Command line

The aidion command covers every API route. It also builds and inspects package archives offline.

10 Deployment

aidion needs Postgres, Nomad, Restate, an S3-compatible store, the key server and the audit service. In production aidion refuses to start in any of these cases: Restate, Nomad or the artefact store is reached over plain HTTP or at a private address; its token-signing secret is unset. A production build that allows private endpoints fails to compile. Services authenticate to one another with mutual TLS, using short-lived certificates issued by an internal certificate authority.

11 Libraries

restate_ex is an SDK for writing Restate services in Elixir. It supports services, virtual objects and workflows, with journalled side effects, durable timers, state, awakeables and workflow promises. It speaks Restate's service protocol versions 6 and 7, in request-response or bidirectional streaming mode. It verifies Restate's signed requests and includes a client for Restate's admin API.

restate_client_ex invokes Restate services through the ingress: calls, one-way sends with an optional delay, workflow submission and attachment, and awakeable completion. It retries requests that carry an idempotency key.

nomad_ex is a client for the Nomad HTTP API. It covers jobs, allocations, nodes, deployments, evaluations and ACL tokens, over TLS, with telemetry on every call.

macaroons implements the capability tokens: minting, attenuation, verification and extraction of capabilities. Its typed caveats cover tenant, subject, resource, domain, action, expiry, path, signer and artefact digest. It includes an authoriser for Ash resources and a Phoenix plug.

audit consists of three applications: a core library holding the event contracts and canonical encoding, a client holding the outbox and forwarder and a server holding the chain and the store.