Governing Intelligent Systems

Every organisation that runs automation is accountable for it: to its board and its customers, and in regulated work to its regulators. Boards ask what AI their organisation runs. The organisation usually finds out from a staff survey. Staff in finance, compliance and elsewhere adopt tools on their own initiative. IT's approved software list records only some of those tools.

In financial services ASIC and APRA ask which systems make decisions that affect clients, what oversight exists and how an organisation would know if something went wrong.

In New South Wales, an expert witness may use generative AI to produce the content of a report only with the leave of the court. A report prepared without leave must state that the expert did not use generative AI. Where the court grants leave, the expert must identify which part was generated, by which program and version, and annex the prompts. An organisation can meet that requirement only if its records show which parts of a report a model generated and which the expert wrote.

Environmental and emissions reporting regimes require the figures in a filing to be traceable to their source.

Surveys

The usual response is a staff survey, run quarterly or more often. It takes staff time and misses much of actual use. By the time the survey is complete, its results are out of date.

The alternative is a register to which every tool reports automatically. It needs every tool to report in a common format through a common hierarchy. Tools report in incompatible formats. Most report little about what they know or may do.

Four surfaces

Governing an automated system means accounting for four governance surfaces. A governance surface is an aspect of a system that an organisation must control. The four apply to any automated system, including one that uses AI.

An agent is a program that pursues a goal: it holds state, decides what happens next and acts through tools someone chose. A model is a function the agent calls. Calling a model is an invocation. Adopting the model's output, or letting the output decide what happens next, is an inference. Each inference introduces uncertainty into a process. All four surfaces concern the agent. An organisation that treats the model as the agent ends up governing a vendor's model, which the organisation has no means to inspect or version.

Structural governance covers what is running, where and in what way: every component, its version, where it runs, how it is configured, what it connects to and what it has been told to do. A staff survey reconstructs what is running from what staff remember.

Authoritative governance covers who defines, builds, operates and uses the system, how authority is delegated and who is responsible for it. Owners, governance bodies and operators hold authority and decide what the system may do. They delegate it to workflows, workloads and users, who act within that mandate. The same people decide which data and knowledge each part of the system may use.

Epistemic governance covers what the system knows: the data and context an inference draws on, what the inference can retrieve and what sensitive information the system holds. Some organisations give automated systems documents nobody has reviewed. When the output is poor, they conclude that AI is unreliable. The unreviewed inputs are a likely cause.

Executory governance covers what the system can do: the actions it can take, their consequences, the confidence it needs before it acts and where a person reviews its work. This surface determines whether a system informs a decision or takes it, and regulators ask about that distinction.

Boundaries

The four surfaces also apply wherever data, instructions or obligations pass between the system and anything outside it. Legacy data brought into a knowledge store, third-party data feeding an inference, a frontier model called at runtime, a broker connection carrying instructions, an investor interaction creating an obligation and a filing reaching a regulator are all such crossings. At each crossing the four questions apply: what crosses and in what form, who approved it, what knowledge enters and with what provenance, and what consequences follow.

In a first automation project, the organisation's existing data sits outside governance until an ingestion process loads it. The data arrives raw and formatted for people to read. Governing that ingestion process under the same four surfaces is the first step.

Declaration

An engineer declares the whole system as a composition before it runs: every component, how the components connect and what each requires of the others. daedal compiles the composition and resolves every part of it before daedal provisions any component. aidion runs the application daedal compiles from the composition. An engineer changes the system by changing its composition.

Structural. The composition lists every component, so it serves as the register a staff survey tries to reconstruct. daedal issues a certificate for each composition: a digest of its normal form, a digest of its resolved form and a content digest for every input. daedal verify recomputes the composition's digests, compares them with the certificate and reports each input that has changed.

Authoritative. Each request to aidion carries a capability token stating what the token's holder may do. A holder can add restrictions to a token. If anyone removes a restriction, the token's signature fails verification and aidion rejects the token. The composition declares each step's grants: the programs, secrets, paths and executor the operator allows the step. Under the container executor, a step reaches only the programs, secrets and paths the operator granted it. daedal logs every run.

Epistemic. daedal gives each step only the secrets the operator granted it. The composition declares what each component consumes: its configuration, the data other components supply to it and the services it uses. We help each team agree with the organisation's experts which documents an inference may draw on and who reviews those documents.

Executory. The composition declares every step and workflow. When a failure interrupts a workflow, aidion resumes the workflow from its last completed step. aidion records each consequential action in an audit chain for each organisation. A separate key server computes each link of the chain with a key only the key server holds. The key server verifies every link, and an altered action fails verification at the next link. Each application's composition declares which model outputs each step uses unchanged. A person reviews any output below the client's confidence threshold.

Demands

Boards, customers, auditors and regulators make four demands of an automated system. The composition, the compiled graph and the audit chain meet them, and the organisation can show all three at any time.

Recovery. An operator recovers the system by having daedal apply an earlier composition to a clean environment. That apply rebuilds the system exactly as it was declared.

Visibility. daedal's compiled graph shows every component, every connection and the order in which the components run, before anything runs.

Accountability. The capability tokens and the audit chain show who authorised each consequential action and who took it.

Growth. An engineer adds a component to the composition, and daedal compiles the addition under the same checks as every other component.

Regimes

Financial regulators, professional bodies, courts and emissions regimes each ask the same four questions: what is running, who is responsible for it, what it knows and what it can do. Organisations whose systems are declared in structure meet each of these regimes from the same composition and the records it makes possible. Operative provides the engines and engineering support. An engineering team, in an agency or in-house, builds the system.