The Runtime Engine

Everything so far has been the contract. This chapter is the machine: how runtime.Runtime takes a pile of controllers and a state broker and turns resource changes into precisely-targeted wakeups — efficiently, safely, and with each controller isolated behind its own guarded adapter. You don't need this to use COSI, but you need it to trust and debug it.

#Construction & registration

runtime.NewRuntime(state, logger, opts...) builds the engine. Registration wraps each controller in an adapter — rruntime.Adapter for classic controllers, qruntime.Adapter for queue ones — and records its inputs/outputs in a dependency.Database. Registration also fails fast on conflicts: two controllers can't both claim the same exclusive output.

type Engine interface {
    RegisterController(ctrl Controller) error    // → wraps in rruntime.Adapter
    RegisterQController(ctrl QController) error   // → wraps in qruntime.Adapter
    Run(ctx context.Context) error               // start watches + all adapters
}

You can register controllers before or after Run — if the engine is already running, a freshly-registered adapter is started immediately in the same errgroup. Run itself: sets up one WatchKindAggregated per distinct (namespace, type) any controller watches, spins up the event-routing goroutines, and launches every adapter.

#The event pipeline

This is the engine's beating heart. A single firehose of state changes is reduced, de-duplicated, and routed only to the controllers that declared an interest. The trick is the two-stage goroutine pipeline that decouples "consume changes fast" from "deliver to controllers."

Two ideas make this cheap:

• Reduced events. The pipeline doesn't carry full resources between stages — it carries reduced.Metadata (namespace, type, id, phase, finalizers-empty, labels). That's all the routing layer needs, and it's tiny.
• Coalescing. Stage 1 collapses many changes to the same key into one entry. If a resource is written 100 times while a controller is busy, the controller sees one wakeup, then reconciles against the final state — never a backlog of 100.

#The dependency database

dependency.Database is the index that answers stage 2's question: "resource X just changed — who cares?" It maintains reverse lookups from (namespace, type) and (namespace, type, id) to controller names, plus the exclusive/shared output maps used to reject conflicts at registration.

exclusiveOutputs map[resource.Type]string         // type → the one owner (conflict check)
sharedOutputs    map[resource.Type][]string        // type → many owners
inputLookup      map[nsType][]string               // (ns,type)    → watchers
inputLookupID    map[nsTypeID][]string             // (ns,type,id) → watchers (specific)

// stage 2 asks:
func (db *Database) GetDependentControllers(in Input) []string

The whole graph is exportable as a DependencyGraph of typed edges (EdgeInputStrong, EdgeOutputExclusive, …). That's not just for debugging — it's the literal documentation of your system, renderable as a diagram.

#Adapters: rruntime & qruntime

An adapter is the bridge between the engine and one controller. It implements the Runtime/QRuntime facade the controller sees, owns the controller's goroutine(s), and translates WatchTrigger calls from the pipeline into the controller's native wakeup mechanism.

#The read-through cache

Reads are far more common than writes, and re-reading the state on every reconcile is wasteful. Opt a resource type into the cache (options.WithCachedResource) and the engine maintains an in-memory, always-current copy fed by the same watch stream.

• The cache bootstraps from the watch: Get/List on a cached type block until the initial contents have loaded (the Bootstrapped marker), then serve instantly from memory.
• Each handler keeps resources sorted by ID for binary-search Get and cheap filtered List; every read returns a deep copy so controllers can't corrupt the shared cache.
• When you genuinely need to bypass it (read-after-write within a reconcile), the UncachedReader — GetUncached / ListUncached — goes straight to the backing state.

#The state adapter — guardrails

The Runtime a controller receives isn't the real state. It's a controllerstate.StateAdapter that enforces every rule we've discussed, on every call:

This is why a COSI controller is hard to misuse: the declarations you wrote in Inputs()/Outputs() aren't documentation the engine hopes you honor — they're the capability boundary it actively enforces at runtime.

#Restart, backoff & metrics

Controllers are expected to fail — a transient state conflict, an unreachable external system, a panic. The engine treats every controller as restartable:

• Panics are caught per run with a stack trace; the controller restarts rather than crashing the process.
• Exponential backoff governs restarts (classic) and per-item requeues (queue), with no max elapsed time — it keeps trying forever, backing off. A clean reconcile calls ResetRestartBackoff.
• Metrics are exported via expvar (enable with options.WithMetrics(true)): per-controller crashes, wakeups, reads, writes; per-QController queue length, requeues, skips, processed, and busy-seconds; and cached-resource counts. The example binary serves them at /debug/vars.

rt, _ := runtime.NewRuntime(st, logger,
    options.WithMetrics(true),                  // expvar counters
    options.WithCachedResource("vms", vm.VMType), // read-through cache for VMs
    options.WithChangeRateLimit(rate.Limit(200), 50), // cap mutations
)
// metrics visible at http://host/debug/vars  →  cosi_controller_wakeups, …