Using It
Now the practical question you actually came for: where does COSI earn its keep in your systems? This chapter gives a decision framework, a one-page mapping from domain concepts to COSI primitives, and two greenfield blueprints — a VM control plane and a workflow/orchestration engine — plus the integration patterns for dropping it into an existing service. The next chapter dissects Talos as the canonical real-world application of these patterns.
#Is COSI the right tool?
COSI shines for exactly one shape of problem: declarative reconciliation — "record the desired state, then continuously drive reality toward it." Score your use case:
✓ Reach for COSI when…
You have desired vs. actual state to reconcile · long-lived objects with lifecycles (create → update → teardown) · derived/dependent objects (a VM needs a network, a disk, an IP) · ordered cleanup requirements · the need to react to changes, not just serve requests · a wish to scale from one process to many later.
✗ Skip COSI when…
It's a stateless request/response API with no background convergence · a one-shot batch job · purely a CRUD database with no reactive logic · you need multi-row transactions or complex relational queries · ultra-low-latency single-digit-µs paths (the reconcile loop adds scheduling latency by design).
The litmus test: if you find yourself writing a polling loop that reads some records, computes what should exist, and makes it so — that loop is a COSI controller waiting to be born.
#The domain → COSI mapping
Almost every COSI design is the same translation exercise. Keep this table next to you:
| Your domain has… | Model it as… | Because |
|---|---|---|
| A managed thing (VM, job, device, tenant) | a resource type with a typed spec | it has identity + desired state |
| "What the user/API asked for" | a spec resource (e.g. VMConfig) | desired state, written by the edge |
| "What's actually true right now" | a status resource (e.g. VMStatus) | observed state, written by a controller |
| A rule that derives B from A | a transform controller A → B | 1:1 reconciliation with cleanup |
| "For each X, ensure side-effect" | a QController over X | per-item, concurrent, retrying |
| "X must outlive Y's cleanup" | a strong input + finalizer | ordered teardown |
| "All things on host H" | a label + WithLabelQuery | queryable grouping, no joins |
| A grouping/partition | a namespace | isolation + independent stores |
| The external API / CLI / UI | the gRPC State service | typed CRUD + Watch for free |
Resist putting observed state into the spec resource the user writes. Keep desired (spec, user-owned) and observed (status, controller-owned) as separate resources. It keeps ownership clean (no two writers), makes the reconcile direction obvious, and lets the UI show "requested 4 CPUs / running with 4 CPUs" by reading both.
#Blueprint: a VM control plane greenfield
VM lifecycle management is COSI's home turf — it's essentially what Talos does for machines. Here's how you'd model it if starting fresh on COSI. Model the platform as a small graph of resources and controllers:
VMConfig fans out to networking, placement, and the hypervisor-driving VMController, which reports back through VMStatus.The resources
- VMConfig — desired VM (CPUs, memory, image, power state). Written by the API/UI. Namespace
vms. - VMStatus — observed VM (actual power state, IP, last error). Written only by
VMController. - NetworkConfig — derived network plumbing for a VM. Owned by
NetworkController. - Host — a hypervisor node with capacity, carrying labels like
host=node-7.
The controllers
- PlacementController (QController over
VMConfig) — readsHostcapacity, assigns a host by stamping ahost=label on aVMPlacementoutput. - NetworkController (qtransform
VMConfig→NetworkConfig) — provisions a bridge/IP; uses an input finalizer so a VM can't be destroyed until its network is torn down (the cooperative-deletion pattern from chapter 01). - VMController (QController over
VMConfig) — the one that touches the hypervisor. Reconcile = "make libvirt match the spec," then writeVMStatus. ReturnsRequeueError(5s)while a VM is mid-boot so it polls untilrunning, without counting as a crash.
The VMController's reconcile, sketched — note how external state (libvirt) is just another thing you converge toward:
func (c *VMController) Reconcile(ctx context.Context, log *zap.Logger,
r controller.QRuntime, ptr resource.Pointer) error {
cfg, err := safe.ReaderGet[*VMConfig](ctx, r, ptr)
if state.IsNotFoundError(err) { return nil } // gone; nothing to do
if err != nil { return err }
if cfg.Metadata().Phase() == resource.PhaseTearingDown {
if err := c.hypervisor.Destroy(ctx, cfg.Metadata().ID()); err != nil { return err }
return r.RemoveFinalizer(ctx, cfg.Metadata(), c.Name()) // safe to delete now
}
if err := r.AddFinalizer(ctx, cfg.Metadata(), c.Name()); err != nil { return err }
// converge the hypervisor toward the desired spec
actual, err := c.hypervisor.Ensure(ctx, toDomainSpec(cfg.TypedSpec()))
if err != nil { return err }
if actual.State != "running" {
// not there yet — poll again soon, without logging a crash
return controller.NewRequeueInterval(5 * time.Second)
}
// publish observed state
return safe.WriterModify(ctx, r, NewVMStatus(cfg.Metadata().ID()),
func(s *VMStatus) error { s.TypedSpec().State = actual.State; s.TypedSpec().IP = actual.IP; return nil })
}
What you get for free here: the API only writes VMConfig; the UI watches VMStatus live; deleting a VM safely tears down its network first; thousands of VMs reconcile concurrently with per-VM backoff; and the whole thing runs in one process for dev and as a central-store-plus-agents fleet in prod with no controller changes.
#Blueprint: a workflow / orchestration engine greenfield
For a system whose job is driving multi-step workflows or coordinating state machines, COSI models each workflow run and each step as resources, and uses controllers as the step executors. The reconcile loop is the state machine's transition function.
Resources
Workflow (desired: the DAG + inputs) · WorkflowStatus (which steps done) · StepRun (one node, shared output, labelled by workflow=) · StepResult (observed output of a step).
Controllers
PlannerController (transform Workflow → StepRuns) expands the DAG into runnable steps as dependencies complete. StepController (QController over StepRun) executes each step, writes StepResult, retries with backoff. StatusController aggregates results into WorkflowStatus.
- Dependencies between steps become strong inputs + finalizers: a downstream
StepRunisn't created until its prerequisites'StepResults exist; teardown unwinds in reverse. - "Fan-out over N items" is a label query: emit N
StepRuns labelledbatch=abc, let the QController chew through them concurrently, aggregate by querying the label. - Durability & resume come from the bolt backing store + bookmarks: restart the process and in-flight workflows resume from exactly where the change stream left off.
- Pause/cancel is just
Teardownon theWorkflow— finalizers ensure running steps stop cleanly before the run is destroyed.
You don't write a scheduler, a retry mechanism, an event bus, or persistence — those are the runtime. You write the transition logic per step type as a small reconcile function, and the graph of step types is the workflow definition. Adding a new step type is adding a controller, not surgery on a monolith.
#Integration patterns
Embed in an existing Go service
COSI is a library, not a framework that owns main. Construct the state and runtime alongside your HTTP server, share the state.State handle, and let your handlers do typed writes that controllers pick up:
st := state.WrapCore(namespaced.NewState(inmem.Build))
rt, _ := runtime.NewRuntime(st, logger, options.WithMetrics(true))
registerControllers(rt) // your transform/qtransform/QControllers
go rt.Run(ctx)
// an HTTP handler just writes a desired-state resource; controllers do the rest
http.HandleFunc("/vms", func(w http.ResponseWriter, req *http.Request) {
spec := decode(req)
_ = st.Create(req.Context(), vm.NewVMConfig(spec.ID, spec))
w.WriteHeader(202) // accepted; reconciliation is async
})
Expose the store as your API
Or skip bespoke handlers entirely: run server.NewState and let clients do typed CRUD + Watch over gRPC, guarded by a state.Filter RBAC rule. Your "API" becomes "the resource model," which is hard to drift out of sync with reality.
#Bridging the outside world
Reconciliation needs to react to things that aren't resources — a hypervisor event, a webhook, a timer, an inotify. Two clean bridges:
External → resources
A small "observer" goroutine watches the external system and writes status resources (or bumps an annotation) when reality changes. That write flows through the normal watch pipeline and wakes the relevant controller. The classic Controller also supports an extra event channel to be poked directly.
Resources → external
That's just a controller's reconcile making the API call (as VMController.Ensure above). Wrap non-idempotent calls so reconcile stays idempotent — reconcile may run many times; the side-effect should converge, not duplicate.
A reconcile function can be called any number of times for the same state. Design every side-effect as "ensure X exists / matches," never "create X." This is the same contract Kubernetes controllers live by, and it's what makes retries and restarts safe.
#Testing & rollout
- Unit-test controllers against in-memory state — construct
inmemstate + runtime, register the controller, write inputs, assert outputs. Thertestutilspackage hasAssertResourceshelpers for exactly this; tests run in milliseconds with no external dependencies. - Conformance suite — if you write a custom
CoreStatebackend,state/conformanceis a ready-made test suite that asserts correct CRUD, watch, bookmark, and finalizer semantics. - Start as a monolith. Ship the in-process topology first. Only split to central-store-plus-agents when you actually need horizontal scale or process isolation — and when you do, no controller changes.
- Migrate incrementally. You can stand up COSI alongside an existing system: model one slice of your domain as resources, run one controller, and expand. It doesn't demand a rewrite to start paying off.
#Pitfalls & anti-patterns
| Anti-pattern | Why it bites | Do instead |
|---|---|---|
| Two controllers writing one exclusive type | Runtime refuses to start (by design) | One owner per type; use shared outputs or split the type |
| Observed state stuffed into the spec resource | Spec gets two writers; ownership conflicts | Separate spec (desired) and status (observed) resources |
| Non-idempotent side-effects in reconcile | Retries duplicate work; restarts double-create | "Ensure/converge," never "create" |
Resource stuck in tearingDown | A finalizer nobody removes | Find the owning controller; it crashed or its dependency never cleared |
| Giant specs / blob storage | Bounded watch history, full copies per event | Keep specs small; reference bulk data, don't embed it |
| Cross-resource transactions | COSI has no multi-object atomicity | Model the invariant as a controller that converges; eventual consistency |
#Where to go from here
Re-read the model
Resources, state, controllers — the three chapters that define your design vocabulary.
Chapters 01–03 ›Read the conformance controllers
pkg/controller/conformance has small, real controllers (IntToStr, Sum) — the best worked examples in the repo.
Prototype one slice
Pick the smallest reconcilable thing in your own domain, model it as one resource + one transform controller, and watch it converge.
Model your domain as typed resources in a state broker, write the convergence logic as single-writer controllers, let the runtime engine wake them precisely, and reach for the gRPC transport when one process isn't enough — and you've turned "a pile of imperative management scripts" into "a self-healing control plane" with the same code from laptop to fleet. The next chapter grounds all of this in Talos Linux — the canonical real-world application of everything here — because the best way to learn the framework is to read it at full depth on real hardware.