apply  /  using it recipes
Chapter 06

Using It

Now the practical question you actually came for: where does COSI earn its keep in your systems? This chapter gives a decision framework, a one-page mapping from domain concepts to COSI primitives, and two greenfield blueprints — a VM control plane and a workflow/orchestration engine — plus the integration patterns for dropping it into an existing service. The next chapter dissects Talos as the canonical real-world application of these patterns.

#Is COSI the right tool?

COSI shines for exactly one shape of problem: declarative reconciliation — "record the desired state, then continuously drive reality toward it." Score your use case:

✓ Reach for COSI when…

You have desired vs. actual state to reconcile · long-lived objects with lifecycles (create → update → teardown) · derived/dependent objects (a VM needs a network, a disk, an IP) · ordered cleanup requirements · the need to react to changes, not just serve requests · a wish to scale from one process to many later.

✗ Skip COSI when…

It's a stateless request/response API with no background convergence · a one-shot batch job · purely a CRUD database with no reactive logic · you need multi-row transactions or complex relational queries · ultra-low-latency single-digit-µs paths (the reconcile loop adds scheduling latency by design).

The litmus test: if you find yourself writing a polling loop that reads some records, computes what should exist, and makes it so — that loop is a COSI controller waiting to be born.

#The domain → COSI mapping

Almost every COSI design is the same translation exercise. Keep this table next to you:

Your domain has…Model it as…Because
A managed thing (VM, job, device, tenant)a resource type with a typed specit has identity + desired state
"What the user/API asked for"a spec resource (e.g. VMConfig)desired state, written by the edge
"What's actually true right now"a status resource (e.g. VMStatus)observed state, written by a controller
A rule that derives B from Aa transform controller A → B1:1 reconciliation with cleanup
"For each X, ensure side-effect"a QController over Xper-item, concurrent, retrying
"X must outlive Y's cleanup"a strong input + finalizerordered teardown
"All things on host H"a label + WithLabelQueryqueryable grouping, no joins
A grouping/partitiona namespaceisolation + independent stores
The external API / CLI / UIthe gRPC State servicetyped CRUD + Watch for free
The spec/status split is the key habit.

Resist putting observed state into the spec resource the user writes. Keep desired (spec, user-owned) and observed (status, controller-owned) as separate resources. It keeps ownership clean (no two writers), makes the reconcile direction obvious, and lets the UI show "requested 4 CPUs / running with 4 CPUs" by reading both.

#Blueprint: a VM control plane greenfield

VM lifecycle management is COSI's home turf — it's essentially what Talos does for machines. Here's how you'd model it if starting fresh on COSI. Model the platform as a small graph of resources and controllers:

VMConfig desired · user-written NetworkController qtransform VMController QController → hypervisor PlacementController picks a Host NetworkConfig bridge · IP VMStatus observed · ctrl-written Host capacity · label target hypervisor libvirt / QEMU side-effect net ready
Desired VMConfig fans out to networking, placement, and the hypervisor-driving VMController, which reports back through VMStatus.

The resources

  • VMConfig — desired VM (CPUs, memory, image, power state). Written by the API/UI. Namespace vms.
  • VMStatus — observed VM (actual power state, IP, last error). Written only by VMController.
  • NetworkConfig — derived network plumbing for a VM. Owned by NetworkController.
  • Host — a hypervisor node with capacity, carrying labels like host=node-7.

The controllers

  1. PlacementController (QController over VMConfig) — reads Host capacity, assigns a host by stamping a host= label on a VMPlacement output.
  2. NetworkController (qtransform VMConfigNetworkConfig) — provisions a bridge/IP; uses an input finalizer so a VM can't be destroyed until its network is torn down (the cooperative-deletion pattern from chapter 01).
  3. VMController (QController over VMConfig) — the one that touches the hypervisor. Reconcile = "make libvirt match the spec," then write VMStatus. Returns RequeueError(5s) while a VM is mid-boot so it polls until running, without counting as a crash.

The VMController's reconcile, sketched — note how external state (libvirt) is just another thing you converge toward:

vm/vmcontroller.gogo
func (c *VMController) Reconcile(ctx context.Context, log *zap.Logger,
    r controller.QRuntime, ptr resource.Pointer) error {

    cfg, err := safe.ReaderGet[*VMConfig](ctx, r, ptr)
    if state.IsNotFoundError(err) { return nil }          // gone; nothing to do
    if err != nil { return err }

    if cfg.Metadata().Phase() == resource.PhaseTearingDown {
        if err := c.hypervisor.Destroy(ctx, cfg.Metadata().ID()); err != nil { return err }
        return r.RemoveFinalizer(ctx, cfg.Metadata(), c.Name())   // safe to delete now
    }
    if err := r.AddFinalizer(ctx, cfg.Metadata(), c.Name()); err != nil { return err }

    // converge the hypervisor toward the desired spec
    actual, err := c.hypervisor.Ensure(ctx, toDomainSpec(cfg.TypedSpec()))
    if err != nil { return err }
    if actual.State != "running" {
        // not there yet — poll again soon, without logging a crash
        return controller.NewRequeueInterval(5 * time.Second)
    }

    // publish observed state
    return safe.WriterModify(ctx, r, NewVMStatus(cfg.Metadata().ID()),
        func(s *VMStatus) error { s.TypedSpec().State = actual.State; s.TypedSpec().IP = actual.IP; return nil })
}

What you get for free here: the API only writes VMConfig; the UI watches VMStatus live; deleting a VM safely tears down its network first; thousands of VMs reconcile concurrently with per-VM backoff; and the whole thing runs in one process for dev and as a central-store-plus-agents fleet in prod with no controller changes.

#Blueprint: a workflow / orchestration engine greenfield

For a system whose job is driving multi-step workflows or coordinating state machines, COSI models each workflow run and each step as resources, and uses controllers as the step executors. The reconcile loop is the state machine's transition function.

Resources

Workflow (desired: the DAG + inputs) · WorkflowStatus (which steps done) · StepRun (one node, shared output, labelled by workflow=) · StepResult (observed output of a step).

Controllers

PlannerController (transform WorkflowStepRuns) expands the DAG into runnable steps as dependencies complete. StepController (QController over StepRun) executes each step, writes StepResult, retries with backoff. StatusController aggregates results into WorkflowStatus.

  • Dependencies between steps become strong inputs + finalizers: a downstream StepRun isn't created until its prerequisites' StepResults exist; teardown unwinds in reverse.
  • "Fan-out over N items" is a label query: emit N StepRuns labelled batch=abc, let the QController chew through them concurrently, aggregate by querying the label.
  • Durability & resume come from the bolt backing store + bookmarks: restart the process and in-flight workflows resume from exactly where the change stream left off.
  • Pause/cancel is just Teardown on the Workflow — finalizers ensure running steps stop cleanly before the run is destroyed.
Why this beats a hand-rolled state machine.

You don't write a scheduler, a retry mechanism, an event bus, or persistence — those are the runtime. You write the transition logic per step type as a small reconcile function, and the graph of step types is the workflow definition. Adding a new step type is adding a controller, not surgery on a monolith.

#Integration patterns

Embed in an existing Go service

COSI is a library, not a framework that owns main. Construct the state and runtime alongside your HTTP server, share the state.State handle, and let your handlers do typed writes that controllers pick up:

embeddinggo
st := state.WrapCore(namespaced.NewState(inmem.Build))
rt, _ := runtime.NewRuntime(st, logger, options.WithMetrics(true))
registerControllers(rt)            // your transform/qtransform/QControllers
go rt.Run(ctx)

// an HTTP handler just writes a desired-state resource; controllers do the rest
http.HandleFunc("/vms", func(w http.ResponseWriter, req *http.Request) {
    spec := decode(req)
    _ = st.Create(req.Context(), vm.NewVMConfig(spec.ID, spec))
    w.WriteHeader(202) // accepted; reconciliation is async
})

Expose the store as your API

Or skip bespoke handlers entirely: run server.NewState and let clients do typed CRUD + Watch over gRPC, guarded by a state.Filter RBAC rule. Your "API" becomes "the resource model," which is hard to drift out of sync with reality.

#Bridging the outside world

Reconciliation needs to react to things that aren't resources — a hypervisor event, a webhook, a timer, an inotify. Two clean bridges:

External → resources

A small "observer" goroutine watches the external system and writes status resources (or bumps an annotation) when reality changes. That write flows through the normal watch pipeline and wakes the relevant controller. The classic Controller also supports an extra event channel to be poked directly.

Resources → external

That's just a controller's reconcile making the API call (as VMController.Ensure above). Wrap non-idempotent calls so reconcile stays idempotent — reconcile may run many times; the side-effect should converge, not duplicate.

Idempotency is the one discipline COSI asks of you.

A reconcile function can be called any number of times for the same state. Design every side-effect as "ensure X exists / matches," never "create X." This is the same contract Kubernetes controllers live by, and it's what makes retries and restarts safe.

#Testing & rollout

  • Unit-test controllers against in-memory state — construct inmem state + runtime, register the controller, write inputs, assert outputs. The rtestutils package has AssertResources helpers for exactly this; tests run in milliseconds with no external dependencies.
  • Conformance suite — if you write a custom CoreState backend, state/conformance is a ready-made test suite that asserts correct CRUD, watch, bookmark, and finalizer semantics.
  • Start as a monolith. Ship the in-process topology first. Only split to central-store-plus-agents when you actually need horizontal scale or process isolation — and when you do, no controller changes.
  • Migrate incrementally. You can stand up COSI alongside an existing system: model one slice of your domain as resources, run one controller, and expand. It doesn't demand a rewrite to start paying off.

#Pitfalls & anti-patterns

Anti-patternWhy it bitesDo instead
Two controllers writing one exclusive typeRuntime refuses to start (by design)One owner per type; use shared outputs or split the type
Observed state stuffed into the spec resourceSpec gets two writers; ownership conflictsSeparate spec (desired) and status (observed) resources
Non-idempotent side-effects in reconcileRetries duplicate work; restarts double-create"Ensure/converge," never "create"
Resource stuck in tearingDownA finalizer nobody removesFind the owning controller; it crashed or its dependency never cleared
Giant specs / blob storageBounded watch history, full copies per eventKeep specs small; reference bulk data, don't embed it
Cross-resource transactionsCOSI has no multi-object atomicityModel the invariant as a controller that converges; eventual consistency

#Where to go from here

📐

Re-read the model

Resources, state, controllers — the three chapters that define your design vocabulary.

Chapters 01–03 ›
🔬

Read the conformance controllers

pkg/controller/conformance has small, real controllers (IntToStr, Sum) — the best worked examples in the repo.

🚀

Prototype one slice

Pick the smallest reconcilable thing in your own domain, model it as one resource + one transform controller, and watch it converge.

The pattern in one sentence.

Model your domain as typed resources in a state broker, write the convergence logic as single-writer controllers, let the runtime engine wake them precisely, and reach for the gRPC transport when one process isn't enough — and you've turned "a pile of imperative management scripts" into "a self-healing control plane" with the same code from laptop to fleet. The next chapter grounds all of this in Talos Linux — the canonical real-world application of everything here — because the best way to learn the framework is to read it at full depth on real hardware.