The API — apid, trustd & talosctl

Talos has no SSH, no shell, no local login. There is exactly one way to touch a node: a single mutual-TLS gRPC endpoint served by apid on TCP :50000. Rebooting, applying config, reading a file, pulling an etcd snapshot, listing processes — every one is a gRPC call, and your identity (and what you may do) is carried in the client certificate itself. This chapter is how that single door works: how apid routes and proxies, how trustd hands out certs, how the role model is enforced, and how talosctl dials in.

#One door, not a hundred

Most operating systems expose a sprawl of management surfaces: SSH, a serial console, a local root shell, a config-management agent, maybe a web UI. Talos collapses all of that into one surface. The API is the interface. If you cannot express something as a gRPC call to apid, you cannot do it — there is no fallback shell to drop into. That single-door design is what makes the rest of the security story (Chapter 08) tractable: there is exactly one authenticated, authorized, audited boundary to reason about.

Three small daemons make it work, and it helps to keep their roles straight:

The end-to-end flow is one sentence: talosctl --(mTLS, client cert)--> apid (endpoint node) --(mTLS or unix socket)--> apid/machined (target node). Everything below unpacks the pieces of that line.

#apid as a near-stateless proxy

The first surprising thing about apid: it registers almost no concrete gRPC services of its own. It is built on grpc-proxy and installs a proxy.TransparentHandler(router.Director, …) as the gRPC UnknownServiceHandler. Raw gRPC frames are forwarded as opaque bytes — which is why a brand-new MachineService RPC works through apid the day it lands, with no apid change. apid never needs to know the shape of the messages it relays.

TLS termination with a live CA. apid is configured reactively from a COSI resource: apidMain() watches runtime.APIServiceConfig and restarts the listener when it changes. The server TLS config comes from provider.NewTLSConfig, which watches the secrets.API resource and hot-reloads its certificate — half-life rotation with no process restart. It requests tls.Mutual with a dynamic client CA, and enforces that client leaf certs carry only ExtKeyUsageClientAuth via verifyExtKeyUsage. A cert minted for server auth cannot be presented as a client identity at the door — remember that, it matters when we get to trustd.

Routing is the heart of apid. For every call, Router.Director decides where it goes. The switch is short, and it is the single most important piece of code on this page:

switch {
case okNodes:   // one-to-many, allow-listed services only
	return r.aggregateDirector(nodes)
case okNode:    // one-to-one to a single remote
	return r.singleDirector(node[0])
default:        // local node, skip another proxy layer
	return proxy.One2One, []proxy.Backend{r.localBackend}, nil
}

Read it top to bottom. If the request carries a nodes (plural) metadata list, it becomes a one-to-many fan-out (aggregateDirector → One2Many). If it carries a single node, it is a one-to-one proxy to that remote (singleDirector → One2One). Otherwise — no routing metadata at all — it is served locally by backend.NewLocal("machined", MachineSocketPath), skipping the extra proxy hop entirely. Two more cases sit upstream of this switch: a skipRouting/no-metadata short-circuit also goes to the local backend, and a proxyfrom marker means the call already arrived from another apid, so it terminates locally instead of bouncing forever (loop prevention).

#Endpoints vs nodes — the one mental model to keep

This is the #1 conceptual trap in the Talos API, and it is worth slowing down for. There are two different "where" questions, and they have two different answers:

So when you run talosctl -e cp1 -n worker3 reboot, you open mTLS to cp1's apid, which reads node=worker3 from the metadata and proxies the Reboot RPC over a second mTLS hop (apid→apid) to worker3. The endpoint is a control-plane node you happen to trust; the node is the worker you want rebooted. And the optimization in the default case above means that if the node you target is the endpoint you dialed, apid short-circuits straight to the local machined socket — no second proxy hop.

#The MachineService breadth

The bulk of the API is MachineService (api/machine/machine.proto) — roughly seventy RPCs. Because apid forwards opaquely, the whole surface is available through one door. It is easiest to see in a taxonomy:

Alongside MachineService, apid fronts cluster.ClusterService, inspect.InspectService, storage.StorageService, time.TimeService, machine.LifecycleService, and machine.ImageService — plus the raw COSI resource API, cosi.resource.State. That last one is the same State broker the COSI runtime uses internally, exposed (read-only) over the wire: talosctl get is literally a List/Watch against it. The whole declarative world of Chapter 02 is visible through the same authenticated door.

#trustd & certificate issuance

Every node's apid needs a TLS server certificate signed by the cluster's OS root CA (secrets.OSRoot, whose IssuingCA holds both cert and key). How a node obtains one depends on what kind of node it is (controllers/secrets/api.go):

1. Control-plane / init nodes hold the issuing CA locally. generateControlPlane mints the apid server certificate and a role-bearing client certificate directly — they can self-sign because they already have the key.
2. Worker nodes have no CA key. generateWorker builds a CSR and calls trustd: remoteGen.IdentityContext(ctx, serverCSR), which is the SecurityService.Certificate RPC against a control-plane node on :50001.

Here is the security-critical detail. trustd will sign a worker's CSR, but only as a server certificate: it forces ExtKeyUsageServerAuth and it strips any Organization from the subject — and the Organization is exactly where the role lives.

if len(request.Subject.Organization) > 0 {
	x509Opts = append(x509Opts, x509.OverrideSubject(func(subject *pkix.Name) {
		subject.Organization = nil   // strip the role
	}))
}

Trace the consequence: a worker can obtain a perfectly valid TLS server identity, but it can never mint itself a role-bearing client certificate. Even if a worker asked for Organization=os:admin, trustd nulls it out. And recall that apid's door enforces ExtKeyUsageClientAuth only on client leaves — so a server cert can't be presented as a client identity anyway. trustd is structurally incapable of privilege escalation. Certs everywhere auto-rotate at half their validity via resource Watch, no restarts.

#The role / RBAC model

Roles are os:-prefixed strings, defined in pkg/machinery/role/role.go:

const (
	Admin    = Role("os:admin")    // every API
	Operator = Role("os:operator") // reader + mgmt (e.g. reboot)
	Reader   = Role("os:reader")   // read-only, no secrets
	EtcdBackup    = Role("os:etcd:backup")
	ImageVerifier = Role("os:image:verifier")
	Impersonator  = Role("os:impersonator")
)

Where roles live: in the client certificate's Subject Organization field. A cert can carry several, so a caller holds a set of roles. The Injector middleware (in Enabled mode) reads them straight off the verified peer cert:

strings := tlsInfo.State.PeerCertificates[0].Subject.Organization
roles, unknownRoles := role.Parse(strings)

Two-stage enforcement. Both apid and machined chain an Injector then an Authorizer, but they inject differently. apid runs the Injector in Enabled mode — it extracts roles from the actual client cert it just verified. machined runs it in MetadataOnly mode — it does not re-parse any cert; it trusts apid's assertion, carried in the talos-role metadata key over the local unix socket. The trust boundary is apid's TLS termination plus the private socket; machined deliberately delegates identity to apid.

Then the Authorizer makes the per-RPC decision against a rules map:

allowedRoles, found := a.Rules[method]
if !found {
	allowedRoles = a.FallbackRoles   // machined: admin-only
}
if allowedRoles.IncludesAny(GetRoles(ctx)) {
	return nil
}
return ErrNotAuthorized   // codes.PermissionDenied

The fallback is the safety net: an RPC with no rule entry falls through to FallbackRoles, which on machined is admin-only. Forget to add a rule for a new method and it fails closed, not open. The rules table (var rules in machined.go) is readable at a glance: Bootstrap, Reset, Upgrade, Kubeconfig, Copy, and Read require os:admin; read-only stats accept {Admin, Operator, Reader}; EtcdSnapshot accepts {Admin, Operator, EtcdBackup}.

#talosctl & talosconfig

The client side mirrors the server model exactly. A talosconfig (pkg/machinery/client/config/config.go) is a set of named contexts, each a small struct:

type Context struct {
	Endpoints []string `yaml:"endpoints"`
	Nodes     []string `yaml:"nodes,omitempty"`
	CA        string   `yaml:"ca,omitempty"`
	Crt       string   `yaml:"crt,omitempty"`
	Key       string   `yaml:"key,omitempty"`
}

• Endpoints — the apid hosts the client dials over mTLS, round-robined. These are your dial targets.
• Nodes — default targets, sent as routing metadata, never dialed. WithNode(ctx, node) (in pkg/machinery/client/context.go) sets the node metadata key per call; WithNodes sets the plural fan-out form.
• CA / Crt / Key — base64 PEM. The client cert's Organization encodes the role, so the same talosconfig that lets you connect also fixes what you may do.

That is why the -e and -n flags are separate, and why mixing them up is the classic beginner mistake:

# dial cp1's apid (the ENDPOINT); proxy Reboot to worker3 (the NODE)
talosctl -e cp1 -n worker3 reboot

# one endpoint, fan out to three nodes (one-to-many, replies tagged per host)
talosctl -e cp1 -n worker1,worker2,worker3 version

# endpoint == node: apid short-circuits to local machined, no proxy hop
talosctl -e cp1 -n cp1 get members

# persist defaults in the context so you can drop the flags
talosctl config endpoint cp1 cp2 cp3
talosctl config node    worker3

Step back and the whole chapter is one idea applied consistently: there is a single mTLS gRPC door, apid; identity and authorization travel in the certificate; apid is a thin proxy that routes by metadata while machined and trustd do the real work behind it; and "where you dial" is deliberately decoupled from "where it runs." Everything you can do to a Talos node, you do through that one surface — which is precisely what makes the trust model auditable. Chapter 08 goes underneath it: the full PKI and secret material, node hardening and the read-only immutable rootfs, measured boot and disk encryption, and the WireGuard mesh (SideroLink) that ties it together.