Kubernetes & etcd

Talos runs Kubernetes without kubeadm, without systemd units, without operator hand-editing. The entire control plane is derived from machine config by COSI controllers: config flows through *Config resources into rendered artifacts, and a reconciliation loop turns those into running processes. Exactly two things run as Talos-supervised containerd services — etcd and the kubelet. Everything else is a kubelet static pod, but for the control plane Talos never writes a manifest to disk: it serves the pods over a localhost HTTP endpoint and points the kubelet at the URL.

#Kubernetes as a reconciled appliance

A conventional Kubernetes install is a sequence of imperative steps: kubeadm init writes certificates, drops static-pod manifests into /etc/kubernetes/manifests, starts a kubelet against them, then kubeadm join bolts on the rest. The resulting state lives on disk and drifts the moment anyone edits it. Talos discards that model entirely. There is no kubeadm binary on the machine and no systemd unit for any Kubernetes component. Instead, the control plane is a reconciliation graph built on the same COSI runtime that drives the rest of the OS, living in internal/app/machined/pkg/controllers/k8s/ and .../controllers/etcd/.

The split is the thing to internalize. Two processes are Talos-managed containerd services in the system namespace: etcd (system/services/etcd.go) and the kubelet (system/services/kubelet.go). Everything else — kube-apiserver, kube-controller-manager, kube-scheduler — is a kubelet static pod. But unlike kubeadm, Talos renders the control-plane pods into in-memory k8s.StaticPod resources, marshals them into a single PodList, serves that over a 127.0.0.1 HTTP endpoint, and hands the kubelet that URL via its staticPodURL.

#etcd: Config → Spec → PKI → Service

etcd is the foundation, so it earns its own controller chain. Four stages turn machine config into a running, TLS-secured member, and each stage is gated so it only fires on a control-plane node — etcd.ConfigController short-circuits unless Machine().Type().IsControlPlane().

1. Config. controllers/etcd/config.go transforms MachineConfig into an etcd.Config resource — the control-plane-only knobs (image, extra args, subnet filters).
2. Spec. controllers/etcd/spec.go resolves node addresses into concrete AdvertisedAddresses, ListenPeerAddresses, and ListenClientAddresses. Crucially it excludes Kubernetes-internal addresses and VIPs (via k8s.NodeAddressFilterNoK8s plus the virtual-IP filter) — etcd must advertise a stable routed address, never the floating control-plane VIP.
3. PKI. controllers/etcd/pki.go materializes the etcd CA plus member, peer, and admin keypairs onto /system/secrets/etcd from secrets.Etcd / secrets.EtcdRoot, and emits etcd.PKIStatus.
4. Service. system/services/etcd.go runs upstream etcd in containerd with dropped capabilities, the host network namespace, a TLS 1.3 floor, and client + peer certificate auth.

Once running, etcd's health is checked with ValidateQuorum every 20 seconds, and that Healthy flag becomes the universal gate for the rest of the chapter — the static-pod controller, the manifest applier, and the member controllers all block on it, because the kube-apiserver points only at local etcd.

#Bootstrap vs. join

argsForControlPlane in the etcd service decides how a member enters the cluster, branching on whether the data directory is empty and whether this node is the bootstrap node:

if e.Bootstrap {
    initialCluster = formatClusterURLs(spec.Name, getEtcdURLs(spec.AdvertisedAddresses, constants.EtcdPeerPort))
} else {
    initialCluster, e.learnerMemberID, err = buildInitialCluster(ctx, r, spec.Name, getEtcdURLs(...))
}

A learner cannot vote, so it cannot break quorum while catching up on the Raft log. When it joins as a learner, the service spawns a background goroutine running promoteMember(), which retries MemberPromote for up to 10 minutes until the member is caught up and becomes a voter — fire-and-forget retry. addMember also defensively removes any stale member sharing this hostname before adding the new one, so a re-imaged node rejoins cleanly.

#The control plane as static pods

With etcd up, the control-plane components are produced by a two-pass transform pipeline plus a producer. The two passes are the heart of the design: they separate "what the user asked for" from "the exact command line that will run."

1. control_plane.go — the first pass. One transform controller per component. NewControlPlaneAPIServerController maps config into k8s.APIServerConfig (image, endpoint, service CIDRs, advertised address), with siblings producing ControllerManagerConfig and SchedulerConfig. The controlplaneMapFunc drops the output entirely on worker nodes. Note the apiserver's etcd endpoint is hard-wired to local etcd:

EtcdServers: []string{fmt.Sprintf("https://%s", nethelpers.JoinHostPort("127.0.0.1", constants.EtcdClientPort))},
AdvertisedAddress: advertisedAddress, // "$(POD_IP)"

2. control_plane_final.go — the second pass. Produces a Final config whose .Args is the full kube-apiserver command line, assembled with argsbuilder using MergeDenied for the security-critical flags. That merge policy is what replaces kubeadm's hardcoded argv: user extra-args simply cannot override flags like the authorization mode or the etcd cert paths.
3. control_plane_static_pod.go — the producer. It waits for etcd to be healthy, then calls k8stemplates.APIServerPod() to build a *v1.Pod and writes a k8s.StaticPod resource. The pod is system-cluster-critical, gets POD_IP from the downward API, and mounts secrets and config from host paths.

if etcdResource != nil && etcdResource.TypedSpec().Healthy && configStatusResource != nil && secretsStatusResource != nil {
    for _, manageFunc := range []func(...){ctrl.manageAPIServer, ctrl.manageControllerManager, ctrl.manageScheduler} {
        if err = manageFunc(ctx, r, logger, secretsVersion, configVersion); err != nil {
            return err
        }
    }
}

Finally, static_pod_server.go lists every k8s.StaticPod, marshals them into one YAML PodList, and serves it on an ephemeral 127.0.0.1 port — recording the URL in StaticPodServerStatus. The control-plane pod manifests never touch disk; they live in HTTP memory.

listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", "127.0.0.1:0")
// ...
r.TypedSpec().URL = fmt.Sprintf("http://%s", listener.Addr().String())

#The kubelet service

The kubelet is the other Talos-managed service, and it too is reconciled in three stages. kubelet_config.go transforms config into a k8s.KubeletConfig and reads StaticPodServerStatus to inject the StaticPodListURL — that is the wire that connects the HTTP PodList above to a running kubelet. kubelet_spec.go then builds the argument set and an opinionated KubeletConfiguration via NewKubeletConfiguration: webhook authn/authz, anonymous auth off, RotateCertificates: true, ProtectKernelDefaults: true, a TLS 1.3 floor, system-reserved resource carve-outs, and both a StaticPodPath of /etc/kubernetes/manifests (on-disk, for user static pods) and a StaticPodURL (HTTP, for the control plane). On control-plane nodes it injects the node-role.kubernetes.io/control-plane:NoSchedule taint unless AllowSchedulingOnControlPlane is set.

kubelet_service.go writes the kubelet PKI, a bootstrap-token kubeconfig for TLS bootstrapping, /etc/kubernetes/kubelet.yaml, and the credential-provider config, then restarts the kubelet. It also self-heals: refreshKubeletCerts wipes the PKI if the client certificate's CN is no longer system:node:<nodename>, forcing a clean re-bootstrap. The service runs the kubelet image with host network and PID namespaces, a large mount set, and a custom seccomp profile.

#Bootstrap & manifests

None of the above produces a cluster until someone fires the one-time ignition. talosctl bootstrap calls the Bootstrap gRPC handler in internal/server/v1alpha1/v1alpha1_server.go, which guards: the call is allowed, this is a control-plane node, time is synced, and — the irreversible part — /var/lib/etcd is empty.

if entries, _ := os.ReadDir(constants.EtcdDataPath); len(entries) > 0 {
    return nil, status.Error(codes.AlreadyExists, "etcd data directory is not empty")
}

That single check is the entire enforcement mechanism for "bootstrap is one-time, one-node." When it passes, the handler calls services.BootstrapEtcd: it stops the join-mode etcd, fakes StateFinished to unblock boot, and reloads etcd with Bootstrap: true. Every other control-plane node skips this entirely and joins as a learner automatically. In practice you run it once against any one control-plane node:

# Run exactly once, against a single control-plane node:
talosctl --nodes 10.0.0.10 bootstrap

# etcd comes up, the apiserver static pod starts, the cluster forms.
# Then pull a working kubeconfig and talk to it:
talosctl --nodes 10.0.0.10 kubeconfig ./kubeconfig
kubectl --kubeconfig ./kubeconfig get nodes

# Re-running bootstrap anywhere now fails fast:
#   rpc error: code = AlreadyExists desc = etcd data directory is not empty

Once the apiserver answers, the bootstrap manifests apply. ControlPlaneBootstrapManifestsController computes CoreDNS, kube-proxy (in nftables mode for k8s ≥ 1.31), and the Flannel CNI (with the MTU defaulting to KubeSpan's if enabled). manifest.go renders these into k8s.Manifest resources, and manifest_apply.go applies them after etcd is healthy, under a distributed etcd lock (EtcdTalosManifestApplyMutex) and idempotently via a server-side-apply inventory: every object is tagged, and anything already present is skipped. Namespaces sort first, then CRDs, then the rest — so an HA control plane with three apiservers never double-applies CNI or CoreDNS.

#Node lifecycle

The last piece is the node's own Kubernetes identity, handled by a small cluster of controllers. nodename.go derives the node name from the hostname (or the FQDN, if RegisterWithFQDN is set). nodeip.go selects --node-ip from the routed addresses, filtering out Kubernetes-internal ones — reusing the same NodeAddress sets the networking chapter produced. And NodeApplyController in node_apply.go watches NodeLabelSpec, NodeTaintSpec, NodeAnnotationSpec, and NodeCordonedSpec and applies them onto the live v1.Node.

That is Kubernetes as a reconciled appliance: etcd and the kubelet supervised as Talos services, the control plane rendered from config into static pods that never touch disk, a one-shot bootstrap gated by an empty-data-dir check, and node identity applied without a scheduling race — all converging from machine config with no kubeadm in sight. We have been talking about this machine from the outside; the next chapter opens up the channel you actually use to do it — the API: apid & talosctl.