The 23 July etcd patch release fixed a watch authorization problem that could disclose events outside a user’s RBAC key range. The fix is available in v3.7.1, v3.6.14, and v3.5.33. This is a good reason to patch, but the release is also a useful reminder that an etcd maintenance window should prove more than a new binary starts.
This guide is for an administrator of a self-managed, TLS-enabled three-member etcd cluster. The finished result is a patched cluster with a verified endpoint health check, a recorded member view, and an off-node snapshot whose integrity has been checked. The commands are documentation-verified examples, not output from a production cluster. Replace the certificate paths, endpoint addresses, service name, package method, and backup location with values from your own installation.
Do not apply this procedure directly to a managed Kubernetes control plane. The provider owns its etcd lifecycle. For a kubeadm or otherwise self-managed control plane, first follow the Kubernetes distribution’s control-plane guidance; Kubernetes also stresses that etcd needs a backup plan before an upgrade.
Decide which release applies
The current etcd project advisory says the affected releases can send a user who has read access to one key watch events for keys at and after that key. That matters where applications use etcd RBAC to separate tenants, environments, or controllers. It is not a reason to disable watches or RBAC; it is a reason to move to the current patch in the minor line that the cluster already supports.
Start by recording the server and client versions on every member. Run the command locally on each node, or use your normal configuration-management inventory if it records package versions.
etcd --version
etcdctl version
A patch maintenance window should not casually become a minor-version migration. A cluster on 3.5 should target 3.5.33, a cluster on 3.6 should target 3.6.14, and a cluster on 3.7 should target 3.7.1. If members report different minor versions, stop and resolve that inconsistency using the documented upgrade path before treating this as a simple patch update. etcd’s upgrade documentation requires a healthy cluster for a rolling upgrade.
Check that the planned package, container image, or release archive is supplied by the official project or the operating-system vendor. For a manually installed archive, verify the published checksum and signature before replacing a binary. Keep the previous known-good binary available according to your change policy, but do not copy an older data directory over a newer one as an improvised rollback.
Establish a client context
etcdctl needs to reach the client URLs and, on a secured cluster, present credentials that the cluster accepts. Set the API version explicitly and keep the TLS material out of shell history and source control. The endpoint list below uses the client listener on port 2379; it is not the peer listener on port 2380.
export ETCDCTL_API=3
export ENDPOINTS='https://etcd-1.example.net:2379,https://etcd-2.example.net:2379,https://etcd-3.example.net:2379'
export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/pki/healthcheck-client.crt
export ETCDCTL_KEY=/etc/etcd/pki/healthcheck-client.key
Use a purpose-specific client identity where possible. It needs enough permission to perform the checks and create a snapshot, not unrestricted administrative access for ordinary application operators. Protect the private key with the same care as any other production credential.
Before changing a member, establish the cluster’s current condition. endpoint status reports the member identity, version, database size, leader state, and raft index. endpoint health performs a write-and-read health check. The --write-out=table option makes the status result easier to save in a change record.
etcdctl --endpoints="$ENDPOINTS" endpoint status --cluster --write-out=table
etcdctl --endpoints="$ENDPOINTS" endpoint health --cluster
etcdctl --endpoints="$ENDPOINTS" member list --write-out=table
Do not proceed merely because one endpoint responds. A three-member cluster normally needs two members available to retain quorum. If a member is unhealthy, the leader is changing repeatedly, or the command returns an incomplete member list, investigate that state before introducing a planned restart. Record the output with the maintenance ticket, taking care not to store certificate paths or credentials where they do not belong.
Take and verify a snapshot
A valid snapshot is part of the maintenance plan, not proof that the service can be restored. Save it to a filesystem with capacity, then copy it to an access-controlled location separate from the member being serviced. The etcd recovery guide warns that copying only the member snap/db file can omit data that is still in the write-ahead log, so use etcdctl snapshot save for an online backup.
backup_dir=/var/backups/etcd
snapshot="$backup_dir/etcd-$(date +%F-%H%M%S).db"
install -d -m 0700 "$backup_dir"
etcdctl --endpoints="$ENDPOINTS" snapshot save "$snapshot"
etcdctl snapshot status "$snapshot" --write-out=table
sha256sum "$snapshot" > "$snapshot.sha256"
The status command should identify a nonzero revision and show the snapshot hash and size. A checksum does not prove that the backup can be restored into your exact environment, but it catches accidental corruption after it is copied. Test restoration periodically in an isolated environment using the same major and minor version family documented for the cluster. Never point a restore test at the production data directory or reuse its member names and peer addresses on the production network.
Patch one member at a time
A rolling patch keeps quorum only when the cluster begins healthy and enough members remain online. Drain or stop one member according to the service manager and package process used by the host. On a systemd host where the unit is named etcd, the local sequence might look like this:
sudo systemctl stop etcd
# Install the approved patch by the site's package, image, or release process.
sudo systemctl start etcd
sudo systemctl status --no-pager etcd
Do not alter --name, --initial-cluster, peer URLs, client URLs, or the existing data directory for a same-minor patch. Those settings identify a member to its peers; changing them turns a routine restart into a membership operation. If the service does not become active, inspect its journal and the etcd logs before attempting another member. A repeated restart loop can make a diagnosable problem harder to understand.
After the member returns, query the whole cluster again from an administrative host. Confirm that every expected endpoint is healthy and that the patched member reports the intended version. A status table with one leader is expected; leadership may move during the maintenance window, so the important result is a stable, healthy quorum rather than preserving a particular leader.
etcdctl --endpoints="$ENDPOINTS" endpoint health --cluster
etcdctl --endpoints="$ENDPOINTS" endpoint status --cluster --write-out=table
etcdctl --endpoints="$ENDPOINTS" member list --write-out=table
Only then repeat the stop, patch, start, and verification sequence for the next member. Patch followers before the current leader when the topology makes that easy, but do not depend on a leader identity remaining fixed. Between members, check that application and control-plane alerts have returned to their normal state.
Finish with evidence, not just a version string
When all members report the intended patch version, retain the pre-change and post-change status, member list, snapshot status, package or image identifiers, and the checksum file with the change record. Confirm that the snapshot has reached its protected backup destination and that its retention policy will not remove the only usable recovery point.
The release fixed an authorization boundary in watch delivery, while the maintenance procedure verifies a different set of operational boundaries: quorum, member identity, client TLS access, and recoverability. That combination makes the next patch less dependent on memory and gives an incident responder evidence to work with if the cluster later needs to be restored.