sean的农场
large picture of the cover

k8s高可用多节点master join集群时报错error execution phase etcd-join

警告信息："logger":"etcd-client","caller":"v3@v3.6.4/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x20a9c5c02b40/192.168.8.70:2379","method":"/etcdserverpb.Cluster/MemberList","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

报错信息：error: error execution phase etcd-join: error creating local etcd static pod manifest file: context deadline exceeded

context deadline exceeded 表示 master03 etcd 通信超时，根本无法与现有 etcd 集群建立 gRPC 连接。

排查：

第一步在master03测试网络连通性

# 测试到两个现有 etcd 成员的 2379 端口
nc -zv 192.168.8.70 2379
nc -zv 192.168.8.71 2379

# 如果 nc 不可用，用 telnet 或 curl
curl -k --connect-timeout 3 https://192.168.8.70:2379/version

第二步 在master01上面检查etcd集群的健康状态

# 检查 etcd 是否健康（master02 刚加入后可能需要时间稳定）
kubectl exec -n kube-system etcd-master01 -- etcdctl endpoint health \
  --endpoints=https://192.168.8.70:2379,https://192.168.8.71:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 输出
https://192.168.8.71:2379 is healthy: successfully committed proposal: took = 312.314852ms
https://192.168.8.70:2379 is healthy: successfully committed proposal: took = 313.036266ms

# 查看成员列表
kubectl exec -n kube-system etcd-master01 -- etcdctl member list -w table \
  --endpoints=https://192.168.8.70:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 输出
+------------------+---------+----------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |   NAME   |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------+---------------------------+---------------------------+------------+
|  1234d2479da2dd2 | started | master02 | https://192.168.8.71:2380 | https://192.168.8.71:2379 |      false |
| 77e99bccb868ac09 | started | master01 | https://192.168.8.70:2380 | https://192.168.8.70:2379 |      false |
| fb51bfef4f9f94b3 | started | master03 | https://192.168.8.72:2380 | https://192.168.8.72:2379 |       true |
+------------------+---------+----------+---------------------------+---------------------------+------------+

预期：两个成员都 healthy 且 state=started。如果有 unhealthy 成员 → 先修复 etcd 集群再尝试 join。

第三步：检查etcd的监听地址

# 在 master01 和 master02 上分别检查
grep -A2 'listen-client-urls' /etc/kubernetes/manifests/etcd.yaml

# master01输出
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.8.70:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://192.168.8.70:2380
# master02输出
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.8.71:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://192.168.8.71:2380

预期：应包含节点 IP，如 https://192.168.8.70:2379。
如果只有 https://127.0.0.1:2379 → 需要修改为包含节点 IP

在第二步发现问题etcd 集群中已经存在 master03 的成员记录（fb51bfef4f9f94b3），但状态是 learner 且 join 过程中断，导致它卡在 learner 状态无法被 promote。

这就是典型的 "脏成员"问题：上次 join 在 etcd-join 阶段超时失败，但 MemberAdd 操作已经成功写入了 etcd 集群，而后续的 MemberPromote 没来得及完成。

修复1：从 etcd 集群中移除残留的 learner 成员

在master01上执行

kubectl exec -n kube-system etcd-master01 -- etcdctl member remove fb51bfef4f9f94b3 \
  --endpoints=https://192.168.8.70:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

验证移除

kubectl exec -n kube-system etcd-master01 -- etcdctl member list -w table \
  --endpoints=https://192.168.8.70:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key


# 输出
+------------------+---------+----------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |   NAME   |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------+---------------------------+---------------------------+------------+
|  1234d2479da2dd2 | started | master02 | https://192.168.8.71:2380 | https://192.168.8.71:2379 |      false |
| 77e99bccb868ac09 | started | master01 | https://192.168.8.70:2380 | https://192.168.8.70:2379 |      false |
+------------------+---------+----------+---------------------------+---------------------------+------------+

最后一步修复问题后重新join

在master03上执行

# 1. 重置失败的 join 状态
kubeadm reset -f

# 2. 清理残留（重要！etcd-join 失败可能留下脏数据）
# 清理所有残留数据（关键！）
rm -rf /etc/kubernetes/pki /etc/kubernetes/*.conf /var/lib/etcd /var/lib/kubelet/config.yaml /var/lib/kubelet/kubeadm-flags.env

#重新join
kubeadm join 192.168.8.79:16443 --token abcdef.0123456789abcdef    --discovery-token-ca-cert-hash sha256:f10e517dded37b07cff1e9f8d32f7a2d2eccad4b86d70ffb4c98614fa48c58f1  --control-plane --certificate-key 59e845a854ffb89932e7b087cacd5b8500a2aee28d91b2cc8a584dbacfc635ba

节点状态为notReady

根因：kube-apiserver-master03 持续崩溃 → 节点无法完成 TLS Bootstrap → NotReady → 所有依赖网络的 Pod 卡住。

查看日志排查原因：

在master03 查看日志

# 1. 查看 apiserver 日志（最关键）
crictl logs $(crictl ps -a --name kube-apiserver  -q | head -1) --tail 80

# 如果 crictl 不好用，用 journalctl
journalctl -u kubelet --since "10 min ago" | grep -i "apiserver\|error\|fatal" | tail -50

错误日志：

9月 09 16:26:31 master03 kubelet[12723]: E0909 16:26:31.686733   12723 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c860acb7dbef8d8c2014e90b5af3b1a20a960a1d81af657b42fac90e63032def\": plugin type=\"calico\" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
9月 09 16:26:31 master03 kubelet[12723]: E0909 16:26:31.686770   12723 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c860acb7dbef8d8c2014e90b5af3b1a20a960a1d81af657b42fac90e63032def\": plugin type=\"calico\" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" pod="calico-system/csi-node-driver-mxp7d"
9月 09 16:26:31 master03 kubelet[12723]: E0909 16:26:31.686781   12723 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c860acb7dbef8d8c2014e90b5af3b1a20a960a1d81af657b42fac90e63032def\": plugin type=\"calico\" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" pod="calico-system/csi-node-driver-mxp7d"
9月 09 16:26:31 master03 kubelet[12723]: E0909 16:26:31.686817   12723 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"csi-node-driver-mxp7d_calico-system(0b50fb5e-d52a-4dce-8872-181490b9bc76)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"csi-node-driver-mxp7d_calico-system(0b50fb5e-d52a-4dce-8872-181490b9bc76)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"c860acb7dbef8d8c2014e90b5af3b1a20a960a1d81af657b42fac90e63032def\\\": plugin type=\\\"calico\\\" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\"" pod="calico-system/csi-node-driver-mxp7d" podUID="0b50fb5e-d52a-4dce-8872-181490b9bc76"

死锁链导致崩溃：

apiserver 崩溃 → 节点 NotReady 
    → calico-node Pod 无法完成初始化（Init:1/2）
        → /var/lib/calico/nodename 文件不存在
            → CNI 插件失败 → 所有新 Pod（包括 apiserver）无法创建网络沙箱
                → apiserver 继续崩溃 🔄

kubelet 日志显示的不是 apiserver 本身的错误，而是 CNI 网络插件在 apiserver 启动前就拦截了它。因为 calico-node 没跑起来，/var/lib/calico/nodename 不存在，导致连 apiserver 的 Pod Sandbox 都创建不了。

🛠️ 打破死锁：手动创建缺失文件 + 重启

在 master03 上依次执行：

# Step 1: 手动创建 calico 所需的 nodename 文件
mkdir -p /var/lib/calico
echo "master03" > /var/lib/calico/nodename

# Step 2: 确认文件已创建
cat /var/lib/calico/nodename

# Step 3: 重启 kubelet，让 apiserver 重新尝试启动
systemctl restart kubelet

# Step 4: 等待 30 秒后检查状态
sleep 30
kubectl get nodes
kubectl get po -n kube-system | grep master03
kubectl get po -n calico-system | grep pw4xb

💡 为什么这样做是安全的？

  • /var/lib/calico/nodename 只是 calico-node 用来标识自身节点名的文件，内容就是主机名

  • calico-node 正常运行后会自动覆盖/维护这个文件

  • 这只是一个临时引导手段，目的是让 apiserver 先跑起来，之后 calico-node 才能完成初始化，形成正向循环

项目

说明

根因

master03 join 后 apiserver 启动慢 → calico-node 未完成初始化 → /var/lib/calico/nodename 缺失 → CNI 阻断所有 Pod 创建 → apiserver 也无法启动 → 死锁

修复

手动创建 /var/lib/calico/nodename 打破循环

预防

未来新增 control-plane 节点时，可提前执行 mkdir -p /var/lib/calico && echo "<节点名>" > /var/lib/calico/nodename 再执行 kubeadm join

site logo

© 2026 sean的农场