RKE 版本:
Docker 版本: (docker version
,docker info
) Server Version: 19.03.8
操作系统和内核: (cat /etc/os-release
, uname -r
) :CentOS 7.9 5.15.13-1.el7.elrepo.x86_64
主机类型和供应商: (VirtualBox/Bare-metal/AWS/GCE/DO) :物理机
cluster.yml 文件:
重现步骤:
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run XXXX --server https://rancher.XXXX.com --token 6h56mmplz42rhxtksmvdkzpsvjld6h5kbjb45knvmlf572kn --worker
结果:
各位路过的大佬帮忙分许下原因,中间有个以下报错,但是节点能正常加入集群
time=“2024-09-11T08:46:33Z” level=error msg=“Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: Operation cannot be fulfilled on nodes.management.cattle.io "m-215caa662fd8": the object has been modified; please apply your changes to the latest version and try again” error=“websocket: bad handshake”
time=“2024-09-11T08:46:33Z” level=error msg=“Remotedialer proxy error” error=“websocket: bad handshake”
ksd
2024 年9 月 12 日 14:11
3
理论上,能加入集群,get pod 都是正常状态,就没问题
但是主节点扩容,依此从3节点到4节点,从4节点到5节点,etcd会至少重启2遍,才能加入集群:相关日志如下:
[INFO ] Initiating Kubernetes cluster
2:15:18 pm[INFO ] Successfully Deployed state file at [management-state/rke/rke-878430202/cluster.rkestate]
2:15:18 pm[INFO ] Building Kubernetes cluster
2:15:18 pm[INFO ] [dialer] Setup tunnel for host [10.2.50.2]
2:15:18 pm[INFO ] [dialer] Setup tunnel for host [10.2.40.2]
2:15:18 pm[INFO ] [dialer] Setup tunnel for host [10.2.48.2]
2:15:18 pm[INFO ] [dialer] Setup tunnel for host [10.2.41.2]
2:15:18 pm[ERROR] Failed to set up SSH tunneling for host [10.2.48.2]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": can not build dialer to [c-5klb8:m-237b71030f75]
2:15:18 pm[ERROR] Removing host [10.2.48.2] from node lists
2:15:18 pm[INFO ] [network] No hosts added existing cluster, skipping port check
2:15:18 pm[INFO ] [certificates] kube-apiserver certificate changed, force deploying certs
2:15:18 pm[INFO ] [certificates] Deploying kubernetes certificates to Cluster nodes
2:15:24 pm[INFO ] [reconcile] Rebuilding and updating local kube config
2:15:24 pm[INFO ] Successfully Deployed local admin kubeconfig at [management-state/rke/rke-878430202/kube_config_cluster.yml]
2:15:24 pm[INFO ] [reconcile] host [10.2.40.2] is a control plane node with reachable Kubernetes API endpoint in the cluster
2:15:24 pm[INFO ] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
2:15:24 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.40.2]
2:15:25 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:15:25 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:25 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:25 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:15:26 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:15:26 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.41.2]
2:15:26 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:15:26 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:26 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:26 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:15:27 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:15:27 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.50.2]
2:15:28 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:15:28 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:28 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:28 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:15:29 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:15:29 pm[INFO ] [/etc/kubernetes/kube-api-authn-webhook.yaml] Successfully deployed authentication webhook config Cluster nodes
2:15:29 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.50.2]
2:15:30 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:15:30 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:30 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:30 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:15:31 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:15:31 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.40.2]
2:15:31 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:15:31 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:31 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:31 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:15:32 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:15:32 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.41.2]
2:15:33 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:15:33 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:33 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:33 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:15:34 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:15:34 pm[INFO ] [/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes
2:15:34 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.40.2]
2:15:35 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:15:35 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:35 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:15:35 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:15:36 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:15:36 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.41.2]
2:15:37 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:15:37 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:37 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:15:37 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:15:38 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:15:38 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.50.2]
2:15:38 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:15:38 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:38 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:15:38 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:15:39 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:15:39 pm[INFO ] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
2:15:39 pm[INFO ] [reconcile] Reconciling cluster state
2:15:39 pm[INFO ] [reconcile] Check etcd hosts to be deleted
2:15:39 pm[INFO ] [reconcile] Check etcd hosts to be added
2:15:39 pm[INFO ] [reconcile] Rebuilding and updating local kube config
2:15:39 pm[INFO ] Successfully Deployed local admin kubeconfig at [management-state/rke/rke-878430202/kube_config_cluster.yml]
2:15:39 pm[INFO ] [reconcile] host [10.2.40.2] is a control plane node with reachable Kubernetes API endpoint in the cluster
2:15:45 pm[INFO ] [restart/kube-apiserver] Successfully restarted container on host [10.2.40.2]
2:15:45 pm[INFO ] [restart/kube-controller-manager] Successfully restarted container on host [10.2.40.2]
2:15:51 pm[INFO ] [restart/kube-apiserver] Successfully restarted container on host [10.2.41.2]
2:15:52 pm[INFO ] [restart/kube-controller-manager] Successfully restarted container on host [10.2.41.2]
2:15:57 pm[INFO ] [restart/kube-apiserver] Successfully restarted container on host [10.2.50.2]
2:15:58 pm[INFO ] [restart/kube-controller-manager] Successfully restarted container on host [10.2.50.2]
2:16:00 pm[INFO ] [restart/etcd] Successfully restarted container on host [k8s-stage-master02]
2:16:01 pm[INFO ] [restart/etcd] Successfully restarted container on host [k8s-stage-master03]
2:16:02 pm[INFO ] [restart/etcd] Successfully restarted container on host [k8s-stage-master01]
2:16:02 pm[INFO ] [reconcile] Reconciled cluster state successfully
2:16:08 pm[INFO ] [restart/kube-apiserver] Successfully restarted container on host [10.2.40.2]
2:16:08 pm[INFO ] Pre-pulling kubernetes images
2:16:08 pm[INFO ] Kubernetes images pulled successfully
2:16:08 pm[INFO ] [etcd] Building up etcd plane..
2:16:08 pm[INFO ] Successfully started [etcd-fix-perm] container on host [10.2.40.2]
2:16:08 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.40.2]
2:16:08 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.40.2]
2:16:08 pm[INFO ] Container [etcd-fix-perm] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:16:09 pm[INFO ] [remove/etcd-fix-perm] Successfully removed container on host [10.2.40.2]
2:16:10 pm[INFO ] [etcd] Successfully started [rke-log-linker] container on host [10.2.40.2]
2:16:10 pm[INFO ] [remove/rke-log-linker] Successfully removed container on host [10.2.40.2]
2:16:11 pm[INFO ] Successfully started [etcd-fix-perm] container on host [10.2.41.2]
2:16:11 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.41.2]
2:16:11 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.41.2]
2:16:11 pm[INFO ] Container [etcd-fix-perm] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:16:12 pm[INFO ] [remove/etcd-fix-perm] Successfully removed container on host [10.2.41.2]
2:16:13 pm[INFO ] [etcd] Successfully started [rke-log-linker] container on host [10.2.41.2]
2:16:13 pm[INFO ] [remove/rke-log-linker] Successfully removed container on host [10.2.41.2]
2:16:14 pm[INFO ] Successfully started [etcd-fix-perm] container on host [10.2.50.2]
2:16:14 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.50.2]
2:16:14 pm[INFO ] Waiting for [etcd-fix-perm] container to exit on host [10.2.50.2]
2:16:14 pm[INFO ] Container [etcd-fix-perm] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:16:15 pm[INFO ] [remove/etcd-fix-perm] Successfully removed container on host [10.2.50.2]
2:16:15 pm[INFO ] [etcd] Successfully started [rke-log-linker] container on host [10.2.50.2]
2:16:16 pm[INFO ] [remove/rke-log-linker] Successfully removed container on host [10.2.50.2]
2:16:16 pm[INFO ] [etcd] Successfully started etcd plane.. Checking etcd cluster health
2:16:52 pm[INFO ] Initiating Kubernetes cluster
2:16:52 pm[INFO ] Successfully Deployed state file at [management-state/rke/rke-1481533941/cluster.rkestate]
2:16:52 pm[INFO ] Building Kubernetes cluster
2:16:52 pm[INFO ] [dialer] Setup tunnel for host [10.2.40.2]
2:16:52 pm[INFO ] [dialer] Setup tunnel for host [10.2.48.2]
2:16:52 pm[INFO ] [dialer] Setup tunnel for host [10.2.41.2]
2:16:52 pm[INFO ] [dialer] Setup tunnel for host [10.2.50.2]
2:16:52 pm[INFO ] [network] Deploying port listener containers
2:17:26 pm[INFO ] [network] Successfully started [rke-etcd-port-listener] container on host [10.2.48.2]
2:17:32 pm[INFO ] [network] Successfully started [rke-cp-port-listener] container on host [10.2.48.2]
2:17:32 pm[INFO ] [network] Port listener containers deployed successfully
2:17:32 pm[INFO ] [network] Running etcd <-> etcd port checks
2:17:33 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.50.2]
2:17:33 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.41.2]
2:17:33 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.40.2]
2:17:37 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.48.2]
2:17:38 pm[INFO ] [network] Running control plane -> etcd port checks
2:17:38 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.50.2]
2:17:38 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.41.2]
2:17:38 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.40.2]
2:17:43 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.48.2]
2:17:43 pm[INFO ] [network] Running control plane -> worker port checks
2:17:44 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.50.2]
2:17:44 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.41.2]
2:17:44 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.40.2]
2:17:49 pm[INFO ] [network] Successfully started [rke-port-checker] container on host [10.2.48.2]
2:17:50 pm[INFO ] [network] Running workers -> control plane port checks
2:17:50 pm[INFO ] [network] Skipping kubeapi port check
2:17:50 pm[INFO ] [network] Removing port listener containers
2:17:50 pm[INFO ] [remove/rke-etcd-port-listener] Successfully removed container on host [10.2.50.2]
2:17:50 pm[INFO ] [remove/rke-etcd-port-listener] Successfully removed container on host [10.2.40.2]
2:17:50 pm[INFO ] [remove/rke-etcd-port-listener] Successfully removed container on host [10.2.41.2]
2:17:51 pm[INFO ] [remove/rke-etcd-port-listener] Successfully removed container on host [10.2.48.2]
2:17:51 pm[INFO ] [remove/rke-cp-port-listener] Successfully removed container on host [10.2.50.2]
2:17:51 pm[INFO ] [remove/rke-cp-port-listener] Successfully removed container on host [10.2.40.2]
2:17:51 pm[INFO ] [remove/rke-cp-port-listener] Successfully removed container on host [10.2.41.2]
2:17:51 pm[INFO ] [remove/rke-cp-port-listener] Successfully removed container on host [10.2.48.2]
2:17:51 pm[INFO ] [network] Port listener containers removed successfully
2:17:51 pm[INFO ] [certificates] kube-apiserver certificate changed, force deploying certs
[INFO ] [certificates] Deploying kubernetes certificates to Cluster nodes
2:18:03 pm[INFO ] [reconcile] Rebuilding and updating local kube config
2:18:03 pm[INFO ] Successfully Deployed local admin kubeconfig at [management-state/rke/rke-1481533941/kube_config_cluster.yml]
2:18:03 pm[INFO ] [reconcile] host [10.2.40.2] is a control plane node with reachable Kubernetes API endpoint in the cluster
2:18:03 pm[INFO ] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
2:18:03 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.41.2]
2:18:03 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:18:03 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:03 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:03 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:18:05 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:18:05 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.50.2]
2:18:05 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:18:05 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:05 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:05 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:18:06 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:18:06 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.40.2]
2:18:07 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:18:07 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:07 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:07 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:18:08 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:18:08 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/kube-api-authn-webhook.yaml] to node [10.2.48.2]
2:18:14 pm[INFO ] Successfully started [file-deployer] container on host [10.2.48.2]
2:18:14 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:14 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:14 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.48.2]
2:18:14 pm[INFO ] [/etc/kubernetes/kube-api-authn-webhook.yaml] Successfully deployed authentication webhook config Cluster nodes
2:18:14 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.48.2]
2:18:20 pm[INFO ] Successfully started [file-deployer] container on host [10.2.48.2]
2:18:20 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:20 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:20 pm[INFO ] Container [file-deployer] is still running on host [10.2.48.2]: stderr: [], stdout: []
2:18:21 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.48.2]
2:18:21 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.41.2]
2:18:22 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:18:22 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:22 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:22 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:18:23 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:18:23 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.50.2]
2:18:23 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:18:23 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:23 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:23 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:18:24 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:18:24 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [10.2.40.2]
2:18:25 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:18:25 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:25 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:25 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:18:26 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:18:26 pm[INFO ] [/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes
2:18:26 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.40.2]
2:18:27 pm[INFO ] Successfully started [file-deployer] container on host [10.2.40.2]
2:18:27 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:27 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.40.2]
2:18:27 pm[INFO ] Container [file-deployer] is still running on host [10.2.40.2]: stderr: [], stdout: []
2:18:28 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.40.2]
2:18:28 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.48.2]
2:18:33 pm[INFO ] Successfully started [file-deployer] container on host [10.2.48.2]
2:18:33 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:33 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.48.2]
2:18:34 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.48.2]
2:18:34 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.41.2]
2:18:35 pm[INFO ] Successfully started [file-deployer] container on host [10.2.41.2]
2:18:35 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:35 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.41.2]
2:18:35 pm[INFO ] Container [file-deployer] is still running on host [10.2.41.2]: stderr: [], stdout: []
2:18:36 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.41.2]
2:18:36 pm[INFO ] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.2.50.2]
2:18:37 pm[INFO ] Successfully started [file-deployer] container on host [10.2.50.2]
2:18:37 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:37 pm[INFO ] Waiting for [file-deployer] container to exit on host [10.2.50.2]
2:18:37 pm[INFO ] Container [file-deployer] is still running on host [10.2.50.2]: stderr: [], stdout: []
2:18:38 pm[INFO ] [remove/file-deployer] Successfully removed container on host [10.2.50.2]
2:18:38 pm[INFO ] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
2:18:38 pm[INFO ] [reconcile] Reconciling cluster state
2:18:38 pm[INFO ] [reconcile] Check etcd hosts to be deleted
2:18:38 pm[INFO ] [reconcile] Check etcd hosts to be added
2:18:38 pm[INFO ] [add/etcd] Adding member [etcd-k8s-stage-worker07] to etcd cluster
2:18:38 pm