K3S Agent 节点安装后，k3s-agent.service状态不是 active (running) 。

Frank · 2023 年1 月 4 日 14:36

环境信息:
K3s 版本: v1.24.4+k3s1

节点 CPU 架构、操作系统和版本： Linux k8s-w158 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

集群配置:

问题描述:

安装K3S AGENT节点后，k3s-agent的服务状态时钟为： Active: activating (start)
复现步骤:

附加上下文/日志:

日志

集群node正常：
# kubectl get nodes
NAME       STATUS   ROLES                       AGE     VERSION
k8s-m156   Ready    control-plane,etcd,master   5h24m   v1.24.4+k3s1
k8s-w158   Ready    <none>                      5h22m   v1.24.4+k3s1

kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-579bc8967b-zzsd8                  1/1     Running     0          5h24m
kube-system   helm-install-traefik-2wl5p                0/1     Completed   1          5h24m
kube-system   helm-install-traefik-crd-9pwrc            0/1     Completed   0          5h24m
kube-system   local-path-provisioner-7946bbdc94-p75dw   1/1     Running     0          5h24m
kube-system   metrics-server-8554948558-cvrsk           1/1     Running     0          5h24m
kube-system   svclb-traefik-658726b5-qh7kj              2/2     Running     0          5h21m
kube-system   svclb-traefik-658726b5-v9tht              2/2     Running     0          5h21m
kube-system   traefik-6cb899fccf-4f2j6                  1/1     Running     0          5h21m

但k3s-agent服务的状态如下：：
# systemctl status k3s-agent -l
● k3s-agent.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: disabled)
   Active: activating (start) since Wed 2023-01-04 17:12:06 CST; 5h 20min ago
     Docs: https://k3s.io
 Main PID: 19502 (k3s-agent)
   CGroup: /system.slice/k3s-agent.service
           ├─19502 /usr/local/bin/k3s agent
           ├─19516 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
           ├─19954 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id bd73a41f2c9f5fda1d085c96edeaeef9f4be300279da5060f85897f6cf957568 -address /run/k3s/containerd/containerd.sock
           └─19966 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835 -address /run/k3s/containerd/containerd.sock

Jan 04 17:13:07 k8s-w158 k3s[19502]: I0104 17:13:07.264796   19502 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95de0a83-a90f-41f0-88df-7e227cd56069-tmp\") pod \"traefik-6cb899fccf-4f2j6\" (UID: \"95de0a83-a90f-41f0-88df-7e227cd56069\") " pod="kube-system/traefik-6cb899fccf-4f2j6"
Jan 04 17:13:07 k8s-w158 k3s[19502]: I0104 17:13:07.264871   19502 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/95de0a83-a90f-41f0-88df-7e227cd56069-kube-api-access-c8cx8\") pod \"traefik-6cb899fccf-4f2j6\" (UID: \"95de0a83-a90f-41f0-88df-7e227cd56069\") " pod="kube-system/traefik-6cb899fccf-4f2j6"
Jan 04 17:13:10 k8s-w158 k3s[19502]: E0104 17:13:10.796497   19502 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29\": not found" containerID="7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29"
Jan 04 17:13:10 k8s-w158 k3s[19502]: I0104 17:13:10.796554   19502 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29" err="rpc error: code = NotFound desc = an error occurred when try to find container \"7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29\": not found"
Jan 04 17:13:10 k8s-w158 k3s[19502]: E0104 17:13:10.797228   19502 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5\": not found" containerID="eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5"
Jan 04 17:13:10 k8s-w158 k3s[19502]: I0104 17:13:10.797252   19502 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5" err="rpc error: code = NotFound desc = an error occurred when try to find container \"eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5\": not found"
Jan 04 17:13:10 k8s-w158 k3s[19502]: E0104 17:13:10.797555   19502 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443\": not found" containerID="b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443"
Jan 04 17:13:10 k8s-w158 k3s[19502]: I0104 17:13:10.797574   19502 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443" err="rpc error: code = NotFound desc = an error occurred when try to find container \"b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443\": not found"
Jan 04 17:13:10 k8s-w158 k3s[19502]: E0104 17:13:10.797802   19502 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1\": not found" containerID="c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1"
Jan 04 17:13:10 k8s-w158 k3s[19502]: I0104 17:13:10.797820   19502 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1" err="rpc error: code = NotFound desc = an error occurred when try to find container \"c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1\": not found"

在此处键入或粘贴代码

ksd · 2023 年1 月 5 日 01:38

你使用的是什么操作系统？
如果可以的话，你可以试试 1.25 是否可以正常安装
你可以把 /var/lib/rancher/k3s/agent/containerd/containerd.log 的日志文件贴出来

Frank · 2023 年1 月 5 日 07:44

感谢回复，以下是操作系统版本和日志。另外：其他版本我回头测试下。

操作系统版本：
cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
containerd.log

time="2023-01-04T17:12:09+08:00" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2023-01-04T17:12:09.759341924+08:00" level=info msg="starting containerd" revision= version=v1.6.6-k3s1
time="2023-01-04T17:12:09.771727827+08:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2023-01-04T17:12:09.771944910+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.773895668+08:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found.\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774120207+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774486605+08:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774533031+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774563448+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-01-04T17:12:09.774594691+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774756028+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.774959398+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.775018815+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.777664617+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.777805908+08:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-01-04T17:12:09.777837393+08:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-01-04T17:12:09.777907830+08:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-01-04T17:12:09.777926777+08:00" level=info msg="metadata content store policy set" policy=shared
time="2023-01-04T17:12:09.782555326+08:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2023-01-04T17:12:09.782626586+08:00" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2023-01-04T17:12:09.782665043+08:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2023-01-04T17:12:09.782808521+08:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782843177+08:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782863871+08:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782883231+08:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782904044+08:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782924475+08:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782938896+08:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782956052+08:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.782977087+08:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2023-01-04T17:12:09.783064953+08:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2023-01-04T17:12:09.783150099+08:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2023-01-04T17:12:09.783477312+08:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2023-01-04T17:12:09.783520342+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783535832+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2023-01-04T17:12:09.783629484+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783648711+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783685224+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783697850+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783710433+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783726527+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783739880+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783753972+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783767315+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2023-01-04T17:12:09.783951734+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783975168+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.783990849+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.784002397+08:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2023-01-04T17:12:09.784015508+08:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2023-01-04T17:12:09.784031184+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2023-01-04T17:12:09.784067692+08:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2023-01-04T17:12:09.784153131+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2023-01-04T17:12:09.784453360+08:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:false] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin NetworkPluginConfDir:/var/lib/rancher/k3s/agent/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:10010 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
time="2023-01-04T17:12:09.784524243+08:00" level=info msg="Connect containerd service"
time="2023-01-04T17:12:09.784582346+08:00" level=info msg="Get image filesystem path \"/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2023-01-04T17:12:09.785242346+08:00" level=info msg="Start subscribing containerd event"
time="2023-01-04T17:12:09.785326702+08:00" level=info msg="Start recovering state"
time="2023-01-04T17:12:09.785380758+08:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock.ttrpc
time="2023-01-04T17:12:09.785421418+08:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2023-01-04T17:12:09.785434398+08:00" level=info msg="Start event monitor"
time="2023-01-04T17:12:09.785442340+08:00" level=info msg="containerd successfully booted in 0.028441s"
time="2023-01-04T17:12:09.785461846+08:00" level=info msg="Start snapshots syncer"
time="2023-01-04T17:12:09.785484043+08:00" level=info msg="Start cni network conf syncer for default"
time="2023-01-04T17:12:09.785492507+08:00" level=info msg="Start streaming server"
time="2023-01-04T17:13:07.231922796+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:svclb-traefik-658726b5-v9tht,Uid:eadc3f16-f746-4842-8834-ea514ada2766,Namespace:kube-system,Attempt:0,}"
time="2023-01-04T17:13:07.453789551+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-6cb899fccf-4f2j6,Uid:95de0a83-a90f-41f0-88df-7e227cd56069,Namespace:kube-system,Attempt:0,}"
time="2023-01-04T17:13:08.687317114+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.688139974+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.691970298+08:00" level=info msg="ImageCreate event &ImageCreate{Name:sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.693034585+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.694031587+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.694973931+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.695746297+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:08.697788198+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"

map[string]interface {}{"cniVersion":"1.0.0", "forceAddress":true, "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"192.168.67.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xc0, 0xa8, 0x43, 0x0}, Mask:net.IPMask{0xff, 0xff, 0xff, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc0000bc900), "name":"cbr0", "type":"bridge"}
{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"192.168.67.0/24"}]],"routes":[{"dst":"192.168.67.0/24"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}
map[string]interface {}{"cniVersion":"1.0.0", "forceAddress":true, "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"192.168.67.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xc0, 0xa8, 0x43, 0x0}, Mask:net.IPMask{0xff, 0xff, 0xff, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc00012c900), "name":"cbr0", "type":"bridge"}
{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"192.168.67.0/24"}]],"routes":[{"dst":"192.168.67.0/24"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2023-01-04T17:13:08.800059586+08:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-01-04T17:13:08.800157609+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-01-04T17:13:08.800180494+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-01-04T17:13:08.800377814+08:00" level=info msg="starting signal loop" namespace=k8s.io path=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/bd73a41f2c9f5fda1d085c96edeaeef9f4be300279da5060f85897f6cf957568 pid=19954 runtime=io.containerd.runc.v2
time="2023-01-04T17:13:08.805159161+08:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-01-04T17:13:08.805235077+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-01-04T17:13:08.805245468+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-01-04T17:13:08.805567827+08:00" level=info msg="starting signal loop" namespace=k8s.io path=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835 pid=19966 runtime=io.containerd.runc.v2
time="2023-01-04T17:13:08.888720567+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-6cb899fccf-4f2j6,Uid:95de0a83-a90f-41f0-88df-7e227cd56069,Namespace:kube-system,Attempt:0,} returns sandbox id \"bd73a41f2c9f5fda1d085c96edeaeef9f4be300279da5060f85897f6cf957568\""
time="2023-01-04T17:13:08.891115052+08:00" level=info msg="PullImage \"registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik:2.6.2\""
time="2023-01-04T17:13:08.897367650+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:svclb-traefik-658726b5-v9tht,Uid:eadc3f16-f746-4842-8834-ea514ada2766,Namespace:kube-system,Attempt:0,} returns sandbox id \"247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835\""
time="2023-01-04T17:13:08.898288267+08:00" level=info msg="PullImage \"registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb:v0.3.5\""
time="2023-01-04T17:13:10.796251691+08:00" level=error msg="ContainerStatus for \"7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"7d11e746ff03a895ea7654b319fa3b4f292f792234a0fc75c817258721cbba29\": not found"
time="2023-01-04T17:13:10.796795317+08:00" level=error msg="ContainerStatus for \"eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"eedac7522a92be83cb2f64807ec78cfb7b9e4ff0fc5f3b6c9587b064985a13d5\": not found"
time="2023-01-04T17:13:10.797430575+08:00" level=error msg="ContainerStatus for \"b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"b6bb508f84bac53e0242330fc5259b2a42bc566d563ddfef8dadacb6d5f26443\": not found"
time="2023-01-04T17:13:10.797716324+08:00" level=error msg="ContainerStatus for \"c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"c5ccc57114e8ddec1534758e0ebcbf2d2777c2b28c6218a222c43b0bc58c21b1\": not found"
time="2023-01-04T17:13:22.145352683+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb:v0.3.5,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:22.148653109+08:00" level=info msg="ImageCreate event &ImageCreate{Name:sha256:dbd43b6716a084c929553d435c5f86f17b331739fe4652eb95e6bdc12df38a10,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:22.151319566+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb:v0.3.5,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:22.153544217+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb@sha256:23286ad7a6adb09adaf66a1d12955e98b47ce1c77aa30082c4d7a4392bc4c95f,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:22.153818770+08:00" level=info msg="PullImage \"registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb:v0.3.5\" returns image reference \"sha256:dbd43b6716a084c929553d435c5f86f17b331739fe4652eb95e6bdc12df38a10\""
time="2023-01-04T17:13:22.156018324+08:00" level=info msg="CreateContainer within sandbox \"247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835\" for container &ContainerMetadata{Name:lb-tcp-80,Attempt:0,}"
time="2023-01-04T17:13:22.189320834+08:00" level=info msg="CreateContainer within sandbox \"247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835\" for &ContainerMetadata{Name:lb-tcp-80,Attempt:0,} returns container id \"cbdf499a7a63c99c2087065a46507c4b92bf3d1005af7b7b1043b56daf23a6b7\""
time="2023-01-04T17:13:22.189871309+08:00" level=info msg="StartContainer for \"cbdf499a7a63c99c2087065a46507c4b92bf3d1005af7b7b1043b56daf23a6b7\""
time="2023-01-04T17:13:22.271241385+08:00" level=info msg="StartContainer for \"cbdf499a7a63c99c2087065a46507c4b92bf3d1005af7b7b1043b56daf23a6b7\" returns successfully"
time="2023-01-04T17:13:22.273631666+08:00" level=info msg="CreateContainer within sandbox \"247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835\" for container &ContainerMetadata{Name:lb-tcp-443,Attempt:0,}"
time="2023-01-04T17:13:22.300665337+08:00" level=info msg="CreateContainer within sandbox \"247fed5a7f30516d7d3632a629af37c59b170a5c719530593338be6ccdd21835\" for &ContainerMetadata{Name:lb-tcp-443,Attempt:0,} returns container id \"db801198a4ea2127d4d8aa87c4857f473899237644c339dfd4e01ad65deb5c76\""
time="2023-01-04T17:13:22.301050716+08:00" level=info msg="StartContainer for \"db801198a4ea2127d4d8aa87c4857f473899237644c339dfd4e01ad65deb5c76\""
time="2023-01-04T17:13:22.365293995+08:00" level=info msg="StartContainer for \"db801198a4ea2127d4d8aa87c4857f473899237644c339dfd4e01ad65deb5c76\" returns successfully"
time="2023-01-04T17:13:38.884212614+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik:2.6.2,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:38.887903031+08:00" level=info msg="ImageCreate event &ImageCreate{Name:sha256:72463d8000a351a929f93834ecbb65b63e1d5dd2990f9c4b8f7cd28a66acec44,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:38.893452935+08:00" level=info msg="ImageUpdate event &ImageUpdate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik:2.6.2,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:38.896469493+08:00" level=info msg="ImageCreate event &ImageCreate{Name:registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik@sha256:e2ea2ac5cf8bfb6a8e41d28915248b01e9df2a86671ec8ef2c41c38fd3285570,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2023-01-04T17:13:38.896761472+08:00" level=info msg="PullImage \"registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik:2.6.2\" returns image reference \"sha256:72463d8000a351a929f93834ecbb65b63e1d5dd2990f9c4b8f7cd28a66acec44\""
time="2023-01-04T17:13:38.898186935+08:00" level=info msg="CreateContainer within sandbox \"bd73a41f2c9f5fda1d085c96edeaeef9f4be300279da5060f85897f6cf957568\" for container &ContainerMetadata{Name:traefik,Attempt:0,}"
time="2023-01-04T17:13:38.918757934+08:00" level=info msg="CreateContainer within sandbox \"bd73a41f2c9f5fda1d085c96edeaeef9f4be300279da5060f85897f6cf957568\" for &ContainerMetadata{Name:traefik,Attempt:0,} returns container id \"a3b767f5243ed1c00ba1497f4baca4e8f67f181fd77f3079b0421e4323f768a3\""
time="2023-01-04T17:13:38.919321488+08:00" level=info msg="StartContainer for \"a3b767f5243ed1c00ba1497f4baca4e8f67f181fd77f3079b0421e4323f768a3\""
time="2023-01-04T17:13:38.992205134+08:00" level=info msg="StartContainer for \"a3b767f5243ed1c00ba1497f4baca4e8f67f181fd77f3079b0421e4323f768a3\" returns successfully"