详解K8s 镜像缓存管理kube-fledged

本文分享自华为云社区《
K8s 镜像缓存管理 kube-fledged 认知
》，作者：山河已无恙。

我们知道
k8s
上的容器调度需要在调度的节点行拉取当前容器的镜像，在一些特殊场景中，

需要
快速启动和/或扩展
的应用程序。例如，由于数据量激增，执行实时数据处理的应用程序需要快速扩展。
镜像比较庞大，涉及多个版本，节点存储有限，需要动态清理不需要的镜像
无服务器函数
通常需要在几分之一秒内立即对传入事件和启动容器做出反应。
在边缘设备上运行的
IoT 应用程序
，需要容忍
边缘设备
和镜像镜像仓库之间的间歇性网络连接。
如果需要从
专用仓库
中拉取镜像，并且无法授予每个人从此
镜像仓库
拉取镜像的访问权限，则可以在群集的节点上提供镜像。
如果集群管理员或操作员需要对应用程序进行升级，并希望事先验证是否可以成功拉取新镜像。

kube-fledged
是一个
kubernetes operator
，用于直接在 Kubernetes 集群的
worker
节点上创建和管理容器镜像缓存。它允许用户定义镜像列表以及这些镜像应缓存到哪些工作节点上（即拉取）。因此，应用程序 Pod 几乎可以立即启动，因为不需要从镜像仓库中提取镜像。

kube-fledged
提供了 CRUD API 来管理镜像缓存的生命周期，并支持多个可配置的参数，可以根据自己的需要自定义功能。

Kubernetes 具有内置的
镜像垃圾回收机制
。节点中的 kubelet 会定期检查磁盘使用率是否达到特定阈值（可通过标志进行配置）。一旦达到这个
阈值
，kubelet 会自动删除节点中所有未使用的镜像。

需要在建议的解决方案中实现自动和定期刷新机制。如果镜像缓存中的镜像被 kubelet 的 gc 删除，下一个刷新周期会将已删除的镜像拉入镜像缓存中。这可确保镜像缓存是最新的。

设计流程

https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png

部署 kube-fledged

Helm 方式部署

──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$mkdir  kube-fledged

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$cd kube-fledged

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$export KUBEFLEDGED_NAMESPACE=kube-fledged

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$kubectl createnamespace${KUBEFLEDGED_NAMESPACE}namespace/kube-fledged created

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
"kubefledged-charts"has been added to your repositories

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$helm repo update

Hang tightwhile we grab the latest fromyour chart repositories...

...Successfully got an updatefrom the "kubefledged-charts"chart repository

...Successfully got an updatefrom the "kubescape"chart repository

...Successfully got an updatefrom the "rancher-stable"chart repository

...Successfully got an updatefrom the "skm"chart repository

...Successfully got an updatefrom the "openkruise"chart repository

...Successfully got an updatefrom the "awx-operator"chart repository

...Successfully got an updatefrom the "botkube"chart repository

Update Complete. ⎈Happy Helming!⎈

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$helm install--verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

实际部署中发现，由于网络问题，
chart
无法下载，所以通过
make deploy-using-yaml
使用 yaml 方式部署

Yaml 文件部署

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$git clone https://github.com/senthilrch/kube-fledged.git
正克隆到 'kube-fledged'...

remote: Enumerating objects:10613, done.

remote: Counting objects:100% (1501/1501), done.

remote: Compressing objects:100% (629/629), done.

remote: Total10613 (delta 845), reused 1357 (delta 766), pack-reused 9112接收对象中:100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done.

处理 delta 中:100% (4431/4431), done.

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$ls

kube-fledged

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$cd kube-fledged/┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$make deploy-using-yaml

kubectl apply-f deploy/kubefledged-namespace.yaml

第一次部署，发现镜像拉不下来

┌──[root@vms100.liruilongs.github.io]-[~]

└─$kubectlget all -n kube-fledged

NAME                                               READY   STATUS                  RESTARTS         AGE

pod/kube-fledged-controller-df69f6565-drrqg        0/1     CrashLoopBackOff        35(5h59m ago)   21h

pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2   0/1     Init:CrashLoopBackOff   35(5h58m ago)   21h

pod/kubefledged-controller-55f848cc67-7f4rl        1/1     Running                 021h

pod/kubefledged-webhook-server-597dbf4ff5-l8fbh    0/1     Init:CrashLoopBackOff   34(6h ago)      21h



NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   21h

service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   21h



NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/kube-fledged-controller       0/1     1            021h

deployment.apps/kube-fledged-webhook-server   0/1     1            021h

deployment.apps/kubefledged-controller        0/1     1            021h

deployment.apps/kubefledged-webhook-server    0/1     1            021h



NAME                                                     DESIRED   CURRENT   READY   AGE

replicaset.apps/kube-fledged-controller-df69f6565        1         1         021h

replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         021h

replicaset.apps/kubefledged-controller-55f848cc67        1         1         021h

replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         021h

┌──[root@vms100.liruilongs.github.io]-[~]

└─$

这里我们找一下要拉取的镜像

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$cat*.yaml |grep image:- image: senthilrch/kubefledged-controller:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0

单独拉取一些，当前使用
ansible
在所有工作节点批量操作

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible k8s_node-m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml

其他相关的镜像都拉取一下

操作完成之后容器状态全部正常

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectl-n kube-fledged getall

NAME                                               READY   STATUS    RESTARTS   AGE

pod/kube-fledged-controller-df69f6565-wdb4g        1/1     Running   013h

pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp   1/1     Running   013h

pod/kubefledged-controller-55f848cc67-klxlm        1/1     Running   013h

pod/kubefledged-webhook-server-597dbf4ff5-ktbsh    1/1     Running   013h



NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   36h

service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   36h



NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/kube-fledged-controller       1/1     1            136h

deployment.apps/kube-fledged-webhook-server   1/1     1            136h

deployment.apps/kubefledged-controller        1/1     1            136h

deployment.apps/kubefledged-webhook-server    1/1     1            136h



NAME                                                     DESIRED   CURRENT   READY   AGE

replicaset.apps/kube-fledged-controller-df69f6565        1         1         136h

replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         136h

replicaset.apps/kubefledged-controller-55f848cc67        1         1         136h

replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         1       36h

验证是否安装成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$kubectlget pods -n kube-fledged -l app=kubefledged

NAME                                          READY   STATUS    RESTARTS   AGE

kubefledged-controller-55f848cc67-klxlm       1/1     Running   016h

kubefledged-webhook-server-597dbf4ff5-ktbsh   1/1     Running   016h

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$kubectlget imagecaches -n kube-fledged

No resources foundin kube-fledged namespace.

使用 kubefledged

创建镜像缓存对象

根据
Demo
文件，创建镜像缓存对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$cd deploy/┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2

kind: ImageCache

metadata:

  # Name of the image cache. A cluster can have multiple image cache objects

  name: imagecache1namespace: kube-fledged

  # The kubernetesnamespace to be used for this image cache. You can choose a different namepace asper your preference

  labels:

    app: kubefledged

    kubefledged: imagecache

spec:

  # The"cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).

  cacheSpec:

  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes inthe cluster-images:- ghcr.io/jitesoft/nginx:1.23.1# Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector-images:- us.gcr.io/k8s-artifacts-prod/cassandra:v7- us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0nodeSelector:

      tier: backend

  # Specifies a list of image pull secrets to pull imagesfrom privaterepositories into the cache

  imagePullSecrets:- name: myregistrykey

官方的 Demo 中对应的镜像拉取不下来，所以换一下

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7

Error responsefrom daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded whileawaiting headers)

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$

为了测试选择器标签的使用,我们找一个节点的标签单独做镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectlget nodes  --show-labels

同时我们直接从公有仓库拉取镜像，所以不需要
imagePullSecrets
对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$vim kubefledged-imagecache.yaml

修改后的
yaml
文件

添加了一个所有节点的 liruilong/my-busybox:latest 镜像缓存
添加了一个
kubernetes.io/hostname: vms105.liruilongs.github.io
对应标签选择器的
liruilong/hikvision-sdk-config-ftp:latest
镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2

kind: ImageCache

metadata:

  # Name of the image cache. A cluster can have multiple image cache objects

  name: imagecache1namespace: kube-fledged

  # The kubernetesnamespace to be used for this image cache. You can choose a different namepace asper your preference

  labels:

    app: kubefledged

    kubefledged: imagecache

spec:

  # The"cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).

  cacheSpec:

  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes inthe cluster-images:- liruilong/my-busybox:latest

  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector-images:- liruilong/hikvision-sdk-config-ftp:latest

    nodeSelector:

      kubernetes.io/hostname: vms105.liruilongs.github.io

  # Specifies a list of image pull secrets to pull imagesfrom privaterepositories into the cache

  #imagePullSecrets:

  #-name: myregistrykey

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$

直接创建报错了

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectl create-f kubefledged-imagecache.yaml

Errorfrom server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io")

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectlget imagecaches -n kube-fledged

No resources foundin kube-fledged namespace.

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$

解决办法，删除对应的对象，重新创建

我在当前项目的一个
issues
下面找到了解决办法
https://github.com/senthilrch/kube-fledged/issues/76

看起来这是因为
Webhook CA
是硬编码的，但是当
webhook
服务器启动时，会生成一个新的 CA 捆绑包并更新 webhook 配置。当发生另一个部署时，将重新应用原始 CA 捆绑包，并且 Webhook 请求开始失败，直到再次重新启动 Webhook 组件以修补捆绑包init-server

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$make remove-kubefledged-and-operator# Remove kubefledged

kubectl delete-f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml

error: resource mapping not foundfor name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2"ensure CRDs are installed first

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]

└─$make deploy-using-yaml

kubectl apply-f deploy/kubefledged-namespace.yamlnamespace/kube-fledged created

kubectl apply-f deploy/kubefledged-crd.yaml

customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged

....................

kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch

Waitingfor deployment "kubefledged-webhook-server" rollout to finish: 0 of 1updated replicas are available...

deployment"kubefledged-webhook-server" successfully rolled outkubectlget pods -n kube-fledged

NAME                                          READY   STATUS    RESTARTS   AGE

kubefledged-controller-55f848cc67-76c4v       1/1     Running   0112s

kubefledged-webhook-server-597dbf4ff5-56h6z   1/1     Running   0          66s

重新创建缓存对象，创建成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectl create-f kubefledged-imagecache.yaml

imagecache.kubefledged.io/imagecache1 created

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectlget imagecaches -n kube-fledged

NAME          AGE

imagecache1   10s

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$

查看当前被纳管的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$kubectlget imagecaches imagecache1 -n kube-fledged -o json

{"apiVersion": "kubefledged.io/v1alpha2","kind": "ImageCache","metadata": {"creationTimestamp": "2024-03-01T15:08:42Z","generation": 83,"labels": {"app": "kubefledged","kubefledged": "imagecache"},"name": "imagecache1","namespace": "kube-fledged","resourceVersion": "20169836","uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"},"spec": {"cacheSpec": [

            {"images": ["liruilong/my-busybox:latest"]

            },

            {"images": ["liruilong/hikvision-sdk-config-ftp:latest"],"nodeSelector": {"kubernetes.io/hostname": "vms105.liruilongs.github.io"}

            }

        ]

    },"status": {"completionTime": "2024-03-02T01:06:47Z","message": "All requested images pulled succesfully to respective nodes","reason": "ImageCacheRefresh","startTime": "2024-03-02T01:05:33Z","status": "Succeeded"}

}

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]

└─$

通过 ansible 来验证

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.102 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.101 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.103 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.105 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.100 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.106 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml192.168.26.102 | FAILED | rc=1 >>non-zero returncode192.168.26.100 | FAILED | rc=1 >>non-zero returncode192.168.26.103 | FAILED | rc=1 >>non-zero returncode192.168.26.105 | CHANGED | rc=0 >>liruilong/hikvision-sdk-config-ftp                                          latest            a02cd03b4342   4months ago    830MB192.168.26.101 | FAILED | rc=1 >>non-zero returncode192.168.26.106 | FAILED | rc=1 >>non-zero returncode

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

开启自动刷新

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectl annotate imagecaches imagecache1-n kube-fledged kubefledged.io/refresh-imagecache=imagecache.kubefledged.io/imagecache1 annotated

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

添加镜像缓存

添加一个新的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectlget imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json

{"apiVersion": "kubefledged.io/v1alpha2","kind": "ImageCache","metadata": {"creationTimestamp": "2024-03-01T15:08:42Z","generation": 92,"labels": {"app": "kubefledged","kubefledged": "imagecache"},"name": "imagecache1","namespace": "kube-fledged","resourceVersion": "20175233","uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"},"spec": {"cacheSpec": [

            {"images": ["liruilong/my-busybox:latest","liruilong/jdk1.8_191:latest"]

            },

            {"images": ["liruilong/hikvision-sdk-config-ftp:latest"],"nodeSelector": {"kubernetes.io/hostname": "vms105.liruilongs.github.io"}

            }

        ]

    },"status": {"completionTime": "2024-03-02T01:43:32Z","message": "All requested images pulled succesfully to respective nodes","reason": "ImageCacheUpdate","startTime": "2024-03-02T01:40:34Z","status": "Succeeded"}

}

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

通过 ansible 确认

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.101 | FAILED | rc=1 >>non-zero returncode192.168.26.100 | FAILED | rc=1 >>non-zero returncode192.168.26.102 | FAILED | rc=1 >>non-zero returncode192.168.26.103 | FAILED | rc=1 >>non-zero returncode192.168.26.105 | FAILED | rc=1 >>non-zero returncode192.168.26.106 | FAILED | rc=1 >>non-zero returncode

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.101 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5years ago     170MB192.168.26.102 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5years ago     170MB192.168.26.100 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5years ago     170MB192.168.26.103 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest                                      17dbd4002a8c   5years ago     170MB192.168.26.105 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5years ago     170MB192.168.26.106 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5years ago     170MB

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

删除镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectl edit imagecaches imagecache1-n kube-fledged

imagecache.kubefledged.io/imagecache1 edited

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectlget imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json

{"apiVersion": "kubefledged.io/v1alpha2","kind": "ImageCache","metadata": {"creationTimestamp": "2024-03-01T15:08:42Z","generation": 94,"labels": {"app": "kubefledged","kubefledged": "imagecache"},"name": "imagecache1","namespace": "kube-fledged","resourceVersion": "20175766","uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"},"spec": {"cacheSpec": [

            {"images": ["liruilong/jdk1.8_191:latest"]

            },

            {"images": ["liruilong/hikvision-sdk-config-ftp:latest"],"nodeSelector": {"kubernetes.io/hostname": "vms105.liruilongs.github.io"}

            }

        ]

    },"status": {"message": "Image cache is being updated. Please view the status after some time","reason": "ImageCacheUpdate","startTime": "2024-03-02T01:48:03Z","status": "Processing"}

}

通过 Ansible 确认，可以看到无论是 mastere 上的节点还是 work 的节点，对应的镜像缓存都被清理

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.102 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.101 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.105 | FAILED | rc=1 >>non-zero returncode192.168.26.100 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.103 | FAILED | rc=1 >>non-zero returncode192.168.26.106 | FAILED | rc=1 >>non-zero returncode

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.105 | FAILED | rc=1 >>non-zero returncode192.168.26.102 | FAILED | rc=1 >>non-zero returncode192.168.26.103 | FAILED | rc=1 >>non-zero returncode192.168.26.101 | FAILED | rc=1 >>non-zero returncode192.168.26.100 | FAILED | rc=1 >>non-zero returncode192.168.26.106 | FAILED | rc=1 >>non-zero returncode

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

这里需要注意如果清除所有的镜像缓存，那么需要把
images
下的数组写成 "".

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectl edit imagecaches imagecache1-n kube-fledged

imagecache.kubefledged.io/imagecache1 edited

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$ansible all-m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.102 | FAILED | rc=1 >>non-zero returncode192.168.26.101 | FAILED | rc=1 >>non-zero returncode192.168.26.100 | FAILED | rc=1 >>non-zero returncode192.168.26.105 | FAILED | rc=1 >>non-zero returncode192.168.26.103 | FAILED | rc=1 >>non-zero returncode192.168.26.106 | FAILED | rc=1 >>non-zero returncode

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$kubectlget imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json

{"apiVersion": "kubefledged.io/v1alpha2","kind": "ImageCache","metadata": {"creationTimestamp": "2024-03-01T15:08:42Z","generation": 98,"labels": {"app": "kubefledged","kubefledged": "imagecache"},"name": "imagecache1","namespace": "kube-fledged","resourceVersion": "20176849","uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"},"spec": {"cacheSpec": [

            {"images": [""]

            },

            {"images": ["liruilong/hikvision-sdk-config-ftp:latest"],"nodeSelector": {"kubernetes.io/hostname": "vms105.liruilongs.github.io"}

            }

        ]

    },"status": {"completionTime": "2024-03-02T01:52:16Z","message": "All cached images succesfully deleted from respective nodes","reason": "ImageCacheUpdate","startTime": "2024-03-02T01:51:47Z","status": "Succeeded"}

}

┌──[root@vms100.liruilongs.github.io]-[~/ansible]

└─$

如果通过下面的方式删除，直接注释调对应的标签

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2

kind: ImageCache

metadata:

  # Name of the image cache. A cluster can have multiple image cache objects

  name: imagecache1namespace: kube-fledged

  # The kubernetesnamespace to be used for this image cache. You can choose a different namepace asper your preference

  labels:

    app: kubefledged

    kubefledged: imagecache

spec:

  # The"cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).

  cacheSpec:

  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes inthe cluster

  #-images:

    #- liruilong/my-busybox:latest

  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector-images:- liruilong/hikvision-sdk-config-ftp:latest

    nodeSelector:

      kubernetes.io/hostname: vms105.liruilongs.github.io

  # Specifies a list of image pull secrets to pull imagesfrom privaterepositories into the cache

  #imagePullSecrets:

  #-name: myregistrykey

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$

那么会报下面的错

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]

└─$kubectl edit imagecaches imagecache1-n kube-fledged

error: imagecaches.kubefledged.io"imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch inno. of image lists

You can run `kubectl replace-f /tmp/kubectl-edit-4113815075.yaml` to try this update again.

博文部分内容参考

https://github.com/senthilrch/kube-fledged

点击关注，第一时间了解华为云新鲜技术~