您现在的位置是：首页 > 其他

当前栏目

k8s拉取镜像失败处理 ImagePullBackOff ErrImageNeverPull

k8s 状态容器镜像

2023-04-18 16:28:35 时间

一、环境描述

系统环境：CentOS Linux release 7.9.2009 (Core)
系统内核：Linux k8s-master01 5.4.153-1.el7.elrepo.x86_64 #1 SMP Tue Oct 12 08:16:11 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
k8s版本：Kubernetes v1.20.14
docker：Docker version 20.10.12
集群有外网, docker已经对接好网络镜像仓库。

二、pod失败状态

1.拉取k8s网上的镜像失败后的状态

[root@k8s-master01 ~]# kubectl get po -n monitoring 
NAME                                   READY   STATUS              RESTARTS    AGE
kube-state-metrics-5d497ddb45-lvtxv     2/3   ImagePullBackOff      2          16h

2.拉取本地的镜像失败后的状态

[root@k8s-master01 ~]# kubectl get po -n monitoring 
NAME                                   READY   STATUS              RESTARTS   AGE
prometheus-adapter-5c976b9ddb-chn6v    0/1     ErrImageNeverPull   0          171m
prometheus-adapter-5c976b9ddb-gbn8l    0/1     ErrImageNeverPull   0          171m
prometheus-adapter-7858d4ddfd-55lnq    0/1     ErrImageNeverPull   0          176m

3.整体状态

[root@k8s-master01 ~]# kubectl get po -n monitoring 
NAME                                   READY   STATUS              RESTARTS   AGE
alertmanager-main-0                    2/2     Running             2          20h
alertmanager-main-1                    2/2     Running             16         20h
alertmanager-main-2                    2/2     Running             3          20h
blackbox-exporter-69894767d5-vbwz8     3/3     Running             3          20h
grafana-564b8d866c-gmgvr               1/1     Running             1          20h
kube-state-metrics-5d497ddb45-lvtxv    2/3     ImagePullBackOff    2          16h
node-exporter-4qsvn                    2/2     Running             2          20h
node-exporter-9d8bw                    2/2     Running             16         20h
node-exporter-ffs7q                    2/2     Running             2          20h
node-exporter-qjdv9                    2/2     Running             4          20h
node-exporter-zkkrr                    2/2     Running             2          20h
prometheus-adapter-5c976b9ddb-chn6v    0/1     ErrImageNeverPull   0          171m
prometheus-adapter-5c976b9ddb-gbn8l    0/1     ErrImageNeverPull   0          171m
prometheus-adapter-7858d4ddfd-55lnq    0/1     ErrImageNeverPull   0          176m
prometheus-k8s-0                       2/2     Running             2          20h
prometheus-k8s-1                       2/2     Running             4          20h
prometheus-operator-545bcb5949-r6h6z   2/2     Running             3          20h

三、整体解决方案

第一步：使用如下命令查询pod详细报错内容，以下面报错为例拉取网络镜像失败，提示拉取失败 “ Failed to pull image "k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)”。

[root@k8s-master01 manifests]#  kubectl get po -n monitoring 
....
prometheus-adapter-6cf5d8bfcf-4jbhq    0/1     ImagePullBackOff   0          3m34s
prometheus-adapter-6cf5d8bfcf-bf4wt    0/1     ImagePullBackOff   0          3m34s
....

[root@k8s-master01 manifests]# kubectl  describe po prometheus-adapter-6cf5d8bfcf-4jbhq -n monitoring 

..........上面内容省略.................

Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       70s                default-scheduler  Successfully assigned monitoring/prometheus-adapter-6cf5d8bfcf-4jbhq to k8s-master02
  Normal   SandboxChanged  48s                kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling         30s (x2 over 64s)  kubelet            Pulling image "k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1"
  Warning  Failed          14s (x2 over 49s)  kubelet            Failed to pull image "k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed          14s (x2 over 49s)  kubelet            Error: ErrImagePull
  Normal   BackOff         2s (x4 over 45s)   kubelet            Back-off pulling image "k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1"
  Warning  Failed          2s (x4 over 45s)   kubelet            Error: ImagePullBackOff

第二步：由上述报错信息知道了pod需要获取 prometheus-adapter:v0.9.1 的镜像文件。那么我们就需要通过docker 获取 prometheus-adapter 有那些镜像。

[root@k8s-master01 manifests]# docker search prometheus-adapter
NAME                                                   DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
willdockerhub/prometheus-adapter                       sync from k8s.gcr.io/prometheus-adapter/prom…   0                    
selina5288/prometheus-adapter                          sync k8s.gcr.io/prometheus-adapter/prometheu…   0                    
v5cn/prometheus-adapter                                sync k8s.gcr.io/prometheus-adapter/prometheu…   0                    
lbbi/prometheus-adapter                                k8s.gcr.io                                      0                    
boy530/prometheus-adapter                                                                              0                    
forging2012/prometheus-adapter                                                                         0                    
antidebug/prometheus-adapter                                                                           0             

###下面的省略###

第三步：拉取合适的镜像到本地，在此我拉取的是 “ lbbi/prometheus-adapter ”。

[root@k8s-master01 manifests]# docker pull lbbi/prometheus-adapter:v0.9.1
v0.9.1: Pulling from lbbi/prometheus-adapter
Digest: sha256:d025d1a109234c28b4a97f5d35d759943124be8885a5bce22a91363025304e9d
Status: Image is up to date for lbbi/prometheus-adapter:v0.9.1
docker.io/lbbi/prometheus-adapter:v0.9.1

注意！

我这里需要在后面加上版本，不然会出现找不到 lbbi/prometheus-adapter 。

根据实际情况来，有的镜像需要在后面加版本有的不需要加版本。
[root@k8s-master01 manifests]# docker pull lbbi/prometheus-adapter
Using default tag: latest
Error response from daemon: manifest for lbbi/prometheus-adapter:latest not found: manifest unknown: manifest unknown

第四步：给下载下来的镜像打标签。（可以选择不打标签，但是第五步的yaml配置文件修改时要写对镜像标签，否则就会出现找不到本地镜像的报错，pod 状态为 “ ErrImageNeverPull ”）

[root@k8s-master01 manifests]# docker images    ## 此处为已经打好标签的镜像。
REPOSITORY                                                        TAG       IMAGE ID       CREATED         SIZE
bitnami/kube-state-metrics                                        latest    3b5c65a36550   13 hours ago    122MB
grafana/grafana                                                   8.4.3     8d0b77430ee9   11 days ago     278MB
quay.io/prometheus-operator/prometheus-config-reloader            v0.54.1   18cec637a88f   2 weeks ago     12.2MB
lbbi/kube-state-metrics                                           v2.4.1    74a618d7bae7   2 weeks ago     38.7MB
k8s.gcr.io/kube-state-metrics                                     v2.4.1    74a618d7bae7   2 weeks ago     38.7MB
quay.io/prometheus/node-exporter                                  v1.3.1    1dbe0e931976   3 months ago    20.9MB
lbbi/prometheus-adapter                                           v0.9.1    179df2737843   4 months ago    68.2MB
k8s.gcr.io/prometheus-adapter                                     v0.9.1    179df2737843   4 months ago    68.2MB
quay.io/prometheus/alertmanager                                   v0.23.0   ba2b418f427c   6 months ago    57.5MB
quay.io/brancz/kube-rbac-proxy                                    v0.11.0   29589495df8d   7 months ago    46.6MB

                                               ## 打标签命令
[root@k8s-master01 manifests]# docker tag docker.io/lbbi/kube-state-metrics:v2.4.1 k8s.gcr.io/kube-state-metrics:v2.4.1

[root@k8s-master01 manifests]# docker tag docker.io/lbbi/prometheus-adapter:v0.9.1 k8s.gcr.io/prometheus-adapter:v0.9.1

注意！

k8s集群内的所以设备都需要拉取一样的镜像，并且打上一样的标签，因为k8s在没有做亲和力的情况下，部署pod是在集群内随机部署的。若没有全部拉取镜像，也会出现找不到本地镜像的问题，pod状态是 “ErrImageNeverPull”。

第五步：为了方便k8s找到本地的镜像文件，需要修改pod对应的yaml配置文件。

***  找到需要镜像的yaml文件           ***
[root@k8s-master01 manifests]# vim prometheusAdapter-deployment.yaml

***  找到拉取镜像失败的部分  “修改前”  ***

        image: k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1
        name: prometheus-adapter

***              修改后               ***

      #  image: k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1
        image: k8s.gcr.io/prometheus-adapter:v0.9.1   #  容器镜像标签，写错拉取本地镜像失败。
        imagePullPolicy: Never                        #  imagePullPolicy: Always 总是网络拉取镜像, 是k8s默认的拉取方式。
                                                      #  imagePullPolicy: Never 从不远程拉取镜像，只读取本地镜像。
                                                      #  imagePullPolicy: IfNotPresent 优先拉取本地镜像。
        name: prometheus-adapter 
        ports:
        - containerPort: 6443

这是拉取本地镜像失败的状态，显示找不到本地镜像。

[root@k8s-master01 ~]# kubectl describe  po prometheus-adapter-7858d4ddfd-55lnq  -n monitoring

.......
............
..................
Events:
  Type     Reason                  Age                    From     Message
  ----     ------                  ----                   ----     -------
  Warning  Failed                  106m (x325 over 176m)  kubelet  Error: ErrImageNeverPull
  Warning  ErrImageNeverPull       101m (x348 over 176m)  kubelet  Container image "lbbi/prometheus-adapter" is not present with pull policy of Never
  Warning  FailedCreatePodSandBox  95m                    kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "743108347acf9031625da151edef7c323c894657048f1f6c14655baa64a0b9ce" network for pod "prometheus-adapter-7858d4ddfd-55lnq": networkPlugin cni failed to set up pod "prometheus-adapter-7858d4ddfd-55lnq_monitoring" network: failed to find plugin "calico-ipam" in path [/opt/cni/bin], failed to clean up sandbox container "743108347acf9031625da151edef7c323c894657048f1f6c14655baa64a0b9ce" network for pod "prometheus-adapter-7858d4ddfd-55lnq": networkPlugin cni failed to teardown pod "prometheus-adapter-7858d4ddfd-55lnq_monitoring" network: failed to find plugin "calico" in path [/opt/cni/bin]]
  Normal   SandboxChanged          95m (x2 over 95m)      kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  ErrImageNeverPull       75m (x94 over 95m)     kubelet  Container image "lbbi/prometheus-adapter" is not present with pull policy of Never
  Warning  Failed                  65m (x139 over 95m)    kubelet  Error: ErrImageNeverPull
  Normal   SandboxChanged          26m (x4 over 28m)      kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  ErrImageNeverPull       25m (x10 over 27m)     kubelet  Container image "lbbi/prometheus-adapter" is not present with pull policy of Never
  Warning  Failed                  3m2s (x114 over 27m)   kubelet  Error: ErrImageNeverPull

第五步：重新加载pod 的yaml 文件，并查询pod状态。

[root@k8s-master01 manifests]# kubectl replace -f prometheusAdapter-deployment.yaml
deployment.apps/prometheus-adapter replaced
[root@k8s-master01 manifests]# kubectl get po -n monitoring 
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   2          22h
alertmanager-main-1                    2/2     Running   16         22h
alertmanager-main-2                    2/2     Running   3          22h
blackbox-exporter-69894767d5-vbwz8     3/3     Running   3          22h
grafana-564b8d866c-gmgvr               1/1     Running   1          22h
kube-state-metrics-6db4889fd4-hcmfn    3/3     Running   0          97m
node-exporter-4qsvn                    2/2     Running   2          22h
node-exporter-9d8bw                    2/2     Running   16         22h
node-exporter-ffs7q                    2/2     Running   2          22h
node-exporter-qjdv9                    2/2     Running   4          22h
node-exporter-zkkrr                    2/2     Running   2          22h
prometheus-adapter-b8d4f7465-t7cxq     1/1     Running   0          108s
prometheus-adapter-b8d4f7465-tlf42     1/1     Running   0          101m
prometheus-k8s-0                       2/2     Running   2          22h
prometheus-k8s-1                       2/2     Running   4          22h
prometheus-operator-545bcb5949-r6h6z   2/2     Running   3          22h

四、补充一下Pod状态解释

CrashLoopBackOff 容器退出，kubelet正在将它重启
InvalidImageName 无法解析镜像名称
ImageInspectError 无法校验镜像
ErrImageNeverPul 策略禁止拉取镜像
ImagePullBackOff 正在重试拉取
RegistryUnavailable 连接不到镜像中心
ErrImagePull 通用的拉取镜像出错
CreateContainerConfigError 不能创建kubelet使用的容器配置
CreateContainerError 创建容器失败
m.internalLifecycle.PreStartContainer 执行hook报错
RunContainerError 启动容器失败
PostStartHookError 执行hook报错
ContainersNotInitialized 容器没有初始化完毕
ContainersNotReady 容器没有准备完毕
ContainerCreating 容器创建中
PodInitializing pod 初始化中
DockerDaemonNotReady docker还没有完全启动
NetworkPluginNotReady 网络插件还没有完全启动
Evicted 即驱赶的意思，意思是当节点出现异常时，kubernetes将有相应的机制驱赶该节点上的Pod。多见于资源不足时导致的驱赶。

猜你喜欢

5 个改善代码可读性的方法
【Win10 应用开发】OCR识别
Vuex真的不适合Vue3了吗？
【Win10应用开发】协议-下篇：自定义多个协议
CTF神器：如何使用HTTPUploadExfil快速实现文件数据的提取和传输
【Win10 应用开发】解决VS 2015 RC不能调试手机应用的问题
vue核心知识点
【Win 10开发】协议-上篇：自定义应用协议
黑色星期五在即，美英发出联合警告
【Win10 开发】读取PDF文档
【Win 10应用开发】SplitView控件
网络犯罪者顶级域名使用分析：.com和发展中国家中招最多
httprunner 2.x学习19 - 参数化显示不同的用例标题$title
【Win 10应用开发】认识一下UAP项目
httprunner 2.x学习18 - 报告validate显示LazyString($msg)问题解决
【WP开发】读写剪贴板
用户复购行为，该如何分析
【WP开发】不同客户端之间传输加密数据
【WP开发】加密篇：单向加密
Rails MVC 和 CRUD(8)

zl程序教程

当前栏目

k8s拉取镜像失败处理 ImagePullBackOff ErrImageNeverPull

一、环境描述

二、pod失败状态

三、整体解决方案

四、补充一下Pod状态解释

相关文章