docker 服务常见报错问题

问题1: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
docker pull 时报错，提示受限: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
[root@localhost ~]# docker pull docker.io/eclipse-mosquitto
Using default tag: latest
Trying to pull repository docker.io/library/eclipse-mosquitto ... 
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
[root@localhost ~]# 

原因：
DockerHub 从 2020 年 11 月 2 日，正式开始限制非付费用户的拉取频率:
匿名用户，每 6 小时只允许 pull 100 次
已登录用户，每 6 小时只允许 pull 200 次

解决办法：更换镜像配置、绕过限制
Azure:  https://dockerhub.azk8s.cn
中科大: https://ustc-edu-cn.mirror.aliyuncs.com
网易:   https://hub-mirror.c.163.com

vi /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://ustc-edu-cn.mirror.aliyuncs.com"]
}

重启服务
systemctl daemon-reload
systemctl restart docker

重新执行 docker pull 成功了：
[root@localhost ~]# docker pull docker.io/eclipse-mosquitto
Using default tag: latest
Trying to pull repository docker.io/library/eclipse-mosquitto ... 
latest: Pulling from docker.io/library/eclipse-mosquitto
97518928ae5f: Pull complete 
b491d2bad818: Pull complete 
1c67a7209b6c: Pull complete 
Digest: sha256:64b7c1729f7d1fff46b5e884fc389071686b2f7ed116c3fc7e17cbcb50fa147e
Status: Downloaded newer image for docker.io/eclipse-mosquitto:latest
[root@localhost ~]# 

问题2: 启动docker服务失败: container-storage-setup: ERROR: Can not setup storage driver overlay2 as system does not support it. Specify a different driver.
[root@localhost ~]# service docker start
Redirecting to /bin/systemctl start docker.service
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root@localhost ~]# 
[root@localhost ~]# service  docker status
Redirecting to /bin/systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2022-04-01 16:09:29 CST; 6s ago
     Docs: http://docs.docker.com
  Process: 2300 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
 Main PID: 2300 (code=exited, status=1/FAILURE)

Apr 01 16:09:29 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Apr 01 16:09:29 localhost.localdomain dockerd-current[2300]: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '}' look...y string
Apr 01 16:09:29 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 01 16:09:29 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Apr 01 16:09:29 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Apr 01 16:09:29 localhost.localdomain systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]# 
[root@localhost ~]# tail -20f /var/log/messages
Apr  1 16:02:35 localhost NetworkManager[644]: <info>  (eno16777984): Activation: Stage 4 of 5 (IPv6 Configure Timeout) started...
Apr  1 16:02:35 localhost NetworkManager[644]: <info>  (eno16777984): Activation: Stage 4 of 5 (IPv6 Configure Timeout) complete.
Apr  1 16:09:29 localhost systemd: Starting Docker Storage Setup...
Apr  1 16:09:29 localhost container-storage-setup: mount: wrong fs type, bad option, bad superblock on none,
Apr  1 16:09:29 localhost container-storage-setup: missing codepage or helper program, or other error
Apr  1 16:09:29 localhost container-storage-setup: In some cases useful info is found in syslog - try
Apr  1 16:09:29 localhost container-storage-setup: dmesg | tail or so.
Apr  1 16:09:29 localhost kernel: overlayfs: failed to resolve '/run/container-storage-setup//lower1:/run/container-storage-setup//lower2': -2
Apr  1 16:09:29 localhost container-storage-setup: ERROR: Failed to run unshare -m mount -t overlay -o lowerdir=/run/container-storage-setup//lower1:/run/container-storage-setup//lower2,upperdir=/run/container-storage-setup//upper,workdir=/run/container-storage-setup//work none /run/container-storage-setup//merged
Apr  1 16:09:29 localhost container-storage-setup: ERROR: Can not setup storage driver overlay2 as system does not support it. Specify a different driver.
Apr  1 16:09:29 localhost systemd: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Apr  1 16:09:29 localhost systemd: Failed to start Docker Storage Setup.
Apr  1 16:09:29 localhost systemd: Unit docker-storage-setup.service entered failed state.
Apr  1 16:09:29 localhost systemd: docker-storage-setup.service failed.
Apr  1 16:09:29 localhost systemd: Starting Docker Application Container Engine...
Apr  1 16:09:29 localhost dockerd-current: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '}' looking for beginning of object key string
Apr  1 16:09:29 localhost systemd: docker.service: main process exited, code=exited, status=1/FAILURE
Apr  1 16:09:29 localhost systemd: Failed to start Docker Application Container Engine.
Apr  1 16:09:29 localhost systemd: Unit docker.service entered failed state.
Apr  1 16:09:29 localhost systemd: docker.service failed.

查找资料：
Docker storage drivers:
Ideally, very little data is written to a container’s writable layer, and you use Docker volumes to write data. However, some workloads require you to be able to write to the container’s writable layer. This is where storage drivers come in.

Docker supports several storage drivers, using a pluggable architecture. The storage driver controls how images and containers are stored and managed on your Docker host. After you have read the storage driver overview, the next step is to choose the best storage driver for your workloads. Use the storage driver with the best overall performance and stability in the most usual scenarios.
The Docker Engine provides the following storage drivers on Linux:

Driver:          Description
overlay2:        overlay2 is the preferred storage driver for all currently supported Linux distributions, and requires no extra configuration.
fuse-overlayfs:  fuse-overlayfsis preferred only for running Rootless Docker on a host that does not provide support for rootless overlay2. On Ubuntu and Debian 10, the fuse-overlayfs driver does not need to be used, and overlay2 works even in rootless mode. Refer to the rootless mode documentation for details.
btrfs and zfs:   The btrfs and zfs storage drivers allow for advanced options, such as creating “snapshots”, but require more maintenance and setup. Each of these relies on the backing filesystem being configured correctly.
vfs:             The vfs storage driver is intended for testing purposes, and for situations where no copy-on-write filesystem can be used. Performance of this storage driver is poor, and is not generally recommended for production use.
aufs:            The aufs storage driver Was the preferred storage driver for Docker 18.06 and older, when running on Ubuntu 14.04 on kernel 3.13 which had no support for overlay2. However, current versions of Ubuntu and Debian now have support for overlay2, which is now the recommended driver.
devicemapper:    The devicemapper storage driver requires direct-lvm for production environments, because loopback-lvm, while zero-configuration, has very poor performance. devicemapper was the recommended storage driver for CentOS and RHEL, as their kernel version did not support overlay2. However, current versions of CentOS and RHEL now have support for overlay2, which is now the recommended driver.
overlay:         The legacy overlay driver was used for kernels that did not support the “multiple-lowerdir” feature required for overlay2 All currently supported Linux distributions now provide support for this, and it is therefore deprecated.

The Docker Engine has a prioritized list of which storage driver to use if no storage driver is explicitly configured, assuming that the storage driver meets the prerequisites, and automatically selects a compatible storage driver. You can see the order in the source code for Docker Engine 20.10.

Some storage drivers require you to use a specific format for the backing filesystem. If you have external requirements to use a specific backing filesystem, this may limit your choices. See Supported backing filesystems.

另外找到docker支持文件系统类型：
Supported backing filesystems:

With regard to Docker, the backing filesystem is the filesystem where /var/lib/docker/ is located. Some storage drivers only work with specific backing filesystems.

Storage driver:    Supported backing filesystems
overlay2, overlay:    xfs with ftype=1, ext4
fuse-overlayfs:        any filesystem
aufs:                  xfs, ext4
devicemapper:          direct-lvm
btrfs:                btrfs
zfs:                  zfs
vfs:                   any filesystem

问题原因：
ERROR: Can not setup storage driver overlay2 as system does not support it. Specify a different driver.
devicemapper was the recommended storage driver for CentOS and RHEL, as their kernel version did not support overlay2. However, current versions of CentOS and RHEL now have support for overlay2, which is now the recommended driver.
RHEL或CentOS 使用新的 docker 存储驱动（overlay or overlay2），需要升级系统内核版本到3.10.0-514以上版本，否则不支持 overlay or overlay2 存储驱动。

我的docker配置的默认的存储驱动为 overlay2：
cat /etc/sysconfig/docker-storage-setup
STORAGE_DRIVER=overlay2

cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=

我的Centos7内核版本为：3.10.0-229，版本太低
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

确认我的系统内核是否加载/支持 overlay2 模块/驱动：
[root@localhost ~]# lsmod | grep overlay2
[root@localhost ~]# 
[root@localhost ~]# lsmod | grep overlay
overlay                37831  0 

我的系统不支持 overlay2，仅支持 overlay

解决方案：
方案1）修改 docker 配置

修改docker启动文件，设置使用overlay 存储，添加选项 --storage-driver=overlay
vi /etc/sysconfig/docker
OPTIONS='--storage-driver=overlay --selinux-enabled --log-driver=journald --signature-verification=false'

systemctl daemon-reload
systemctl restart docker

方案2）不修改存储类型，升级内核版本，使用 overlay2
Centos7 升级内核版本 for 支持  docker overlay2 存储驱动

升级软件和内核
yum -y update 
注意：升级所有软件，也会升级内核版本。注意：yum -y upgrade 和 yum -y update的区别，upgrade 只升级所有包，不升级软件和系统内核，update 升级所有包同时也升级软件和系统内核

内核升级完毕后，问题解决，docker 服务正常启动了

[root@localhost ~]# systemctl  status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-04-01 18:00:11 CST; 10min ago
     Docs: http://docs.docker.com
 Main PID: 2604 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─2604 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdrive...
           └─2612 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir ...

Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.135550079+08:00" level=info msg="Loading containers: start."
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.169777432+08:00" level=warning msg="Running modprobe bridge br_netfilter failed w...
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.201924213+08:00" level=info msg="Firewalld running: false"
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.385050677+08:00" level=info msg="Default bridge (docker0) is assigned wit...address"
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.415252165+08:00" level=info msg="Loading containers: done."
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.416744386+08:00" level=warning msg="Not using native diff for overlay2, t...rgument"
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.488179968+08:00" level=info msg="Daemon has completed initialization"
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.488227398+08:00" level=info msg="Docker daemon" commit="7d71120/1.13.1" g...n=1.13.1
Apr 01 18:00:11 localhost.localdomain dockerd-current[2604]: time="2022-04-01T18:00:11.496220774+08:00" level=info msg="API listen on /var/run/docker.sock"
Apr 01 18:00:11 localhost.localdomain systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]# 

另外：如果有条件的话，建议另外准备一块独立磁盘或者独立分区，加参数 -n ftype=1 格式化为 xfs 格式，然后将 /var/lib/docker 挂载上去
格式化：
mkfs.xfs -f -n ftype=1 /dev/sdb2

查看磁盘信息
lsblk -a -f

配置开启自动加载磁盘 
vi /etc/fstab
UUID=c8c91281-1dc2-4da0-af06-c2f182c0ce03   /var/lib/docker                 xfs     defaults        0 0 

手动挂载
mount -a

修改docker启动文件，设置使用overlay 存储，添加选项 --storage-driver=overlay
vi /etc/sysconfig/docker
OPTIONS='--storage-driver=overlay --selinux-enabled --log-driver=journald --signature-verification=false'

systemctl daemon-reload
systemctl restart docker
systemctl enable docker  #如果没有设置开启自启动docker，设置自启动服务

uname -a
docker version
docker info