一.系统环境

二.前言

当生产环境中部署了多个容器，我们需要部署监控工具来监控容器的资源利用率和性能。本文介绍两种容器监控工具：cAdvisor，weavescope。

cAdvisor 是谷歌公司用来分析运行中的 Docker 容器的资源占用以及性能特性的工具。cAdvisor 是一个运行中的守护进程用来收集、聚合、处理和导出运行容器相关的信息，每个容器保持独立的参数、历史资源使用情况和完整的资源使用数据。cadvisor监控容器主要是监控物理机的目录：/var/run，/sys，/var/lib/docker。

Weave Scope是Docker和Kubernetes的可视化和监视工具。它提供了自上而下的应用程序视图以及整个基础架构视图，并允许您实时诊断将分布式容器化应用程序部署到云提供商时遇到的任何问题。github开源项目地址: https://github.com/weaveworks/scope

Weave Scope可以监控Kubernetes集群的一系列资源的状态、资源使用情况、应用拓扑、还可以直接通过UI界面进行调试，查看日志等操作。

三.容器监控的原理

Docker是基于Namespace、Cgroups和联合文件系统实现的，Cgroups不仅可以用于容器资源的限制，还可以提供容器的资源使用率。不管用什么监控方案，底层数据都来源于Cgroups，Cgroups的工作目录/sys/fs/cgroup 下包含了Cgroups的所有内容。Cgroups包含了很多子系统，可以对CPU，内存，PID，磁盘IO等资源进行限制和监控。

查看 /sys/fs/cgroup 文件夹：

每一个文件夹都代表了Cgroups的子系统，docker会在Cgroups的子系统下创建docker文件夹。

四.使用docker stats监控容器

容器的监控可以使用命令docker stats查看

[root@k8smaster ~]# docker stats
CONTAINER ID   NAME                                                                                                            CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O         PIDS
7b38d98d8db5   nginx                                                                                                           0.07%     4.355MiB / 3.32GiB   0.13%     603MB / 604MB    1.91MB / 0B       3
f52a518746ec   harbor-jobservice                                                                                               0.20%     31.14MiB / 3.32GiB   0.92%     16.3MB / 229MB   75.5MB / 0B       8
d2422dc8ed83   harbor-core                                                                                                     0.11%     48.21MiB / 3.32GiB   1.42%     608MB / 609MB    78.8MB / 0B       8
cd780fa84d5f   redis                                                                                                           0.21%     4.336MiB / 3.32GiB   0.13%     230MB / 16.8MB   4.42MB / 2.27MB   5
7fba7cc774eb   harbor-portal                                                                                                   0.06%     8.703MiB / 3.32GiB   0.26%     1.3MB / 5.92MB   13.4MB / 0B       3
9080c39aea05   registry                                                                                                        0.08%     115.7MiB / 3.32GiB   3.40%     304MB / 301MB    26.8MB / 193MB    5
fbeb3dbb7f29   harbor-db                                                                                                       0.09%     75.74MiB / 3.32GiB   2.23%     1.6MB / 1.54MB   18.7MB / 77.4MB   8
7142a107f480   registryctl                                                                                                     0.06%     14.28MiB / 3.32GiB   0.42%     914kB / 709kB    29.7MB / 0B       9
187b4a82403a   harbor-log                                                                                                      0.06%     6.738MiB / 3.32GiB   0.20%     2.3MB / 658kB    12.8MB / 126kB    11
0797cbb90156   k8s_kube-controller-manager_kube-controller-manager-k8smaster_kube-system_0bac7daeb78f382a2a6fb01d62288896_29   0.27%     43.24MiB / 3.32GiB   1.27%     0B / 0B          156MB / 0B        5
2949d6764ab5   k8s_POD_kube-controller-manager-k8smaster_kube-system_0bac7daeb78f382a2a6fb01d62288896_20                       0.00%     1.395MiB / 3.32GiB   0.04%     0B / 0B          0B / 0B           1
7288858a1ecc   k8s_POD_kube-apiserver-k8smaster_kube-system_463a955551c752a0b753700879aec991_20                                0.00%     1.461MiB / 3.32GiB   0.04%     0B / 0B          0B / 0B           1
e3323c5d994c   k8s_POD_etcd-k8smaster_kube-system_1dcb59df47c677756cdf25f28d920325_21                                          0.00%     1.359MiB / 3.32GiB   0.04%     0B / 0B          0B / 0B           1
511e94439641   k8s_kube-scheduler_kube-scheduler-k8smaster_kube-system_298ac610fc4625e596fecc2821dc924e_30                     1.04%     32.47MiB / 3.32GiB   0.96%     0B / 0B          84.2MB / 0B       8
45a5df586d7e   k8s_POD_kube-scheduler-k8smaster_kube-system_298ac610fc4625e596fecc2821dc924e_20                                0.00%     1.516MiB / 3.32GiB   0.04%     0B / 0B          668kB / 0B        1
1edbf56b01ee   redhat7.2                                                                                                       0.00%     904KiB / 3.32GiB     0.03%     996B / 0B        4.73MB / 0B       1
c2d01513585b   redhat                                                                                                          0.00%     860KiB / 3.32GiB     0.02%     1.09kB / 0B      10.2MB / 0B       2

五.部署cadvisor监控容器

拉取cadvisor镜像

[root@k8smaster ~]# docker pull hub.c.163.com/xbingo/cadvisor:latest
latest: Pulling from xbingo/cadvisor
09d0220f4043: Pull complete 
151807d34af9: Pull complete 
14cd28dce332: Pull complete 
Digest: sha256:20ddac7ab8fdaff0549c5dcf046f0cb6f8ad064e20d9004dcc07132b907c4b68
Status: Downloaded newer image for hub.c.163.com/xbingo/cadvisor:latest
hub.c.163.com/xbingo/cadvisor:latest

查看cadvisor镜像的历史信息，端口为EXPOSE 8080/tcp

[root@k8smaster ~]# docker history hub.c.163.com/xbingo/cadvisor:latest
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
4bc3588563b1   5 years ago   /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/cadv…   0B        
<missing>      5 years ago   /bin/sh -c #(nop) EXPOSE 8080/tcp               0B        
<missing>      5 years ago   /bin/sh -c #(nop) ADD file:d8a1cb9e0c5b0d90d…   23MB      
<missing>      5 years ago   /bin/sh -c apk add --update ca-certificates …   20MB      
<missing>      5 years ago   /bin/sh -c #(nop) ENV GLIBC_VERSION=2.23-r1     0B        
<missing>      5 years ago   /bin/sh -c #(nop) MAINTAINER dengnan@google.…   0B        
<missing>      5 years ago   /bin/sh -c #(nop) ADD file:af671c91a986f389f…   5.26MB    

cadvisor监控容器主要是监控物理机的目录：/var/run，/sys，/var/lib/docker，创建容器，挂载数据卷，使cadvisor容器可以监控相关目录。

[root@k8smaster ~]# docker run -dit --name=cadvisor --restart=always -p 8080:8080 -v /var/run:/var/run -v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:ro hub.c.163.com/xbingo/cadvisor:latest
db87f17745b6c3764aaa2597b94eac7e7464af333b0efcb7d87d026c9e78fde5

[root@k8smaster ~]# docker ps 
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED         STATUS                 PORTS                                       NAMES
db87f17745b6   hub.c.163.com/xbingo/cadvisor:latest                  "/usr/bin/cadvisor -…"   5 seconds ago   Up 5 seconds           0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor

在浏览器输入物理机IP:8080访问cadvisor，浏览器打开http://192.168.110.137:8080/，即可看见cadvisor监控页面

六.部署weavescope监控容器

上传scope脚本并授予可执行权限，也可以自行下载scope脚本

[root@k8smaster ~]# curl -L git.io/scope -o /usr/local/bin/scope

[root@k8smaster ~]# ll -h scope 
-rw-r--r-- 1 root root 12K 10月  7 02:54 scope

[root@k8smaster ~]# chmod +x scope 

[root@k8smaster ~]# ll -h scope 
-rwxr-xr-x 1 root root 12K 10月  7 02:54 scope

执行scope脚本会拉取weaveworks/scope:1.13.1镜像

[root@k8smaster ~]# ./scope 
Usage:
scope launch {OPTIONS} {PEERS} - Launch Scope
scope stop                     - Stop Scope
scope command                  - Print the docker command used to start Scope
scope help                     - Print usage info
scope version                  - Print version info

PEERS are of the form HOST[:PORT]
HOST may be an ip or hostname.
PORT defaults to 4040.

Launch options:
Unable to find image 'weaveworks/scope:1.13.1' locally
1.13.1: Pulling from weaveworks/scope
c9b1b535fdd9: Pull complete 
550073704c23: Pull complete 
8738e5bbaf1d: Pull complete 
0a8826d26027: Pull complete 
387c1aa951b4: Pull complete 
e72d45461bb9: Pull complete 
75cc44b65e98: Pull complete 
11f7584a6ade: Pull complete 
a5aa3ebbe1c2: Pull complete 
7cdbc028c8d2: Pull complete 
Digest: sha256:4342f1c799aba244b975dcf12317eb11858f9879a3699818e2bf4c37887584dc
Status: Downloaded newer image for weaveworks/scope:1.13.1
Usage of /home/weave/scope:
  -app-only
    	Only run the app.
  -app.aws.create.tables
    	Create the tables in DynamoDB
......
  -weave
    	Enable Weave Net integrations. (default true)
  -weave.hostname string
    	Hostname to advertise/lookup in WeaveDNS (default "scope.weave.local.")

[root@k8smaster ~]# docker images
REPOSITORY                                                        TAG        IMAGE ID       CREATED         SIZE
weaveworks/scope                                                  1.13.1     cd4cd8fcce1b   19 months ago   89.7MB

安装weavescope

[root@k8smaster ~]# ./scope launch
cf1b38f8dcdd5f61ae1d157800f70e59fdc11ed12bfb7b76f78c73e4048be86d
Scope probe started
Weave Scope is listening at the following URL(s):
  * http://172.28.0.1:4040/
  * http://172.18.0.1:4040/
  * http://192.168.110.137:4040/
  
[root@k8smaster ~]# docker ps
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS                 PORTS                                       NAMES
cf1b38f8dcdd   weaveworks/scope:1.13.1                               "/home/weave/entrypo…"   4 minutes ago    Up 4 minutes                                                       weavescope  

浏览器打开http://192.168.110.137:4040/，即可查看监控界面，监控项目还是多样的