提供了编程的基础技术教程

zl程序教程

您现在的位置是：首页 >  系统

当前栏目

云原生|kubernetes |使用Prometheus监控k8s cAdvisor篇（进阶篇--- 一）（centos操作系统）

k8scentosKubernetes监控操作系统 --- 原生 Prometheus
2023-09-14 09:13:14 时间

前言:

为了能够提前发现kubernetes集群的问题以及方便快捷的查询容器的各类参数，比如，某个pod的内存使用异常高企 等等这样的异常状态（虽然kubernetes有自动重启或者驱逐等等保护措施，但万一没有配置或者失效了呢），容器的内存使用量限制，过去10秒容器CPU的平均负载等等容器的运行参数，这些情况我们自然还是将kubernetes集群纳入到监控系统中好些，毕竟能够发现问题和解决问题更加的优雅嘛。

因此，我们需要能够有一个比较全面的监测容器运行的实时的监控系统，版本答案当然就是Prometheus了。Prometheus监控系统可以多维度的，方便的将我们需要的信息收集起来，然后通过Grafana做一个华丽的展示。

那么，对于容器这个对象来说，我们要使用的收集器就是cAdvisor啦，但cAdvisor这个收集器和node_exporter，mysqld_exporter 这些收集器不太一样，它是集成在kubelet这个服务内的，因此，我们不需要额外的安装cAdvisor收集器，也就是说不需要像node_exporter这样的系统信息收集器一样单独部署了，只要kubernetes的节点上有运行kubelet这个服务就可以了。

下面就kubernetes集群的Prometheus专用于容器的实时信息收集器cAdvisor如何认识它，如何部署它并集成到Prometheus内做一个详细的介绍。

一，

cAdvisor的简介

cAdvisor是一个谷歌开发的容器监控工具，它被内嵌到k8s中作为k8s的监控组件。cAdvisor对Node机器上的资源及容器进行实时监控和性能数据采集，包括CPU使用情况、内存使用情况、网络吞吐量及文件系统使用情况，由于cAdvisor是集成在Kubelet中的，因此，当kubelet启动时会自动启动cAdvisor，即一个cAdvisor仅对一台Node机器进行监控。

二，

其它的用于收集容器信息的信息收集器

heapster和Metrics server以及kube-state-metrics都可用于提供容器信息，但它们所提供的信息维度是不同的，heapster已经被Metrics server所取代 了，如果没记错的话，应该是从k8s的1.16版本后放弃了heapster。

Metrics server作为新的容器信息收集器，是从 api-server 中获取容器的 cpu、内存使用率这种监控指标，并把他们发送给存储后端，可以算作一个完整的监控系统。cAdvisor是专有的容器信息收集，是一个专有工具的地位，而kube-state-metrics是偏向于kubernetes集群内的资源对象，例如deployment，StateFulSet，daemonset等等资源，可以算作一个特定的数据源。

三，

cAdvisor的初步使用

本文以一个minikube搭建的kubernetes单实例为例子，IP地址为：192.168.217.23

A，

关于kubelet

kubelet的API：

kubelet是kubernetes集群中真正维护容器状态，负责主要的业务的一个关键组件。每个节点上都运行一个 kubelet 服务进程，默认监听 10250 端口，接收并执行 master 发来的指令，管理 Pod 及 Pod 中的容器。kubernetes的节点IP+10250端口就是kubelet的API。

几个重要的端口：

10250（kubelet API）：kubelet server 与 apiserver 通信的端口，定期请求 apiserver 获取自己所应当处理的任务，通过该端口可以访问获取 node 资源以及状态。kubectl查看pod的日志和cmd命令，都是通过kubelet端口10250访问。

10248端口是什么呢？是kubelet的健康检查端口，可以通过 kubelet 的启动参数 –healthz-port 和 –healthz-bind-address 来指定监听的地址和端口。

需要注意的是，Kubernetes 1.11+ 版本以后，kubelet 就移除了 10255 端口， metrics 接口又回到了 10250 端口中，我的minikube版本是1.18.8，自然是没有10255端口了。

低版本的kubernetes还有一个4194端口，此端口是cAdvisor的web管理界面的端口，可能是出于安全漏洞的考虑，后续版本移除了此端口，因此，此端口在我这个版本内并没有开启。

kubernetes的版本查询：

[root@node3 ~]# k get no
NAME    STATUS   ROLES    AGE   VERSION
node3   Ready    master   18d   v1.18.8

kubelet的端口情况： 

[root@node3 ~]# netstat -antup |grep kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      29681/kubelet       
tcp        0      0 127.0.0.1:39937         0.0.0.0:*               LISTEN      29681/kubelet       
tcp        0      0 192.168.217.23:51058    192.168.217.23:8443     ESTABLISHED 29681/kubelet       
tcp6       0      0 :::10250                :::*                    LISTEN      29681/kubelet       
tcp6       0      0 192.168.217.23:10250    192.168.217.1:59295     ESTABLISHED 29681/kubelet

 kube-apiserver的service：

[root@node3 ~]# k get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        18d

B，

API的使用

既然都说了节点IP+10250是kubelet的API了，那么，我们肯定可以从这个API里获取到一些信息了，这些信息其实就是cAdvisor收集到的，如何使用这个API呢？这个API可是需要使用证书的https哦。因此，计划建立一个sa，通过sa的token来登陆这个API

（1）

利用ServiceAccount访问API

找一个具有cluster-admin权限的ServiceAccount，其实每个集群内都很容易找到这样的sa，但为了说明问题还是新建一个任意的具有最高权限的sa吧，实际的生产中可不要这么搞哦。

新建一个sa：

kubectl create ns monitor-sa  #创建一个monitor-sa的名称空间
kubectl create serviceaccount monitor -n monitor-sa  #创建一个sa账号
kubectl create clusterrolebinding monitor-clusterrolebinding -n monitor-sa --clusterrole=cluster-admin  --serviceaccount=monitor-sa:monitor

删除这些都会吧，我就不演示了。

查看secret： 

[root@node3 ~]# k get secrets -n monitor-sa 
NAME                  TYPE                                  DATA   AGE
default-token-fw7pq   kubernetes.io/service-account-token   3      81s
monitor-token-tf48k   kubernetes.io/service-account-token   3      81s

获取登录用的token：

[root@node3 ~]# k describe secrets -n monitor-sa monitor-token-tf48k 
Name:         monitor-token-tf48k
Namespace:    monitor-sa
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: monitor
              kubernetes.io/service-account.uid: 1c902004-bd78-4aca-b37e-5fe751dfcd4a

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1066 bytes
namespace:  10 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IllIZ2VhUzN6Q2lSUmE3MEM0ZWpJU255ZXRGRjlkam9rT2JrN0FjNEd1WDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tdGY0OGsiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjOTAyMDA0LWJkNzgtNGFjYS1iMzdlLTVmZTc1MWRmY2Q0YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.aUW1XRfGsHiBDWME4JtCs_8144m308ioQ9W1zOg80YIA1VZq5SKAuq_R4XUTyDW7stuSUtTpXhG_HkzoO5sxU7SW6EBIvnOhz-hp3P3S_7BTd5QgZ2PZP9PnJP46lSNS2g0VpqThqtJzXPNZWZnquRV9oHpdeKeeC9b8dcdIwMw_HYC30ydCzt15axf3YNUzVsW1xgLM9fkhTthBm1Z02kcPMqa49nXRQFS3AwVOnlh7Mn4z8OxufVuFY_f5PDkHwnYX4zRgN0PL3On5k_yZgDWgT2kh63fTi4Skmlee7i_1t_lucU-B_8JfLOqiUCUCkt9nr3W4Qj2FOxRyOJ2JtQ

使用10250这个API：

#将token保存到变量TOKEN里面，然后下面的API调试接口命令里面引用
[root@node3 ~]# TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IllIZ2VhUzN6Q2lSUmE3MEM0ZWpJU255ZXRGRjlkam9rT2JrN0FjNEd1WDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tdGY0OGsiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjOTAyMDA0LWJkNzgtNGFjYS1iMzdlLTVmZTc1MWRmY2Q0YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.aUW1XRfGsHiBDWME4JtCs_8144m308ioQ9W1zOg80YIA1VZq5SKAuq_R4XUTyDW7stuSUtTpXhG_HkzoO5sxU7SW6EBIvnOhz-hp3P3S_7BTd5QgZ2PZP9PnJP46lSNS2g0VpqThqtJzXPNZWZnquRV9oHpdeKeeC9b8dcdIwMw_HYC30ydCzt15axf3YNUzVsW1xgLM9fkhTthBm1Z02kcPMqa49nXRQFS3AwVOnlh7Mn4z8OxufVuFY_f5PDkHwnYX4zRgN0PL3On5k_yZgDWgT2kh63fTi4Skmlee7i_1t_lucU-B_8JfLOqiUCUCkt9nr3W4Qj2FOxRyOJ2JtQ

curl https://127.0.0.1:10250/metrics/cadvisor -k -H "Authorization: Bearer $TOKEN"

OK，输出茫茫多，稍微截一点吧，剩下的就不贴了，具体的含义后面在说吧：

[root@node3 ~]# curl https://127.0.0.1:10250/metrics/cadvisor -k -H "Authorization: Bearer $TOKEN" |less


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="19.03.9",kernelVersion="5.16.9-1.el7.elrepo.x86_64",osVersion="CentOS Linux 7 (Core)"} 1
# HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
# TYPE container_cpu_cfs_periods_total counter
container_cpu_cfs_periods_total{container="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019fe473_4f9f_4827_bc76_526f7aa2f97d.slice",image="",name="",namespace="kube-system",pod="kube-flannel-ds-amd64-6cdl5"} 19477 1669053701394
container_cpu_cfs_periods_total{container="addon-resizer",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1fcd63_b032_421c_b929_5c7c905e7ef3.slice/docker-9edba1d252251d7e2886f8a278ee30bb055e73e0f07466cd6680fb13788ab755.scope",image="sha256:ee3db71672efc913209ec07e9912723dd3fc1acfbf92203743bdd847f1eb1578",name="k8s_addon-resizer_kube-state-metrics-c96857b7b-twtpz_kube-system_7f1fcd63-b032-421c-b929-5c7c905e7ef3_1",namespace="kube-system",pod="kube-state-metrics-c96857b7b-twtpz"} 4382 1669053703291
container_cpu_cfs_periods_total{container="kube-flannel",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019fe473_4f9f_4827_bc76_526f7aa2f97d.slice/docker-84a994ffae8ca838967510c7a79c8b7a5b8befb7c9be78acb4e697cb10a46953.scope",image="sha256:4e9f801d2217e98e94de72cefbcb010a7f2caccf03834dfd12a8e60abcaaecfd",name="k8s_kube-flannel_kube-flannel-ds-amd64-6cdl5_kube-system_019fe473-4f9f-4827-bc76-526f7aa2f97d_35",namespace="kube-system",pod="kube-flannel-ds-amd64-6cdl5"} 19467 1669053704667
# HELP container_cpu_cfs_throttled_periods_total Number of throttled period intervals.

（2）

将上面的命令改造一下，使用kube-apiserver 的服务来访问kube-apiserver 的API接口：

curl https://10.96.0.1/api/v1/nodes/node3/proxy/metrics -k -H "Authorization: Bearer $TOKEN"

这个命令输出的也比较多，就截取一部分吧：

[root@node3 ~]# curl https://10.96.0.1/api/v1/nodes/node3/proxy/metrics -k -H "Authorization: Bearer $TOKEN" |less





  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0

OK，以上是通过一个具有admin权限的serviceAccount账户直接连接cadvisor和kube-apiserver的API动态获得节点的pod，endpoint等等资源的各个维度的数据。

现在需要将这些集成到Prometheus server里了。

四，

二进制部署的Prometheus server 接入minikube的数据： 

将上面的token内容写入/opt/k8s.token这个文件：

echo "eyJhbGciOiJSUzI1NiIsImtpZCI6IllIZ2VhUzN6Q2lSUmE3MEM0ZWpJU255ZXRGRjlkam9rT2JrN0FjNEd1WDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tdGY0OGsiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjOTAyMDA0LWJkNzgtNGFjYS1iMzdlLTVmZTc1MWRmY2Q0YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.aUW1XRfGsHiBDWME4JtCs_8144m308ioQ9W1zOg80YIA1VZq5SKAuq_R4XUTyDW7stuSUtTpXhG_HkzoO5sxU7SW6EBIvnOhz-hp3P3S_7BTd5QgZ2PZP9PnJP46lSNS2g0VpqThqtJzXPNZWZnquRV9oHpdeKeeC9b8dcdIwMw_HYC30ydCzt15axf3YNUzVsW1xgLM9fkhTthBm1Z02kcPMqa49nXRQFS3AwVOnlh7Mn4z8OxufVuFY_f5PDkHwnYX4zRgN0PL3On5k_yZgDWgT2kh63fTi4Skmlee7i_1t_lucU-B_8JfLOqiUCUCkt9nr3W4Qj2FOxRyOJ2JtQ" >/opt/k8s.token

Master节点发现 

在二进制部署的Prometheus服务器，找出配置文件并增加下面这一段（API Serevr 节点自动发现，如果是HA高可用的apiserver）：

  - job_name: 'kubernetes-apiserver'
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://192.168.217.23:8443
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /opt/k8s.token
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /opt/k8s.token
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
      action: keep
      regex: default;kubernetes;https
    - target_label: __address__
      action: replace
      replacement: 192.168.217.23:8443   # api集群地址

Node节点发现

在通用配置文件prometheus.yml，末尾添加Node节点发现：

  - job_name: 'kubernetes-nodes-monitor'
    kubernetes_sd_configs:
    - role: node
      api_server: https://192.168.217.23:8443
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /opt/k8s.token
    scheme: http
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /opt/k8s.token
    relabel_configs:
    - source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:9100'
      target_label: __address__
      action: replace
    - source_labels: [__meta_kubernetes_node_label_failure_domain_beta_kubernetes_io_region]
      regex: '(.*)'
      replacement: '${1}'
      action: replace
      target_label: LOC
    - source_labels: [__meta_kubernetes_node_label_failure_domain_beta_kubernetes_io_region]
      regex: '(.*)'
      replacement: 'NODE'
      action: replace
      target_label: Type
    - source_labels: [__meta_kubernetes_node_label_failure_domain_beta_kubernetes_io_region]
      regex: '(.*)'
      replacement: 'K8S-test'
      action: replace
      target_label: Env
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)

部署kube-state-metrics 到minikube：

RBAC文件：

cat >kube-state-metrics-rbac-new.yaml <<EOF

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["extensions","apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-state-metrics-resizer
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get"]
- apiGroups: ["extensions","apps"]
  resources:
  - deployments
  resourceNames: ["kube-state-metrics"]
  verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-syste
EOF

service文件：

 cat >kube-state-metrics-svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "kube-state-metrics"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
    nodePort: 32222
  - name: telemetry
    port: 8081
    targetPort: telemetry
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics
EOF

deployment 文件：

cat >kube-state-metrics-deploy-new.yaml <<EOF

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.9.0
        ports:
        - name: http-metrics
          containerPort: 8080
        - name: telemetry
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/addon-resizer:1.8.6
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 30Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        volumeMounts:
          - name: config-volume
            mountPath: /etc/config
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --container=kube-state-metrics
          - --cpu=100m
          - --extra-cpu=1m
          - --memory=100Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=kube-state-metrics
      volumes:
        - name: config-volume
          configMap:
            name: kube-state-metrics-config
---

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-state-metrics-config
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration
EOF

kube-state-metrics接入Prometheus：

  - job_name: "kube-stat-metrics"
    static_configs:
      - targets: ["192.168.217.23:32222"]

测试环节：

打开Prometheus的管理界面的service discovery：

打开Prometheus的管理界面的targets：

 通过cadvisor采集器查询版本信息：

cadvisor_version_info

 node_boot_time_seconds

node_cpu_seconds_total

 计算平均五分钟内的节点的CPU使用率：

1- sum(increase(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)/sum(increase(node_cpu_seconds_total[5m])) by (instance)

 查询节点的内存使用率：

(1- (node_memory_Buffers_bytes + node_memory_Cached_bytes + node_memory_MemFree_bytes) / node_memory_MemTotal_bytes) * 100

查询每个节点的总内存：

 (node_memory_MemTotal_bytes)

etcd存储的对象数目：

etcd_object_counts

 这些都是可以验证的，比如clusterrole的数量，查询出来的是62个，在集群上实际查询也是可以验证到这个数字是对的：

[root@node3 kube-stat]# k get clusterrole |wc -l
63

相关文章