您现在的位置是：首页 > Javascript
当前栏目
总结：Promethus配置文件
2023-04-18 16:28:38 时间
一、介绍
Prometheus 启动的时候，可以加载运行参数 -config.file 指定配置文件，默认为 prometheus.yml。
Prometheus的配置文件是YAML格式。Prometheus的解压包里自带了一个默认的配置文件prometheus.yml。让我们来看一下：
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']在这个缺省的配置文件里定义了4个单元：global、alerting、rule_files和scrape_configs。
在配置文件中我们可以指定 global, alerting, rule_files, scrape_configs, remote_write, remote_read 等属性。
global：配置全局的信息，如抓取监控数据的间隔，抓取业务数据接口的超时时间，告警规则执行周期等
alerting：配置告警发送到的alermanager的地址
rule_files：告警规则文件，数据聚合配置
scrape_configs：配置抓取业务监控数据的相关信息，如url，拉取时间间隔，拉取的超时时间等
remote_write：将数据投递到远程地址，如聚合数据投递到hubble-adapter
remote_read：
下面介绍下每个单元。
二、global
global 属于全局的默认配置，它主要包含 4 个属性，
- scrape_interval: 拉取 targets 的默认时间间隔，即拉取业务监控数据的间隔时间。
- scrape_timeout: 拉取一个 target 的超时时间，即拉取业务监控数据接口的超时时间。
- evaluation_interval: 执行 rules 的时间间隔。即多久遍历一次告警规则列表，判断每个规则是否触发告警。和rule_files的加载没关系
- external_labels: 额外的属性，会添加到拉取的数据并存到数据库中。
配置文件结构大概为：
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # By default, scrape targets every 15 seconds.
scrape_timeout: 10s # is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
三、alerting
通常我们可以使用运行参数 -alertmanager.xxx 来配置 Alertmanager， 但是这样不够灵活，没有办法做到动态更新加载，以及动态定义告警属性。
所以 alerting 配置主要用来解决这个问题，它能够更好的管理 Alertmanager, 主要包含 2 个参数：alert_relabel_configs 和 alertmanagers
1、alertmanagers
用于动态发现 Alertmanager 的地址。
如下配置：alertmanager.prom-alert.svc:9093，通过K8S自动发现机制找到本集群内的alertmanager的地址并将告警发送过去。
所以，每个prometheus集群一定会部署一个alertmanager组件。
如下，系统中指定了Alertmanager路径，因为最终需要投递告警到这个服务，如下图：
2、alert_relabel_configs
作用：在告警发生时，动态修改标签内容，一般作用是在告警产生时修改标签，如保留哪些标签（labelkeep），删除哪些标签（labeldrop）。具体的有哪些属性，请参考：Configuration | Prometheus
下面着重说明两个属性：action和regex
action
基于正则表达式匹配执行的操作。包括移除标签，保留标签等，具体可参考：Configuration | Prometheus
action枚举：
replace: 将正则表达式与串联的source_labels匹配。然后，将target_label设置为replace，用替换中的匹配组引用(${1}, ${2}, ...)替换为其值。 如果正则表达式不匹配，则不会进行替换。
keep: 删除其正则表达式与串联的source_labels不匹配的目标。
drop: 删除其正则表达式与串联的source_labels匹配的目标。
hashmod: 将target_label设置为串联的source_labels的哈希的模数。
labelmap: 将正则表达式与所有标签名称匹配。 然后，将匹配标签的值复制到通过替换为它们的值替换的匹配组引用(${1}, ${2}, ...)给出的标签名称。
labeldrop: 将正则表达式与所有标签名称匹配。 任何匹配的标签将从标签集中删除。
labelkeep: 将正则表达式与所有标签名称匹配。 任何不匹配的标签将从标签集中删除。regex
作用是匹配标签的正则表达式。
案例
下面的案例中，action为labeldrop，就是字面意思，需要移除key为prometheus_replica的标签。
prometheus_replica是自定义的标签，告警的时候就会带上这个标签，由于我们prometheus有两个节点pod0和pod1，但是告警产生的时候我们只需要报出来一条就行了，因此把pod的标签去掉后，两个节点产生的告警就完全一样了，就能控制只产生一条。
source_labels
__开头的是保留label
source_labels也可以定义自定义的label
以下为QKE上的案例：下面写错了一句话，应该是将两个标签合并替换为node，而不是Node
四、rule_files
1、介绍
作用：获取所有规则文件中的规则，包括记录规则(recording rules)与告警规则(alerting rule)。
2、记录规则(recording rules)
record规则其实就是数据处理的规则，如数据聚合，如下：
我们可以单独定义数据聚合规则文件，也可以和告警规则文件放一起，但是一般分开放好理解
注意：一定是规则文件，不包含配置文件，如果指定的文件中包含配置文件内容，则会报错。
按照配置的目录，找了下rancher上对应武汉集群的prometheus项目下的prometheus服务，进入控制台：
我们的配置文件是这样的：
rule_files:
- /etc/prometheus/rules/*rules.yaml
所以进入此目录下发现只有一个alert-rules.yaml，恰好就是我们的告警的配置。
3、rule_files的修改
关于rule_files的修改途径：
1、rule_files文件可以在rancher上修改；
2、通过Prometheus Operator提供的CRD修改。Prometheus Operator会去创建Prometheus、PodMonitor、ServiceMonitor、AlertManager以及PrometheusRule这5个CRD资源对象，所以，可以直接调用K8S的API去修改PrometheusRule，从而达到修改rule以及其他配置的效果，如增加record配置等。
4、rule_files的加载
rule_files文件经常会配置多个规则文件，prometheus是同时去加载这些文件，当某个文件中有错误的配置时，则本次加载所有的规则都会失败。
所以，多个规则文件会相互影响，一旦有配置错误，则会导致其它的规则文件加载失败。
但是，prometheus加载失败并不会影响之前的已经加载好的规则。
五、scrape_configs
scrape_configs 主要用于配置拉取数据节点，每一个拉取配置主要包含以下参数：
- job_name：任务名称
- honor_labels： 用于解决拉取数据标签有冲突，当设置为 true, 以拉取数据为准，否则以服务配置为准
- params：数据拉取访问时带的请求参数
- scrape_interval： 拉取时间间隔
- scrape_timeout: 拉取超时时间
- metrics_path： 拉取节点的 metric 路径
- static_configs：配置访问路径前缀，如ip+port，或者域名地址，或者通过服务发现，类似alertmanager.prom-alert.svc:9093
- scheme： 拉取数据访问协议，如http
- sample_limit： 存储的数据标签个数限制，如果超过限制，该数据将被忽略，不入存储；默认值为0，表示没有限制
- relabel_configs： 拉取数据重置标签配置
- metric_relabel_configs：metric 重置标签配置
六、remote_write
remote_write 主要用于可写远程存储配置，主要包含以下参数：
- url: 访问地址
- remote_timeout: 请求超时时间
- write_relabel_configs: 标签重置配置, 拉取到的数据，经过重置处理后，发送给远程存储
案例：
remote_write:
- url: http://xxx:9988/prom2hubble/push?group=xxx
remote_timeout: 30s
write_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: obser:(.*)
replacement: $1
action: keep
- separator: ;
regex: (.*)
target_label: hubble_endpoint
replacement: hubble_qpaas_obser
action: replace
- separator: ;
regex: (.*)
target_label: group
replacement: hubble
action: replace
- separator: ;
regex: (.*)
target_label: hubble_step
replacement: "60"
action: replace
- separator: ;
regex: label_qke_cloud_qiyi_domain_(.*)
replacement: $1
action: labelmap
- separator: ;
regex: (job|endpoint|service|pod|instance|namespace|prometheus.*|label_qke_cloud_qiyi_domain_.*)
replacement: $1
action: labeldrop
queue_config:
capacity: 2500
max_shards: 200
min_shards: 1
max_samples_per_send: 500
batch_send_deadline: 5s
min_backoff: 30ms
max_backoff: 100ms
metadata_config:
send: true
send_interval: 1m
七、remote_read
remote_read 主要用于可读远程存储配置，主要包含以下参数：
- url: 访问地址
- remote_timeout: 请求超时时间
八、服务发现
1、介绍
ServiceDiscoveryConfig 主要用于 target 发现，大体分为两类，静态配置和动态发现。
在 Prometheus 的配置中，一个最重要的概念就是数据源 target，而数据源的配置主要分为静态配置和动态发现, 大致为以下几类：
- static_configs: 静态服务发现
- eureka_sd_config：eureka服务发现，发现真实的实例节点的ip+port，参考：Configuration | Prometheus
- dns_sd_configs: DNS 服务发现
- file_sd_configs: 文件服务发现
- consul_sd_configs: Consul 服务发现
- serverset_sd_configs: Serverset 服务发现
- nerve_sd_configs: Nerve 服务发现
- marathon_sd_configs: Marathon 服务发现
- kubernetes_sd_configs: Kubernetes 服务发现
- gce_sd_configs: GCE 服务发现
- ec2_sd_configs: EC2 服务发现
- openstack_sd_configs: OpenStack 服务发现
- azure_sd_configs: Azure 服务发现
- triton_sd_configs: Triton 服务发现
它们具体使用以及配置模板，请参考服务发现配置模板。
它们中最重要的，也是使用最广泛的应该是 static_configs, 其实那些动态类型都可以看成是某些通用业务使用静态服务封装的结果。
2、eureka_sd_configs案例介绍
代码地址：
prometheus/prometheus-eureka.yml at release-2.36 · prometheus/prometheus · GitHub
3、kubernetes_sd_configs
我们的微服务本质上是采用的kubernetes_sd_configs。
但是我们是通过Prometheus Operator提供的ServiceMonitor间接创建了kubernetes_sd_configs
如下关注点：
1、kubernetes_sd_configs中的api_server：抓取指标的地址前缀
2、metrics_path：抓取指标的具体路径
- job_name: qke-generic-hubble-manager/hubble-alarm-agg-condition-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: m-agg-condition
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
十、关于relabel_configs
relabel_configs分为<metric_relabel_configs>和<alert_relabel_configs>两类。
指标：
告警：
七、AlertManager配置
案例：
为什么是webhook_configs？不支持remote_write？
global:
resolve_timeout: 10m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 24h #重复报警的时间间隔为24h
receiver: hubble
receivers:
- name: 'hubble'
webhook_configs:
- url: 'http://hubble.adapter.qiyi.domain:9988/prom2hubble/alert'
八、告警规则配置
案例：
主要包含几部分：
groups:
- name: PrometheusRule #报警规则组的名字，可以类比为hubble的策略模板
rules: #策略列表
- expr: up{job="alertmanager"} == 0 #表达式
alert: alertmanagerInstanceDown #告警的triggername
for: 2m #2分钟比较一次，和连续几个点类似
annotations: #告警信息必要的信息，labels是告警消息的tag信息
alertlevel: "P2"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{ $value }}"
summary: "[prometheus-cluster-wh] alertmanager is down"
- expr: increase(alertmanager_notifications_failed_total{job="alertmanager"}[5m])/increase(alertmanager_notifications_total{job="alertmanager"}[5m]) > 0.3
alert: alertmanagerSendOutFail
for: 5m
annotations:
alertlevel: "P2"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{ $value }}" #value 就是数据的当前值
summary: "[prometheus-cluster-wh] failed to sendout alerts >30%"
description: "应用名： {{ $labels.job }} 实例名： {{ $labels.instance }} ， 环境： {{ $labels.env }} , 当前值为 ： {{ $value }}" # labels其实就是数据中的tag，如job，instance等
八、配置文件案例
QKE配置文件案例：
global:
scrape_interval: 30s
scrape_timeout: 10s
evaluation_interval: 30s
external_labels:
prometheus: 58-hubble/k8s
prometheus_replica: prometheus-k8s-0
alerting:
alert_relabel_configs:
- separator: ;
regex: prometheus_replica
replacement: $1
action: labeldrop
alertmanagers:
- scheme: http
path_prefix: /
timeout: 10s
api_version: v1
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: alertmanager
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- default
rule_files:
- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml
scrape_configs:
- job_name: qke-generic-hubble-grafana-dashboard/hubble-grafana-dashboard-servicemonitor/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: hubble-grafana-dashboard
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: http
action: replace
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: hubble-grafana-dashboard
replacement: $1
action: keep
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: (grafana_stat_.*|grafana_.*_response_status_total|process_.*)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-grafana-dashboard
- job_name: qke-generic-hubble-manager/hubble-api-open-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-api-open
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-aiops-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-biz-aiops
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-cm-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-biz-cm
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-stat-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-biz-stat
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-third-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: m-biz-third
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-task-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-hubble-task
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-transfer-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-hubble-transfer
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: m-hubble-transfer
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: m-hubble-transfer
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/network-screen-sm/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /prometheus
scheme: http
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: s-network-screen
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-manager
- job_name: qke-generic-hubble-p-hbs/hubble-p-hbs-svcm/0
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /json2metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: hubble-p-hbs
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: exporter-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: exporter-metrics
action: replace
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: hubble_p_hbs_(.*)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-p-hbs
- job_name: qke-generic-hubble-p-query/hubble-p-query-svcm/0
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /json2metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: hubble-p-query
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: exporter-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: exporter-metrics
action: replace
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: hubble_p_query_(.*)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-p-query
- job_name: qke-generic-hubble-p-transfer/hubble-p-transfer-svcm/0
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /json2metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: hubble-p-transfer
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: exporter-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: exporter-metrics
action: replace
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: hubble_p_transfer_(.*)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-p-transfer
- job_name: qke-generic-hubble-platform/kube-state-metrics/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/k8s-auth/token
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: kube-state-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: https-main
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: https-main
action: replace
- separator: ;
regex: (pod|service|endpoint|namespace)
replacement: $1
action: labeldrop
metric_relabel_configs:
- source_labels: [namespace]
separator: ;
regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- monitoring
- job_name: qke-generic-hubble-platform/kubelet/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /metrics/cadvisor
scheme: https
bearer_token_file: /var/k8s-auth/token
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: kubelet
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: https-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: https-metrics
action: replace
- source_labels: [__metrics_path__]
separator: ;
regex: (.*)
target_label: metrics_path
replacement: $1
action: replace
metric_relabel_configs:
- source_labels: [namespace]
separator: ;
regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- kube-system
- job_name: qke-generic-hubble-self-monitor/hubble-p-transfer-svcm/0
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /json2metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: hubble-p-transfer
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: exporter-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: exporter-metrics
action: replace
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: hubble_p_transfer_(.*)
replacement: $1
action: keep
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: endpoints
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-activator/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: activator
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-activator
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_activator]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: knative-serving;activator;metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- knative-serving
- job_name: qke-generic-hubble-platform/knative-autoscaler/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: autoscaler
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-autoscaler
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_autoscaler]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: knative-serving;autoscaler;metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- knative-serving
- job_name: qke-generic-hubble-platform/knative-controller/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: controller
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-controller
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_controller_app]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: knative-serving;controller;metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- knative-serving
- job_name: qke-generic-hubble-platform/knative-queue-proxy/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-queue-proxy
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: .+;http-usermetric
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-platform
- qke-generic-hubble-p-updater-server
- qke-generic-hubble-grafana-dashboard
- qke-generic-hubble-p-transfer
- qke-generic-hubble-grafana-api
- qke-generic-hubble-aiops
- qke-generic-hubble-p-hbs
- qke-generic-hubble-p-query
- qke-generic-hubble-manager
- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-queue-proxy/1
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-queue-proxy
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: .+;http-autometric
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- qke-generic-hubble-platform
- qke-generic-hubble-p-updater-server
- qke-generic-hubble-grafana-dashboard
- qke-generic-hubble-p-transfer
- qke-generic-hubble-grafana-api
- qke-generic-hubble-aiops
- qke-generic-hubble-p-hbs
- qke-generic-hubble-p-query
- qke-generic-hubble-manager
- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-webhook/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: webhook
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: qke-generic-hubble-platform/knative-webhook
action: replace
- source_labels: [__meta_kubernetes_pod_label_knative_webhook]
separator: ;
regex: (.+)
target_label: job
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: knative-serving;webhook;metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
kubernetes_sd_configs:
- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
role: pod
bearer_token_file: /var/k8s-auth/token
tls_config:
ca_file: /var/k8s-auth/ca.crt
insecure_skip_verify: true
namespaces:
names:
- knative-serving全链路配置案例：
global:
evaluation_interval: 60s
external_labels:
prometheus_replica: $(POD_NAME)
scrape_interval: 60s
alerting:
alert_relabel_configs:
- action: labeldrop
regex: prometheus_replica
alertmanagers:
- scheme: http
static_configs:
- targets:
- alertmanager.prom-alert.svc:9093
rule_files:
- /etc/prometheus/rules/*rules.yaml
scrape_configs:
- job_name: 48e5a419-4721-5921-899c-86aa7122dfb6-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5eef17ab-f6b6-5d79-942b-ac41d35ba870-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-notice
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5f476f71-29c9-527e-b765-58b30c425751-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-event
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 79f80b60-0ced-55b5-a439-504b15a620ce-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-storm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 441b1a98-6c0c-5d8c-970d-425d2c1d412e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-transfer
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5b5d4de9-2c2e-579b-8854-8b4942b09e5e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: dfbfbb26-9632-5200-9b91-7e989c43969d-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: f6dc47d3-7e74-5ab4-8502-4b2fe9cb8123-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-cm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 24e69042-7557-5860-913e-9c7eeab76660-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-network-screen
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
remote_write:
- url: http://hubble.adapter.qiyi.domain:9988/prom2hubble/push?group=hubble-test
write_relabel_configs:
- source_labels: [__name__]
regex: trace_(.*)
action: keep
- source_labels: [project, app]
separator: ':'
target_label: hubble_endpoint
replacement: prometheus_$1
- source_labels: [project]
target_label: hubble_group
- regex: (app|project|prometheus_replica)
action: labeldrop
- target_label: hubble_step
replacement: "60"
name: trace
九、告警规则案例
QKE告警规则案例：
全链路告警规则案例：
groups:
- name: qytrace-agg.rules
rules:
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, span, zone, status_code)
record: trace_span_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, zone, status_code)
record: trace_service_requests_zone_code
- expr: |
avg by (project, app, zone, status_code) (sum by(instance, project, app, zone, status_code) (irate(http_server_requests_duration_seconds_count{env="prod"}[1m])))
record: trace_service_avg_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
record: trace_span_success_requests_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
record: trace_service_success_requests_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_success_rate_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_success_rate_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_avg_latency_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_avg_latency_zone
- expr: |
sum(trace_span_requests_zone_code) by (project, app, span, zone)
record: trace_span_requests_zone
- expr: |
sum(trace_service_requests_zone_code) by (project, app, zone)
record: trace_service_requests_zone
- expr: sum(jvm_gc_pause_seconds_count{env="prod"} - jvm_gc_pause_seconds_count{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_cnt_zone
- expr: sum(jvm_gc_pause_seconds_sum{env="prod"} - jvm_gc_pause_seconds_sum{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_elapsed_zone
- expr: |
sum(trace_service_requests_zone_code) by (app)
record: trace_service_requests_app
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_success_rate_app
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_avg_latency_app
samples-scraped-rules.yaml groups:
- name: samples-monitoring
rules:
- alert: SamplesScrapedTotal
expr: sum(scrape_samples_scraped{}) > 3000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "total samples in hubble-prod > 300w"
- alert: SamplesScrapedByJob
expr: sum by (job) (scrape_samples_scraped{}) > 1000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "samples from {{$labels.job}} in hubble-prod > 100w"
参考：
相关文章
- DjangoForm组件
- 前端之清除浮动的方式
- CSS
- jQuery中$.each()常见使用方法有哪些
- jQuery中text()方法如何获取/设置文本？
- javascript中事件是什么
- js获取dom节点的方法有哪些
- JavaScript 输出数据的方式有哪些
- JavaScript中jquery框架的特点
- js中jquery动画移动到最右边
- js中this指向有几种情况
- jQuery如何使用文档操作detach()方法
- Node.js 16.5.0正式发布，版本更新速看
- Vue框架在PostCSS中使用sass的方法
- JavaScript快速学习设计模式
- js new操作符的使用步骤
- js中co模块是什么
- 面试准备：快速学习JS防抖与节流
- http代理有哪些类别？
- 8月Web服务器排行榜出炉，Nginx崭露头角