Grafana监控java应用以及vCenter的方法

背景

最开始弄过vCenter的监控.
但是发现很多地方已经不合适了.
今天看了下jmx监控 java的应用. 顺便监控了下vCenter.
这里简单记录一下, 以便备忘.
需要注意的是 这里其实并不优秀 因为使用了agent 可能对产品有负面影响. 

jmx监控

思路:
利用 javaagent的方式暴露一个端口出来.
然后通过prometheus 收集暴露出来的数据
通过grafana进行展示. 

需要下载的文件主要有:
jmx_prometheus_javaagent-0.17.2.jar
prometheus-2.39.0-rc.0.linux-amd64.tar.gz
grafana-enterprise-9.1.6-1.x86_64.rpm

jmx监控准备

• 添加一个配置文件:

cat >simple-config.yml <<-EOF
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem"]
blacklistObjectNames: []
rules:
  - pattern: 'java.lang<type=OperatingSystem><>(committed_virtual_memory|free_physical_memory|free_swap_space|total_physical_memory|total_swap_space)_size:'
    name: os_$1_bytes
    type: GAUGE
    attrNameSnakeCase: true
  - pattern: 'java.lang<type=OperatingSystem><>((?!process_cpu_time)\w+):'
    name: os_$1
    type: GAUGE
    attrNameSnakeCase: true
EOF

启动脚本修改

在启动脚本的地方增加如下配置.
 -javaagent:./jmx_prometheus_javaagent-0.17.2.jar=8080:simple-config.yml

添加之后启动服务就可以了. 

修改prometheus

安装prometheus. 然后进行相应的配置
tar包安装非常简单, 建议直接在 /prometheus 目录下进行解压缩.
然后修改 prometheus.yaml 文件, 添加对应就可以了. 
我这边的配置文件如下: 
注意1: 需要记住 job_name. grafana里面需要进行设置.
注意2: instance: grafana里面可以进行区分. 

scrape_configs:
  - job_name: jmx
    static_configs:
    - targets: ['localhost:8080']
      labels:
        instance: xxx-server
    - targets: ['10.110.83.113:8080']
      labels:
        instance: xxx-dm

安装Grafana

rpm包安装grafana就可以了.
安装成功后:
systemctl enable  grafana-server.service
systemctl restart grafana-server.service
注意需要设置一下密码
默认密码是 admin/admin

配置Grafana

第一步: 增加数据源
点击左下角的齿轮状配置按钮.
add data source
选择prometheus就可以了.
注意可以讲本地IP地址设为白名单.
然后将本地IP地址添加进去.

第二步: 导入dashboard
点击左上角 四个正方形的按钮. 
browse 然后点击new 的下拉列表. 选择import
输入: 8563
就可以导入  JVM dashboard

效果图

vCenter的展示

跟之前写的一样, 这里面用到了influxdb以及telegraf
需要的文件主要有:

influxdb-1.8.9.x86_64.rpm
telegraf-1.19.2-1.x86_64.rpm

注意安装方式有是本地安装就可以.
这里吐槽一下OpenEuler:
第一: ESXi使用 6.7 More的兼容配置,无法读取安装介质.无法安装
第二: influxdb 安装完没有形成 influxdb 的配置文件的步骤.

influxdb的设置.

这里还是用的1.x
本着够用就好的方针..
安装完成之后 命令行输入 influx 就可以登录数据库
需要注意的是 需要先启动服务
systemctl restart influxdb
添加用户以及修改密码的命令:
CREATE USER "influxdb" WITH PASSWORD 'Test@xxxxxxx'
GRANT ALL PRIVILEGES TO influxdb

Telegraf 的设置

注意 Telegraf里面添加了配置之后会自动创建数据库. 
另外需要注意的是 要监控vCenter 可能需要两个Telegraf来运行
但是可以导入到一个influxdb的数据库里面. 
配置文件举例为:
/etc/telegraf/{telegraf01.conf,telegraf02.conf}
注意两个配置文件唯一不同的可能就是 vcenter的用户以及密码. 

需要注意 主要有input和output 两处设置. 
模板就是如下:

Telegraf模板配置文件

[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb]]
#这里需要修改。
urls = ["http://127.0.0.1:8086"]
database = "vm187"
timeout = "0s"
username = "zhaobsh"
password = "Test@xxxx"
[[inputs.vsphere]]
# 这里需要设置为密码
  vcenters = [ "https://10.110.xx.xx/sdk" ]
  username = "administrator@vsphere.local"
  password = "somepassword"
   vm_metric_include = [
     "cpu.demand.average",
     "cpu.idle.summation",
     "cpu.latency.average",
     "cpu.readiness.average",
     "cpu.ready.summation",
     "cpu.run.summation",
     "cpu.usagemhz.average",
     "cpu.used.summation",
     "cpu.wait.summation",
     "mem.active.average",
     "mem.granted.average",
     "mem.latency.average",
     "mem.swapin.average",
     "mem.swapinRate.average",
     "mem.swapout.average",
     "mem.swapoutRate.average",
     "mem.usage.average",
     "mem.vmmemctl.average",
     "net.bytesRx.average",
     "net.bytesTx.average",
     "net.droppedRx.summation",
     "net.droppedTx.summation",
     "net.usage.average",
     "power.power.average",
     "virtualDisk.numberReadAveraged.average",
     "virtualDisk.numberWriteAveraged.average",
     "virtualDisk.read.average",
     "virtualDisk.readOIO.latest",
     "virtualDisk.throughput.usage.average",
     "virtualDisk.totalReadLatency.average",
     "virtualDisk.totalWriteLatency.average",
     "virtualDisk.write.average",
     "virtualDisk.writeOIO.latest",
     "sys.uptime.latest",
   ]
   host_metric_include = [
     "cpu.coreUtilization.average",
     "cpu.costop.summation",
     "cpu.demand.average",
     "cpu.idle.summation",
     "cpu.latency.average",
     "cpu.readiness.average",
     "cpu.ready.summation",
     "cpu.swapwait.summation",
     "cpu.usage.average",
     "cpu.usagemhz.average",
     "cpu.used.summation",
     "cpu.utilization.average",
     "cpu.wait.summation",
     "disk.deviceReadLatency.average",
     "disk.deviceWriteLatency.average",
     "disk.kernelReadLatency.average",
     "disk.kernelWriteLatency.average",
     "disk.numberReadAveraged.average",
     "disk.numberWriteAveraged.average",
     "disk.read.average",
     "disk.totalReadLatency.average",
     "disk.totalWriteLatency.average",
     "disk.write.average",
     "mem.active.average",
     "mem.latency.average",
     "mem.state.latest",
     "mem.swapin.average",
     "mem.swapinRate.average",
     "mem.swapout.average",
     "mem.swapoutRate.average",
     "mem.totalCapacity.average",
     "mem.usage.average",
     "mem.vmmemctl.average",
     "net.bytesRx.average",
     "net.bytesTx.average",
     "net.droppedRx.summation",
     "net.droppedTx.summation",
     "net.errorsRx.summation",
     "net.errorsTx.summation",
     "net.usage.average",
     "power.power.average",
     "storageAdapter.numberReadAveraged.average",
     "storageAdapter.numberWriteAveraged.average",
     "storageAdapter.read.average",
     "storageAdapter.write.average",
     "sys.uptime.latest",
   ]
cluster_metric_include = [] 
datastore_metric_include = [] 
  datacenter_metric_include = []
  datacenter_metric_exclude = [ "*" ] 
insecure_skip_verify = true

手动启动telegraf

nohup telegraf -config /etc/telegraf/telegraf01.conf >/dev/null 2>1.txt & 
nohup telegraf -config /etc/telegraf/telegraf02.conf >/dev/null 2>2.txt & 

注意 手动启动就可以. 重启机器后需要重新启动. 

Grafana监控vCenter

第一步: 添加数据源:
注意选择influxdb
地址可以输入ip地址比如我这个:
http://10.110.136.70:8086
influxdb数据库选择 telegraf里面定义的那个
用户密码输入 创建的有权限的用户和密码就可以了. 

Grafana监控vCenter

导入grafana的配置项目主要有如下三个:
其他的好像需要更高的版本
名称                             编号
vSphere - Overview              12786
vSphere - Host details          12852
vSphere - VM details            12874

Grafana监控vCenter的简要效果

• 概览信息
  

• 宿主机的信息
  

• 虚拟机的信息