kafka生产者性能监控:Monitor Kafka Producer for Performance
Introduction
The producer sends data directly to the broker that is the leader for the partition without any intervening routing tier.
Optimization Approach
Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed latency bound (say 64k or 10 ms). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. This buffering is configurable and gives a mechanism to trade off a small amount of additional latency for better throughput.
In order to find the optimal batch size and latency, iterative test supported by producer statistics monitoring is needed.
Enable Monitoring
Start the producer with the JMX parameters enabled:
JMX_PORT=10102 bin/kafka-console-producer.sh --broker-list localhost:9092--topic testtopic
Producer Metrics
Use jconsole application via JMX at port number 10102.
Tip: run jconsole application remotely to avoid impact on broker machine.
See metrics in MBeans tab.
The<strong>clientId</strong>
parameter is the producer client ID for which you want the statistics.
<strong>kafka.producer:type=ProducerRequestMetrics,name=ProducerRequestRateAndTimeMs,clientId=console-producer</strong>
This MBean give values for the rate of producer requests taking place as well as latencies involved in that process. It gives latencies as a mean, the 50th, 75th, 95th, 98th, 99th, and 99.9thlatency percentiles. It also gives the time taken to produce the data as a mean, one minute average, five minute average, and fifteen minute average. It gives the count as well.
<strong>kafka.producer:type=ProducerRequestMetrics,name=ProducerRequestSize,clientId=console-producer</strong>
This MBean gives the request size for the producer. It gives the count, mean, max, min, standard deviation, and the 50th, 75th, 95th, 98th, 99th, and 99.9thpercentile of request sizes.
<strong>kafka.producer:type=ProducerStats,name=FailedSendsPerSec,clientId=console-producer</strong>
This gives the number of failed sends per second. It gives this value of counts, the mean rate, one minute average, five minute average, and fifteen minute average value for the failed requests per second.
<strong>kafka.producer:type=ProducerStats,name=SerializationErrorsPerSec,clientId=console-producer</strong>
This gives the number of serialization errors per second. It gives this value of counts, mean rate, one minute average, five minute average, and fifteen minute average value for the serialization errors per second.
<strong>kafka.producer:type=ProducerTopicMetrics,name=MessagesPerSec,clientId=console-producer</strong>
This gives the number of messages produced per second. It gives this value of counts, mean rate, one-minute average, five-minute average, and fifteen-minute average for the messages produced per second.
References
相关文章
- Python 操作 Kafka --- kafka-python
- Hudi-Flink SQL实时读取kafka数据写入Hudi表
- 《Kafka官方文档》主页
- kafka监控系统
- 如何监控开源 Apache Kafka?
- Kafka - kafka环境配置以及参数分析
- Kafka - kafka生产者 消费者 监听配置属性
- Kafka监控工具KafkaOffsetMonitor配置及使用
- 【收藏】什么是Kafka、原理、作用、架构?
- Kafka如何修改分区Leader | 文末送书8本
- 【kafka运维】Leader重新选举运维脚本
- 3.【kafka运维】Topic的生产和消费运维脚本
- Kafka的灵魂伴侣Logi-KafkaManger(2)之kafka针对Topic粒度的配额管理(限流)
- Kafka集群管理工具kafka-manager的安装使用
- Kafka(五)Kafka的API操作和拦截器
- kafka最佳实践:Kafka Best Practices
- ranger kafka - Authorizing Kafka access over non-authenticated channel via Ranger
- KSQL: Streaming SQL for Apache Kafka
- kafka-rest:A Comprehensive, Open Source REST Proxy for Kafka
- Flafka: Apache Flume Meets Apache Kafka for Event Processing
- Spark Streaming和Kafka整合保证数据零丢失
- 原理解析 | Apache Flink 结合 Kafka 构建端到端的 Exactly-Once 处理
- Kafka运维大全来了!优化、监控、故障处理
- scribe、chukwa、kafka、flume日志系统对比 -摘自网络