kafka2.3性能测试:Kafka 2.3 Performance testing
2023-09-11 14:18:40 时间
BOOTSTRAP=10.0.0.4:9092,10.0.0.5:9092,10.0.0.6:9092
First we need to create the various topics, depending on the partition count and replicas:
kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-rep-one --partitions 6 --replication-factor 1 kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-rep-three --partitions 6 --replication-factor 3 kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-7k --partitions 18 --replication-factor 3
Kafka ships with two handy scripts you can use to test your cluster, kafka-producer-perf-test and kafka-consumer-perf-test
Test results
Single producer, no consumer, no replication | 1 628 081 msg/sec (155.27 MB/sec) |
Single producer, no consumer, 3x async replication | 1 463 136 msg/sec (140.46 MB/sec) |
Single producer, no consumer, 3x syncronous replication | 1 226 439 msg/sec (125.11 MB/sec) |
3 producers, no consumer, 3 asynchronous replication | 3 960 110 msg/sec (377.69 MB/sec) |
No producer, single consumer | 4 096 100 msg/sec (390.63 MB/sec) |
No producer, three consumers | 11 813 321 msg/sec (1125 MB/sec) |
At this point we've duplicated the testing with the original tests done, hopefully with much improved numbers.
However, as the tests were done with 100 byte records, the tests were re-run with 7KB records and optimised Kafka settings (larger Heap size at 8GB, larger batch sizes and some snappy compression applied).
Optimised Kafka results, 7k records
6 producers, no consumers, 3x async replication (larger batch sizes, snappy compression) | 1 070 970 msg/sec (7321 MB/sec) |
0 producers, 6 consumers | 963 071 msg/sec (6896 MB/sec) |
Over 1 million messages a second, reading and writing 7kb per message. We've reached the networking limit!
The commands used for each test if you would like to reproduce yourself:
Test 1: kafka-producer-perf-test --topic test-rep-one --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} Test 2: kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} Test 3: kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=1 bootstrap.servers=${BOOTSTRAP} Test 4 (run three instances in parallel, one on each node): kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} Test 5: kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-rep-three --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 6 Test 6 (run three instances in parallel, one on each node): kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-rep-three --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 6 Test 7 (run a producer on each node, including the Kafka brokers): kafka-producer-perf-test --topic test-7k --num-records 50000000 --record-size 7168 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} linger.ms=100 compression.type=snappy Test 8 (run a consumer on each node, including the Kafka brokers): kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-7k --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 18
相关文章
- Kafka源码分析2:Kafka产品选择和Kafka版本选择(史上最全)
- Python 操作 Kafka --- kafka-python
- 使用Python进行 kafka的生产与消费
- Kafka-为什么选择kafka(kafka的优点)
- Kafka-常用术语(消息、生产者、消费者、集群、broker解释)
- 获取Kafka每个分区最新Offset的几种方法
- 淘东电商项目(49) -ELK+Kafka分布式日志收集(docker下搭建kafka)
- 淘东电商项目(48) -ELK+Kafka分布式日志收集(原理篇)
- 128 Kafka消息的分发
- Apache Kafka源码分析 - KafkaApis
- 腾讯云Kafka海量服务自动化运营实践
- Spring Kafka中通过参数配置解决超时问题
- Kafka入门
- 【kafka运维】Kafka全网最全最详细运维命令合集(精品强烈建议保存)
- 7.【kafka运维】 kafka-consumer-groups.sh消费者组管理
- 【kafka原理】Kafka生产者 (分区策略和ACK应答机制)
- 《kafka问答100例 -8》 如果写入`/brokers/topics/{TopicName}`节点之后Controller挂掉了会怎么样|文末送书
- 【Kafka】Windows环境配置测试
- kafka可视化客户端工具(Kafka Tool)的基本使用
- Kafka(二)CentOS7.5搭建Kafka2.11-1.1.0集群与简单测试
- Benchmarking Apache Kafka, Apache Pulsar, and RabbitMQ: Which is the Fastest?
- kafka生产者性能监控:Monitor Kafka Producer for Performance
- 如何使用JMeter测试Kafka:Apache Kafka - How to Load Test with JMeter
- Linkedin官方kafka性能压测-kafkaBenchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
- kafka集群安全化之启用kerberos与acl
- Kafka监控系统Kafka Eagle:支持kerberos认证
- Kafka vs RocketMQ——多Topic对性能稳定性的影响-转自阿里中间件
- 2023年java面试题之kafka(5道)
- kafka性能测试1.0.0