Custom partition assignment and migration kafka集群扩充迁移指定partition
The partition reassignment tool can also be used to selectively move replicas of a partition to a specific set of brokers. When used in this manner, it is assumed that the user knows the reassignment plan and does not require the tool to generate a candidate reassignment, effectively skipping the --generate step and moving straight to the --execute step
For instance, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3:
The first step is to hand craft the custom reassignment plan in a json file:
1
2
|
> cat custom-reassignment.json { "version" :1, "partitions" :[{ "topic" : "foo1" , "partition" :0, "replicas" :[5,6]},{ "topic" : "foo2" , "partition" :1, "replicas" :[2,3]}]} |
Then, use the json file with the --execute option to start the reassignment process:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
> bin /kafka-reassign-partitions .sh --zookeeper localhost:2181 --reassignment-json- file custom-reassignment.json --execute Current partition replica assignment { "version" :1, "partitions" :[{ "topic" : "foo1" , "partition" :0, "replicas" :[1,2]}, { "topic" : "foo2" , "partition" :1, "replicas" :[3,4]}] } Save this to use as the --reassignment-json- file option during rollback Successfully started reassignment of partitions { "version" :1, "partitions" :[{ "topic" : "foo1" , "partition" :0, "replicas" :[5,6]}, { "topic" : "foo2" , "partition" :1, "replicas" :[2,3]}] } |
The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option:
1
2
3
4
|
> bin /kafka-reassign-partitions .sh --zookeeper localhost:2181 --reassignment-json- file custom-reassignment.json --verify Status of partition reassignment: Reassignment of partition [foo1,0] completed successfully Reassignment of partition [foo2,1] completed successfully |
相关文章
- 数据源管理 | Kafka集群环境搭建,消息存储机制详解
- 淘东电商项目(50) -ELK+Kafka分布式日志收集(实现篇)
- Apache Kafka源码分析 - kafka controller
- 如何判断一个kafka集群是否稳定
- Kafka_Kafka 消费者 偏移量 与 积压 查询脚本 kafka-consumer-groups.sh
- Kafka - kafka环境配置以及参数分析
- Kafka启动遇到ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
- 【kafka源码】分区副本重分配之kafka跨目录数据迁移实现源码解析
- 【kafka运维】分区副本重分配、数据迁移、副本扩缩容 (附教学视频)
- Kafka的灵魂伴侣Logi-KafkaManger(4)之运维管控–集群运维(数据迁移和集群在线升级)
- Kafka的灵魂伴侣Logi-KafkaManger(2)之kafka针对Topic粒度的配额管理(限流)
- kafka可视化客户端工具(Kafka Tool)的基本使用
- ranger kafka - Authorizing Kafka access over non-authenticated channel via Ranger
- Linkedin官方kafka性能压测-kafkaBenchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
- Problem of Creating Topics in Kafka with Kerberos
- kafka-connect-hdfs连接hadoop hdfs时候,竟然是单点的,太可怕了。。。果断改成HA
- 为什么kafka使用磁盘而不是内存
- kubernetes statefulset kafka 部署后, 外部访问超时问题解决
- 基于Rsyslog+Kafka+Flink+Clickhouse的高性能日志采集、分析、汇聚存储方法
- kafka基础概念
- 【大数据】Spark读取Hive/Hbase/Elasticsearch/Kafka/Mysql等数据源
- RabbitMQ和kafka从几个角度简单的对比--转
- 【Kafka】Apache Kafka消息队列Topic管理