using kafkacat reset kafka offset
Kafka Using reset offset
2023-09-27 14:27:40 时间
1. install kafkacat
Ubuntu
apt-get install kafkacat
CentOS
install deepenency
yum install librdkafka-devel
download source from github
build source on centos
./configure <usual-configure-options> make sudo make install
2. watch the target topic data
lenmom@M1701:~/workspace/software/confluent-community-5.1.0-2.11$ bin/kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic connect-offsets --property print.key=true ["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":1005}
there is only one record in kafka topic connect-offsets.
3. dump the record from topic
lenmom@M1701:~/workspace/software/confluent-community-5.1.0-2.11$ kafkacat -b localhost:9092 -t connect-offsets -C -K# -o-1 % Reached end of topic connect-offsets [0] at offset 0 % Reached end of topic connect-offsets [5] at offset 0 % Reached end of topic connect-offsets [10] at offset 0 % Reached end of topic connect-offsets [20] at offset 0 % Reached end of topic connect-offsets [15] at offset 0 % Reached end of topic connect-offsets [9] at offset 0 % Reached end of topic connect-offsets [11] at offset 0 % Reached end of topic connect-offsets [4] at offset 0 % Reached end of topic connect-offsets [16] at offset 0 % Reached end of topic connect-offsets [17] at offset 0 % Reached end of topic connect-offsets [3] at offset 0 % Reached end of topic connect-offsets [24] at offset 0 % Reached end of topic connect-offsets [23] at offset 0 % Reached end of topic connect-offsets [13] at offset 0 % Reached end of topic connect-offsets [18] at offset 0 ["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":1005} % Reached end of topic connect-offsets [8] at offset 0 % Reached end of topic connect-offsets [2] at offset 0 % Reached end of topic connect-offsets [12] at offset 0 % Reached end of topic connect-offsets [19] at offset 0 % Reached end of topic connect-offsets [14] at offset 0 % Reached end of topic connect-offsets [1] at offset 0 % Reached end of topic connect-offsets [6] at offset 0 % Reached end of topic connect-offsets [7] at offset 0 % Reached end of topic connect-offsets [21] at offset 0 % Reached end of topic connect-offsets [22] at offset 1
the value:
["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":1005}
is what we want!
4. use the value get in step 3 as template to and send to the topic again
lenmom@M1701:~/workspace/software/confluent-community-5.1.0-2.11$ echo '["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":1}' | \ > kafkacat -b localhost:9092 -t connect-offsets -P -Z -K#
here, we modify the incrementing value from 1005 to 1.
for timestamp+increment
echo '["jdbc_source_inventory_orders",{"query":"query"}]#{"timestamp_nanos":0,"incrementing":0,"timestamp":0}' | \ kafkacat -b localhost:9092 -t connect-offsets -P -Z -K#
5. watch the topic again
lenmom@M1701:~/workspace/software/confluent-community-5.1.0-2.11$ bin/kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic connect-offsets --property print.key=true ["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":1005} ["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":1}
we can see, there are two values with the same key in the topic now.
refernce
https://docs.confluent.io/current/app-development/kafkacat-usage.html
相关文章
- 《KAFKA官方文档》设计与实现(一)
- 《KAFKA官方文档》第三章:快速入门(二)
- Apache Kafka源码分析 – Broker Server
- Apache Kafka源码分析 - ReplicaStateMachine
- flume+kafka+SparkStreaming+mysql+ssm+高德地图热力图项目
- Kafka
- 图解Kafka的RecordBatch结构
- 【kafka】常用命令汇总
- 《kafka问答100例 -4》 如果我手动在zk中添加/brokers/topics/{TopicName}节点会怎么样?
- Linkedin官方kafka性能压测-kafkaBenchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
- kafka Authentication using SASL/Kerberos
- Kafka分区分配策略-RangeAssignor、RoundRobinAssignor、StickyAssignor
- kafka broker Leader -1引起spark Streaming不能消费的故障解决方法
- kafka controller脑裂(多个controller)问题
- kafka 幂等生产者及事务(kafka0.11之后版本新特性)
- Spark+Kafka的Direct方式将偏移量发送到Zookeeper实现(转)
- 传统消息系统的不孝子嗣mqtt和kafka
- Kafka的架构原理,你真的理解吗?