Kafka 0.8
0.8 is a huge step forward in functionality from 0.7.x
This release includes the following major features:
- Partitions are now replicated. 支持partition的复本, 避免broker失败导致的数据丢失
Previously the topic would remain available in the case of server failure, but individual partitions within that topic could disappear when the server hosting them stopped. If a broker failed permanently any unconsumed data it hosted would be lost.
Starting with 0.8 all partitions have a replication factor and we get the prior behavior as the special case where replication factor = 1.
Replicas have a notion of committed messages and guarantee that committed messages won't be lost as long as at least one replica survives. Replica logs are byte-for-byte identical across replicas. - Producer and consumer are replication aware. 支持replica的Producer和Consumer
When running in sync mode, by default, the producer send() request blocks until the messages sent is committed to the active replicas. As a result the sender can depend on the guarantee that a message sent will not be lost.
Latency sensitive producers have the option to tune this to block only on the write to the leader broker or to run completely async if they are willing to forsake this guarantee.
The consumer will only see messages that have been committed. - The consumer has been moved to a "long poll" model where fetch requests block until there is data available.
This enables low latency without frequent polling. In general end-to-end message latency from producer to broker to consumer of only a few milliseconds is now possible. - We now retain the key used in the producer for partitioning with each message, so the consumer knows the partitioning key.
会保存producer用于partitioning的key, 并让consumer知道这个key - We have moved from directly addressing messages with a byte offset to using a logical offset (i.e. 0, 1, 2, 3...). 使用逻辑offset代替之前的物理offset
The offset still works exactly the same - it is a monotonically increasing number that represents a point-in-time in the log - but now it is no longer tied to byte layout.
This has several advantages:
(1) it is aesthetically (美学观点上地) nice,
(2) it makes it trivial to calculate the next offset or to traverse messages in reverse order,
(3) it fixes a corner case (极端情况) interaction between consumer commit() and compressed message batches. Data is still transferred using the same efficient zero-copy mechanism as before. - We have removed the zookeeper dependency from the producer and replaced it with a simple cluster metadata api.
- We now support multiple data directories (i.e. a JBOD setup).
- We now expose both the partition and the offset for each message in the high-level consumer.
在high-level consumer中expose具体的partition和offset信息 - We have substantially improved our integration testing, adding a new integration test framework and over 100 distributed regression and performance test scenarios that we run on every checkin.
在我看来, 主要的改动
1. 增加broker的安全性, 原来的方案, broker的fail就会导致数据丢失, 确实有点太说不过去, 所以replica feature是必须的
2. 使用逻辑offset, 上面说了些优点, 但是之前使用物理offset时, 也说了一堆优点
其实就是效率和易用性的balance, 之前出于对效率的追求, 所以使用物理offset
而现在考虑到物理offset实在用的太麻烦, 做出妥协, 改为逻辑offset, 本质没有区别, 只是需要增加一个逻辑offset到物理offset的映射, 以使物理offset对用户透明
3. 对python更好的支持, kafka-python
Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python
Kafka Replication High-level Design
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication
参考,Apache Kafka Replication Design – High level
本文章摘自博客园,原文发布日期:2013-05-08
相关文章
- std::string的Copy-on-Write:不如想象中美好
- 计算机架构的新黄金时代,为什么到了2021年还没有开始
- React库+GraphQL服务器+Relay架构联合作战(上)
- 「React进阶」一文吃透react事件原理
- 如何做 Nginx 安全日志分析可视化,看这一篇就够了
- 和阿里P8大佬面试互怼了半小时的Fork/Join原理
- 有了CopyOnWrite为何又要有ReadWriteLock?
- Github标星7.9K!程序员专属的命名宝典来了
- 工作六年后,对软件开发的一些新观点
- 如何在.NET Core中为gRPC服务设计消息文件(Proto)
- PHP继承竟然也需要显性基因?
- PHP弱类型安全问题总结
- vscode-icons: 提高生产力的 Visual Studio Code插件
- 字符串离奇消失之谜
- 编写简洁的React代码建议
- 影响了一代代前端人的 20 个里程碑式的顶级开源项目
- 程序员每天都在使用的6个惊讶的软技能
- 用 Go 搭建一个自己的照片管理神器
- 提高开发质量和速度的四个软件质量保证指标
- 软件开发者,你为什么会讨厌用户?