Zookeeper free Kafka
What are the benefits of abandoning Zookeeper?
ill it improve performance somehow?
I think the benefits come in several dimensions: Scalability, Reliability, and Performance.
Let’s explore-
Scalability: If we look at the scalability of Kafka, we should consider the complexity of operations. Currently, controller failover triggers an O(total partitions) operation in order to reload all of the states from zookeepers. Zookeeper removal provides an immediate path to having a standby controller.
Reliability: The Kafka/Zookeeper mix is prone to certain network partition failures. The typical one is when a broker can reach the zookeeper, but it cannot reach the controller. Currently, it is nevertheless considered “live.” in the post zookeeper world, the controller and zookeeper are the same entity, so this problem goes away.
There is a bunch more to add to this in terms of metadata management. The main benefit of having the raft implementation internal is that we get access to the log of events, this means all brokers see the same events and in the same order so some of the metadata inconsistencies that are possible with the older version, also go away.
Performance: This is also related to controller failover because we don’t have the metadata log, it is not possible for the controller to know what metadata has and has not been seen by the brokers, so it just sends all of it. Once we have the metadata log, then metadata updates can be incremental so even if there is a controller failover, we are only sending the deltas.
Whether removing the zookeeper(s) will cause an increased load on the brokers?
The things that the zookeeper was doing will be done by Kafka itself, so there will be increased load in that sense. In another sense, though, we hope to have a more efficient system that can do more metadata operations with the same number of nodes.
Will older Kafka clients still be usable once Zookeeper has been removed?
They will keep supporting older Kafka clients in the post zookeeper world. The initial alpha version may have limited support, but before KIP-500 is in production, they will support all the older clients.
What protocol is getting used for the Meta Data Quoram?
References —
1. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
2. https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum
