High SLA Kafka - Kafka Across Multiple DCs Kafka Internals Control Plane and Data Plane Kafka Cluster Broker Broker Data Plane Broker Broker Controller Broker Broker Zookeeper Zookeeper Zookeeper Control Plane ● Cluster Membership ● Broker and Controller ● Topic Partition and Partition Leader ● Access Control List Inside The Apache Kafka Broker Broker Broker Broker Broker Broker Broker Network Thread Adds Request to Queue IO Thread Verifies Record Batch And Stores Kafka Physical Storage /var/lib/kafka/data/account-deposits-1 00000000000047926734.log 00000000000047926734.index ... 00000000000052497535.log 00000000000052497535.index ... Purgatory Holds Requests Being Replicated Response Added to Socket Send Buffer The Nature of Multi DC and Availability RPO and RTO Recovery Point Objective (RPO) Recovery Time Objective (RTO) ZERO RPO requires synchronous replication ZERO RTO requires seamless failover Stretch Cluster and Replication ZK ZK ZK Tokyo ZK ZK Osaka ZK ZK ZK Tokyo ZK ZK ZK Osaka High Availability and Disaster Recovery Active-Active - 2DC Active-Warm - DC に Active-Passive - DR ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK ZK DC Failure and Split Brain ZK ZK ZK Tokyo ZK ZK Osaka ZK ZK ZK Tokyo ZK ZK Osaka Replication Kafka Cluster B Kafka Cluster A Broker Broker L p0 Broker F p0 Broker F p0 F p0 F p0 L p0 Broker Broker Replication (MM2, uReplicator, Confluent Replicator) leader L follower F Tokyo Osaka ● Consumer/Producer ● Offset Translation ● Consumer Rag = Replication Rag Confluent Cluster Linking Destination Cluster Broker Source Cluster Broker L p0 Broker F p0 Broker F p0 F p0 F p0 L p0 Broker Broker Broker Broker leader L follower F Tokyo Osaka ● Broker (Replica Fetcher) ● Offset Preservation Replication Patterns Tokyo Osaka tokyo-topic osaka-topic osaka-topic-mirror tokyo-topic-mirror Tokyo Osaka tokyo-topic osaka-topic osaka-topic-mirror topic-aggregated ● RPO / RTO ● 200% ● For OLTP ● RPO / RTO ● 100% - 200% ● For OLAP Challenges Stretch Cluster