A brief introduction to message queue Kafka

Message queue is a widely used middleware today and has become a required question in almost every interview. Internet companies often use message queues to achieve "peak shaving" and decoupling complex businesses. As one of the most famous message queues, Kafka has excellent processing performance. Therefore, we need to have a deep understanding of how it works and how to use it.

Noun Literacy

• Topic: A grouping container in the message queue. Messages with the same Topic will be stored together. Topic can be compared to the classification of books in a library (science fiction theme, novel theme, historical theme)

• Partition: Each Topic can be divided into multiple Partitions, and Partitions can be stored across multiple servers, thereby improving throughput and fault tolerance. Partition can be compared to a bookshelf in a library. There are many books (analogized to news) under a certain topic (science fiction theme). Because one bookshelf cannot accommodate all the books under this topic, multiple independent bookshelves are needed for storage. The books stored inside the bookshelf are also orderly. Multiple Partitions also ensure reliability. Even if one Partition fails, other Partitions will continue to work.

• Broker: A node in the message queue cluster, responsible for accepting messages from producers and processing requests from consumers. Broker can be compared to different buildings in a library. Each building is independent of each other and has its own management system and administrator.

High availability

In order to ensure high availability, any large-scale system usually implements a backup strategy to avoid the risk of system interruption due to a single point of failure. Kafka enhances its reliability by implementing data redundancy - it will store multiple copies of partitions on different Brokers and maintain real-time synchronization of these copies. In this way, if the main Broker fails, the system can quickly switch to the backup Broker to ensure high availability of services.

Data backup will encounter the problem of rapid replication of a large number of messages, which is a common IO bottleneck in computers. There are similar solution mechanisms in other applications such as MySQL and Redis, such as the binlog in MySQL and the AOF log of Redis, mainly to provide data backup capabilities.

High reliability

The message queue needs to ensure that the message will not be lost during the transmission process, mainly in three stages: the message sent by the producer to Kafka cannot be lost, the message accumulated in Kafka cannot be lost, and the message sent to the consumer cannot be lost.

1. Producer sending stage

   The producer sends a message to Kafka, and Kafka will return an ACK after successfully accepting it. Therefore, as long as the producer receives the ACK, it means that the message was sent successfully.

2. Broker storage stage

   The Broker periodically flushes a set of messages asynchronously to disk for persistence.

3. consumer consumption stage

   The consumer consumes messages from the Broker and sends status to the Broker once the consumption is successful. However, due to network problems, inevitable repeated consumption problems may occur, so consumers need to achieve consumption idempotence.

Kafka is a typical IO-intensive application, and a lot of technical details are devoted to solving the IO performance bottleneck. Similar applications such as mysql and redis have adopted similar solutions. Binlog in MySQL implements asynchronous disk flushing, AOF log in redis, and even zero copy in the kernel

High performance

In order to achieve efficient and fast sending and receiving of messages, Kafka classifies messages based on Topic and distributes them in different queues, reducing the processing pressure of a single queue. The message queue processing capacity of a single Topic is still limited, so the messages can be split into different Partitions, and different consumers consume messages in different Partitions, which greatly reduces the processing pressure.

The above-mentioned high-performance solutions are all based on the idea of ​​"split". Since a single point has limited processing power, splitting the single point content into different smaller parts can reduce stress. This is reflected at two levels: splitting a single queue into queues of different topics, and splitting a single topic queue into different partitions. From an analogy perspective, this is in the same vein as the library metaphor mentioned earlier.

The splitting strategy is common in all message queues (such as RocketMQ, Kafka) ("People have it, I have it"). However, Kafka has other unique performance optimization measures compared to RocketMQ ("No one has it, I have it"), which can achieve data processing speeds of up to 17w/s. Zero copy is one of the important performance optimization measures.

During the sending process of messages, four steps generally occur:

1. (read call) disk data is read into the kernel buffer

2. (read returns) The data in the kernel buffer is copied to user space.

3. (write call) Copies user space data to the socket buffer

4. (write returns) The socket send buffer is sent to the network card

A total of 4 switchings between user space and kernel space and 4 data copies occurred here. It can be found that the efficiency of copying the same data back and forth is very low, so the operating system kernel provides two optimization solutions: mmap and sendfile.

• mmap: Maps kernel space to user space so that data copying from kernel space to user space does not occur. Through mmap optimization, only 2 system calls (mmap, write), 4 switching between user space and kernel space, and 3 data copies occur in the entire process. The "zero copy" of mmap means to avoid a copy from kernel space to user space.

• sendfile: It is also an optimization solution provided by the kernel. It can directly copy disk data to the kernel buffer, and then copy from the kernel buffer to the network card. The entire process only requires one system call (sendfile), two switching between user space and kernel (sendfile call and return), and two data copies (disk → kernel buffer → network card). "Zero copy" of sendfile refers to zero CPU copy. Both data copies are handled by DMA and the CPU does not interfere.

Back to the message queue, RocketMQ uses mmap technology, while Kafka uses sendfile technology, so the performance is better.

Although the performance of mmap is slightly worse, compared to sendfile, which has no CPU involvement at all, mmap can know the specific content of the message sent by the network card, which provides the basis for some functions of RocketMQ, so RocketMQ is more powerful.

High scalability

Scalability provides the system with the ability to adapt to different traffic flows, and traffic peaks can be alleviated by temporarily adding machines. Kafka also has this elastic scaling capability, which can distribute Partitions to newly added Brokers, thereby reducing the processing load of a single machine.