In the AI era, which streaming solution stands out?
"TLDR: This article explores the necessity of streaming technology in the AI era, comparing and analyzing the technical implementations and applicable scenarios of three mainstream solutions: SSE, WebSocket, and gRPC. It focuses on the limitations of traditional HTTP requests in addressing LLM response latency, as well as the core value of server-side push mechanisms. By comparing the differences among these technologies in terms of protocol foundations, communication modes, performance overhead, and cognitive burden, it clarifies gRPC's performance advantages in Agent applications and its browser compatibility limitations."
Recently at work, I've been doing some Agent-related tasks that don't feel very useful. Although everyone else is busy, I've been relatively idle, just slacking off, reading docs, and learning. From my shallow observation, the actual work feels like manual labor—outputting existing knowledge and tool proficiency without much skill growth. It's the docs and sharing sessions that are the real input opportunities for knowledge growth.
Enough complaining—let me briefly review and recap streaming transmission in computer networking. Due to the decoder structure, all current LLM responses take an extremely long time per request, decoding word by word. If we used the traditional HTTP request method, there would likely be two outcomes: 1. Response timeout failure, or 2. After a long wait, the response succeeds and returns results, but the user has already uninstalled the app. Therefore, in the AI era, we need server-push technologies where the server tells the client whatever the LLM has generated so far—user experience matters most.
SSE used by ChatGPT
SSE is built on top of HTTP/1.1 and allows the server to send messages to the client one-way. In our most common HTTP/1.1 request headers, we write content-type: application/json, indicating the response data type is JSON. SSE, on the other hand, sets the request header to content-type: text/event-stream, meaning the server sends a server-side event to the client each time it has data ready. When the client sends an HTTP request to the server, it opens a long-lived HTTP connection and simply waits for the server's data.
WebSocket
This is the most common technology for bidirectional communication and push between server and client, also built on HTTP/1.1. Since HTTP/1.1 defaults to a request-response model, modifications are made on top of it—setting the request headers Connection: Upgrade and Upgrade: websocket—to implement the WebSocket protocol and enable mutual communication. However, it's somewhat more complex to implement, requiring manual maintenance of heartbeats, timeouts, and so on.
gRPC
gRPC is built on HTTP/2 and, like WebSocket, supports bidirectional communication. It's also an excellent streaming technology choice for building Agent applications. Additionally, since WebSocket requires a TCP three-way handshake and an upgrade on the HTTP connection, these multiple operations make WebSocket relatively heavy, with first-connection latency significantly higher than gRPC. Moreover, WebSocket requires manually maintaining heartbeats, sending events, receiving events, timeouts, etc., which places a heavier cognitive burden on developers. gRPC, thanks to HTTP/2's multiplexing and fewer TCP handshakes, offers better performance, lower latency, and greater ease of use. Its biggest drawback might be insufficient browser support—but for app clients, it's very well suited.