Streaming Response Disconnects Midway: Network Long-Connection Timeouts and Heartbeat Keepalive
The typewriter effect cuts out halfway and the connection drops? Comprehensively troubleshoot Nginx proxy buffering, SSE long-connection timeouts, and TCP Keepalive heartbeat configuration to ensure smooth interaction.
1. The Fragility of Streaming Long Connections
Server-Sent Events (SSE) typewriter-style streaming output relies on the client and server maintaining a continuous, long-lived TCP connection.
If traffic passes through Nginx or a reverse proxy server, the default proxy timeout (Proxy Read Timeout is usually 60s) can easily trigger a timeout and cut the connection while the model is thinking deeply.
2. Key Nginx Configuration Directives
The following directives must be added to the proxy configuration block:
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
proxy_set_header Connection '';
Completely disable proxy buffering to ensure every generated character chunk is pushed instantly to the frontend browser without obstruction.