TCP Flow and Congestion Control

Two different "slow down" signals

A TCP sender has to answer one question continuously: how much data may I have in flight right now? Two different things can limit it.

• The receiver. Its application may read slowly, so its buffer fills up. If the sender keeps going, data is dropped at the receiver. Preventing this is flow control.
• The network. Routers along the path have finite link capacity and buffers. If every sender pushes as fast as it can, queues overflow, packets drop, senders retransmit, and useful throughput collapses. Preventing this is congestion control.

TCP handles both with windows. The sender may have at most this much unacknowledged data outstanding:

effective window = min(rwnd, cwnd) - (bytes in flight)

• rwnd (receive window) is advertised by the receiver in every segment's window field. It protects the receiver.
• cwnd (congestion window) is a variable kept privately by the sender. It is the sender's estimate of what the network can take. Nobody tells the sender this value; it is inferred from ACKs, losses and delay.

This lesson works through both mechanisms. You will see sliding windows and zero windows, silly window syndrome and Nagle's algorithm, how TCP estimates RTT and sets its retransmission timeout, and how slow start, congestion avoidance, fast retransmit and fast recovery shape cwnd over time, with a round-by-round trace. Then you will compare Tahoe, Reno, CUBIC and BBR, look at fairness and head-of-line blocking, and finish by writing and running a TCP echo server in Python.

Interviewers usually probe: the difference between flow and congestion control, drawing cwnd over time, what happens on timeout versus three duplicate ACKs, why the RTO formula uses variance, and Nagle versus delayed ACK. The basics of TCP (handshake, sequence numbers, Go-Back-N and Selective Repeat) are in the transport layer lesson.

Flow control: the sliding window

The receive buffer

Arriving bytes go into the receiver's receive buffer (in the kernel) until the application reads them. The free space is:

rwnd = RcvBuffer - (LastByteRcvd - LastByteRead)

The receiver puts rwnd in the 16-bit window field of every segment it sends. When the application stops reading, rwnd shrinks; when it reads, rwnd grows again.

The sender's view

The sender's byte stream divides into four regions:

          LastByteAcked          LastByteSent
                |                      |
 ... acked ... | sent, not acked      | can send now | cannot send yet
 --------------+----------------------+--------------+----------------
               |<-------- window (min of rwnd, cwnd) -------->|

• Bytes left of the window are acknowledged and can be freed.
• Bytes inside the window that have been sent wait for ACKs.
• Bytes inside the window not yet sent may be sent immediately.
• Bytes right of the window must wait.

When an ACK arrives, the left edge moves right (the window slides). When the receiver advertises a larger rwnd, the right edge moves right (the window opens). The rule is that the sender keeps LastByteSent - LastByteAcked at or below the window.

Worked example: rwnd changing

The receiver has a 4,000-byte buffer. MSS is 1,000 bytes.

At step 2 the window is zero: the sender must stop.

Zero window and persist timer

After a zero window, the receiver will announce the new space in a window update when the application reads. But that update is a pure ACK with no data, and TCP does not retransmit pure ACKs. If it is lost, the sender waits for an update and the receiver waits for data: a deadlock.

TCP breaks it with the persist timer. While the advertised window is zero, the sender periodically sends a zero-window probe, a segment with one byte of data (or a byte already sent). The receiver must answer with an ACK carrying its current window. If still zero, the sender backs off the probe interval exponentially; once the window opens, sending resumes. Probes continue indefinitely as long as the receiver keeps answering.

Window scaling

The window field is 16 bits, so it can advertise at most 65,535 bytes. On a path with a large bandwidth-delay product this caps throughput (65,535 bytes per RTT is about 5.2 Mbps at 100 ms RTT; see networking basics).

The window scale option (RFC 7323), sent only in the SYN segments, gives a shift count S from 0 to 14. The real window is the field value shifted left by S (multiplied by 2^S), up to about 1 GB.

Worked example. To fill a 1 Gbps path with 50 ms RTT, the window must reach the BDP: 10^9 × 0.05 / 8 = 6,250,000 bytes. That is 6,250,000 / 65,535 ≈ 95.4 times the unscaled maximum. The smallest power of two at least 95.4 is 128 = 2^7, so a scale factor of S = 7 is needed. Modern operating systems negotiate scaling and auto-tune buffer sizes automatically.

Silly window syndrome

Silly window syndrome is when TCP ends up sending a steady stream of tiny segments, each carrying a few bytes of data under 40 bytes of headers. It can be caused by either side.

Receiver-caused. The application reads one byte at a time from a full buffer. After each read the receiver advertises a window of 1 byte, the sender dutifully sends 1 byte, the buffer is full again, and so on.

• Fix (Clark's solution): the receiver does not advertise a small window. It keeps advertising zero until it can offer at least one MSS or half its buffer, whichever is smaller.

Sender-caused. The application writes one byte at a time (for example, a keystroke at a time), and TCP sends each byte at once.

• Fix: Nagle's algorithm (below).

silly:   [hdr 40 B][1 B] [hdr 40 B][1 B] [hdr 40 B][1 B] ...
            efficiency = 1 / 41, about 2.4%
fixed:   [hdr 40 B][1460 B]                ...
            efficiency = 1460 / 1500, about 97%

Nagle's algorithm

Nagle's rule (RFC 896): if there is unacknowledged data in flight, buffer small writes until either an ACK arrives or a full MSS is collected. If nothing is in flight, send immediately.

So a stream of small writes becomes one small segment per RTT, with everything typed in the meantime batched behind it. This is self-clocking: on a fast LAN, ACKs return quickly and little is delayed; on a slow WAN, batching saves a lot.

Delayed ACK

The receiver side has its own optimisation: delayed ACK (RFC 1122). Instead of acknowledging every segment at once, the receiver waits a short time, hoping to piggyback the ACK on response data or to acknowledge two segments with one ACK. The rules: send an ACK for at least every second full-size segment, and never delay more than 500 ms. Typical implementations delay up to 40 ms (Linux, adaptive) or 200 ms (Windows default).

When Nagle and delayed ACK collide

Together they can cause a noticeable stall. Consider a client that sends a request in two writes, a small header then a small body, and waits for the response:

1. The first write goes out immediately (nothing was in flight).
2. The second write is small and data is unacknowledged, so Nagle holds it until an ACK arrives.
3. The server received only the header, so it has nothing to reply with yet; it delays its ACK hoping for data to piggyback.
4. Both wait until the delayed-ACK timer fires (up to 40 or 200 ms). Then the ACK releases the body.

 Client                                 Server
   |-- write 1: header ------------------->|
   |   write 2: body (held by Nagle)       | (waits to piggyback ACK)
   |          ... up to 40-200 ms ...      |
   |<----------------------------- ACK ----| (delayed-ACK timer fires)
   |-- body ------------------------------>|
   |<------------------------- response ---|

Fixes: combine writes into one (build the whole request first, or use writev), or set TCP_NODELAY to disable Nagle. Latency-sensitive software (interactive protocols, RPC frameworks, games, many database drivers) commonly sets TCP_NODELAY. Linux also offers TCP_CORK and TCP_QUICKACK for finer control.

RTT estimation and the retransmission timeout

When TCP sends a segment it starts a retransmission timer. If no ACK arrives before the retransmission timeout (RTO), it resends. Choosing the RTO is delicate:

• Too short: spurious retransmissions of segments that were only delayed, wasting bandwidth and confusing congestion control.
• Too long: slow recovery from real losses.

The RTO must therefore track both the average RTT and how much it varies.

Sample RTT

TCP measures SampleRTT: the time from sending a segment to receiving the ACK that covers it. Samples bounce around as queues change, so TCP smooths them.

Exponentially weighted moving average

The classic smoothed estimate:

EstimatedRTT = (1 - alpha) × EstimatedRTT + alpha × SampleRTT     (alpha = 1/8)

Each new sample counts for 1/8; history counts for 7/8, and older samples fade away geometrically. Original TCP set RTO = 2 × EstimatedRTT, which failed badly when RTT varied a lot.

Jacobson's algorithm (RFC 6298)

Van Jacobson's 1988 fix also tracks the mean deviation of RTT and pads the timeout by four times it. In RFC 6298 names, SRTT is the smoothed RTT and RTTVAR the RTT variation:

First sample R:
    SRTT   = R
    RTTVAR = R / 2
    RTO    = SRTT + max(G, 4 × RTTVAR)

Each later sample R':
    RTTVAR = (1 - beta) × RTTVAR + beta × |SRTT - R'|     beta  = 1/4
    SRTT   = (1 - alpha) × SRTT  + alpha × R'             alpha = 1/8
    RTO    = SRTT + max(G, 4 × RTTVAR)

G is the clock granularity (tiny on modern systems). Note the order: RTTVAR is updated using the old SRTT, then SRTT is updated. RFC 6298 also says RTO should be at least 1 second and may be capped at no less than 60 seconds; Linux uses a lower minimum of 200 ms.

Worked example: RTO from four samples

Samples arrive: 100 ms, 120 ms, 80 ms, 140 ms. Ignore G and the minimum RTO.

Sample 1: R = 100.

• SRTT = 100. RTTVAR = 100 / 2 = 50.
• RTO = 100 + 4 × 50 = 300 ms.

Sample 2: R = 120.

• RTTVAR = 0.75 × 50 + 0.25 × |100 - 120| = 37.5 + 5 = 42.5.
• SRTT = 0.875 × 100 + 0.125 × 120 = 87.5 + 15 = 102.5.
• RTO = 102.5 + 4 × 42.5 = 102.5 + 170 = 272.5 ms.

Sample 3: R = 80.

• RTTVAR = 0.75 × 42.5 + 0.25 × |102.5 - 80| = 31.875 + 5.625 = 37.5.
• SRTT = 0.875 × 102.5 + 0.125 × 80 = 89.6875 + 10 = 99.6875.
• RTO = 99.6875 + 4 × 37.5 = 249.69 ms.

Sample 4: R = 140.

• RTTVAR = 0.75 × 37.5 + 0.25 × |99.6875 - 140| = 28.125 + 10.078 = 38.203.
• SRTT = 0.875 × 99.6875 + 0.125 × 140 = 87.227 + 17.5 = 104.727.
• RTO = 104.727 + 4 × 38.203 = 257.54 ms.

The average RTT is near 100 ms, but because samples swing by 20 to 40 ms, the RTO stays around 250 ms, safely above the likely range of real RTTs.

def rto_trace(samples, alpha=1/8, beta=1/4):
    srtt = rttvar = None
    for r in samples:
        if srtt is None:
            srtt, rttvar = r, r / 2
        else:
            rttvar = (1 - beta) * rttvar + beta * abs(srtt - r)
            srtt = (1 - alpha) * srtt + alpha * r
        print(f"R={r} SRTT={srtt:.3f} RTTVAR={rttvar:.3f} RTO={srtt + 4*rttvar:.2f}")

rto_trace([100, 120, 80, 140])

Karn's algorithm

When a segment is retransmitted and then an ACK arrives, which transmission is it acknowledging? If TCP guesses wrong, its RTT sample is garbage. Karn's algorithm says:

1. Do not take RTT samples from retransmitted segments.
2. Back off the timer exponentially: on each timeout, double the RTO (for example 1 s, 2 s, 4 s, 8 s), keeping the doubled value until an ACK for a segment that was not retransmitted gives a fresh sample.

The timestamps option (RFC 7323) removes the ambiguity entirely: each segment carries a timestamp that the ACK echoes back, so RTT can be measured even for retransmissions.

Congestion control

Why it is needed

If senders ignore congestion, queues at bottleneck routers grow, delays soar, and packets are dropped. Senders then retransmit, adding even more load; packets dropped late in their journey waste capacity on every earlier link. In the 1980s the early internet suffered congestion collapse, where throughput fell by orders of magnitude while links stayed busy carrying retransmissions. Jacobson's congestion-control algorithms were the response.

TCP congestion control is end to end: routers (classically) give no explicit feedback. Senders infer congestion from:

• Loss, seen as a timeout or as three duplicate ACKs.
• Optionally ECN (explicit congestion notification), where routers mark packets instead of dropping them.
• In delay-based and model-based algorithms, rising RTT.

The sender limits itself with cwnd. Roughly, its sending rate is cwnd / RTT bytes per second.

Slow start

A new connection does not know the available bandwidth, so it probes. In slow start:

• cwnd starts small: 1 MSS in classic texts; modern systems use an initial window of 10 MSS (RFC 6928).
• For each ACK received, cwnd increases by 1 MSS. Since a full window of segments produces a full window of ACKs, cwnd doubles every RTT.

RTT 1:  [1]                         cwnd = 1
RTT 2:  [1][2]                      cwnd = 2
RTT 3:  [1][2][3][4]                cwnd = 4
RTT 4:  [1][2][3][4][5][6][7][8]    cwnd = 8

"Slow" refers to the small starting point compared with blasting a full window at once; the growth itself is exponential. Slow start ends when cwnd reaches the slow start threshold (ssthresh) or a loss occurs.

Congestion avoidance

Once cwnd is at or above ssthresh, TCP is near the point where loss happened before, so it grows carefully: cwnd increases by about 1 MSS per RTT (in practice, by MSS × MSS / cwnd per ACK). This is linear growth.

AIMD

The combination of additive growth and multiplicative cutback is AIMD (additive increase, multiplicative decrease):

• Additive increase: +1 MSS per RTT while no loss.
• Multiplicative decrease: on loss, cut cwnd (to half in Reno).

The result is the famous sawtooth: cwnd climbs linearly, drops by half, climbs again.

cwnd
 W  |        /|        /|        /|
    |       / |       / |       / |
    |      /  |      /  |      /  |
W/2 |     /   |/    /   |/    /   |/
    |    /
    |   /
    +-----------------------------------> time

If losses happen when cwnd reaches W, cwnd oscillates between W/2 and W, so the average is about 0.75 W per RTT. Example: W = 20 MSS of 1,460 bytes and RTT = 100 ms gives average throughput 0.75 × 20 × 1,460 × 8 / 0.1 ≈ 1.75 Mbps.

A widely used approximation (the Mathis formula) relates Reno-style throughput to loss rate p:

throughput ≈ 1.22 × MSS / (RTT × sqrt(p))

With MSS 1,460 bytes, RTT 100 ms and p = 0.0001 (one loss in 10,000 segments): 1.22 × 11,680 bits / (0.1 × 0.01) ≈ 14.2 Mbps. To reach 10 Gbps over a long path, Reno would need an impossibly low loss rate, which motivated CUBIC and BBR.

Fast retransmit

Waiting for a timeout after a loss is slow (the RTO is several RTTs). But the receiver sends a duplicate ACK each time an out-of-order segment arrives, repeating the ack number of the missing byte. A single duplicate may just be reordering; three duplicate ACKs (four identical ACKs in total) strongly suggest that segment was lost while later ones arrived.

Fast retransmit: on the third duplicate ACK, resend the missing segment immediately, without waiting for the timer.

 Sender                        Receiver
 seg 1 (1-1000) -------------> ACK 1001
 seg 2 (1001-2000) ---X lost
 seg 3 (2001-3000) ----------> ACK 1001   dup 1
 seg 4 (3001-4000) ----------> ACK 1001   dup 2
 seg 5 (4001-5000) ----------> ACK 1001   dup 3
 resend seg 2 --------------->  ACK 5001  (cumulative: holes filled)

Fast recovery

Three duplicate ACKs mean later segments are getting through, so the network is congested but still working. Restarting from cwnd = 1 would be an overreaction. Fast recovery (TCP Reno) responds more gently:

1. Set ssthresh = cwnd / 2.
2. Retransmit the missing segment.
3. Set cwnd = ssthresh + 3 MSS (the three duplicates mean three segments have left the network) and add 1 MSS for each further duplicate ACK ("window inflation"), allowing new data to keep flowing.
4. When a new ACK (one that acknowledges the retransmitted data) arrives, set cwnd = ssthresh and continue in congestion avoidance.

NewReno (RFC 6582) improves step 4 when several segments from one window are lost, by staying in fast recovery until all data outstanding at the time of loss is acknowledged. With SACK, the sender knows exactly which segments are missing.

A timeout, by contrast, means even duplicate ACKs stopped arriving, which suggests serious congestion. Every variant reacts the same way: ssthresh = cwnd / 2, cwnd = 1 MSS (the RFC 5681 "loss window"), and slow start again.

Summary of reactions

Worked example: cwnd trace over time

Track cwnd in MSS units, one row per RTT ("transmission round"). Start: cwnd = 1, ssthresh = 16. Simplifications used in exam problems: cwnd changes once per round; in slow start, doubling is capped at ssthresh; after fast recovery, cwnd is shown as the deflated value ssthresh (ignoring the brief inflation); the minimum ssthresh is 2.

Events: a triple duplicate ACK is detected at the end of round 9, and a timeout at the end of round 13. TCP Reno:

Now compare TCP Tahoe for the same triple duplicate ACK at round 9 (no later timeout). Tahoe treats it like a timeout: ssthresh = 10, cwnd = 1.

Reno keeps far more data in flight after a single loss, which is why it replaced Tahoe.

round  cwnd (Reno, one # per MSS)
   1   # 1
   2   ## 2
   3   #### 4
   4   ######## 8
   5   ################ 16
   6   ################# 17
   7   ################## 18
   8   ################### 19
   9   #################### 20  <- 3 dup ACKs
  10   ########## 10
  11   ########### 11
  12   ############ 12
  13   ############# 13  <- timeout
  14   # 1
  15   ## 2
  16   #### 4
  17   ###### 6
  18   ####### 7

The trace was generated and checked with this script:

def trace(variant, rounds, events, cwnd=1, ssthresh=16):
    for t in range(1, rounds + 1):
        ev = events.get(t, "")
        phase = "SS" if cwnd < ssthresh else "CA"
        print(t, cwnd, ssthresh, phase, ev)
        if ev == "3dup":
            ssthresh = max(cwnd // 2, 2)
            cwnd = ssthresh if variant == "reno" else 1
        elif ev == "timeout":
            ssthresh = max(cwnd // 2, 2)
            cwnd = 1
        elif cwnd < ssthresh:
            cwnd = min(cwnd * 2, ssthresh)
        else:
            cwnd += 1

trace("reno", 18, {9: "3dup", 13: "timeout"})
trace("tahoe", 15, {9: "3dup"})

Tahoe, Reno, CUBIC and BBR

CUBIC

Reno grows by one MSS per RTT, so on a path with a huge BDP it takes a very long time to recover from a halving, and connections with longer RTTs grow more slowly than short ones. CUBIC (RFC 9438) grows cwnd as a cubic function of the time since the last loss, independent of RTT:

• Right after a loss, cwnd climbs quickly back towards W_max, the window where the loss happened.
• Near W_max the curve flattens (a plateau), probing gently where trouble occurred.
• Past W_max it accelerates again to find new bandwidth.

On loss it reduces cwnd by a factor of 0.7 (a 30% cut, gentler than Reno's 50%). Because growth depends on time rather than ACK arrivals, flows with different RTTs share more fairly. CUBIC is the default congestion control on Linux and widely deployed elsewhere.

BBR

Loss-based algorithms keep increasing until buffers overflow, so they tend to fill router queues. On paths with large buffers this causes bufferbloat (high latency); on paths with random non-congestion loss (some wireless links) they back off needlessly.

BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a model-based approach. It continuously estimates:

• BtlBw: the bottleneck bandwidth, from the maximum recent delivery rate.
• RTprop: the minimum RTT, the round trip with empty queues.

It then paces packets at about BtlBw and keeps roughly BtlBw × RTprop (one BDP) in flight, periodically probing for more bandwidth and briefly draining queues to re-measure RTprop. The aim is full throughput with short queues. It is used heavily by large content providers and is available in Linux. Early versions were criticised for being unfair to loss-based flows like CUBIC in some conditions; later versions try to address this.

Other names worth recognising: Vegas (delay-based, slows down when RTT rises), DCTCP (data centres, reacts to the fraction of ECN-marked packets).

Fairness

If K TCP connections share a bottleneck link of rate R, a fair outcome gives each about R/K. AIMD drives connections towards fairness. Picture two flows on a graph with flow 1's throughput on one axis and flow 2's on the other:

• Additive increase moves both flows up by the same amount, along a 45-degree line, which keeps their difference the same.
• Multiplicative decrease halves both, which also halves their difference.

Repeated cycles shrink the gap until both share equally. (Pure additive decrease or multiplicative increase would not converge.)

Caveats that interviewers like:

• RTT unfairness: with Reno, a flow with a shorter RTT gets ACKs faster and grows faster, so it takes more than its share. CUBIC reduces this.
• Multiple connections: fairness is per connection, so an application that opens 10 parallel TCP connections (as download accelerators and older browsers did) gets roughly 10 shares.
• UDP: UDP has no congestion control. Applications sending large UDP flows (video, QUIC) must implement their own, or they can starve TCP flows. QUIC implements congestion control (often CUBIC or BBR) in user space for exactly this reason.

Head-of-line blocking

Head-of-line (HOL) blocking happens when one stuck item at the front of a queue holds up everything behind it.

In TCP, the byte stream must be delivered in order. If segment 5 is lost, segments 6 to 20 may already be sitting in the receiver's buffer, but the application cannot read any of them until segment 5 is retransmitted and arrives, at least one extra RTT later.

This matters most when one TCP connection carries several independent streams:

• HTTP/1.1 sends one request at a time per connection (pipelining was rarely usable), so browsers open several connections. Head-of-line blocking happens at the HTTP level.
• HTTP/2 multiplexes many streams over one TCP connection, fixing HTTP-level blocking. But a single lost TCP segment now stalls all streams, because TCP does not know they are independent. On lossy networks HTTP/2 can be worse than several HTTP/1.1 connections.
• HTTP/3 over QUIC gives each stream its own ordering inside QUIC (over UDP). A lost packet stalls only the streams whose data it carried.

HTTP/2 over TCP: one ordered byte stream
  [A1][B1][A2][xx lost][B2][A3]   -> A and B both wait for the lost packet

HTTP/3 over QUIC: independent streams
  stream A: [A1][A2][A3]          -> delivered
  stream B: [B1][xx lost][B2]     -> only B waits

More on HTTP versions in HTTP and the web.

Socket programming: a TCP echo server and client

Theory is easier to remember after you have used it. Below is a working TCP echo server (it sends back whatever it receives) and a client, in Python's standard socket module. Both were run and tested on Python 3.

The socket call sequence

        Server                          Client
   socket()                        socket()
   bind(host, port)
   listen(backlog)
   accept()  <---- 3-way handshake ----  connect(host, port)
      |  (blocks until a client                |
      |   connection is ready)                 |
   recv() <------------------------------- send()
   send() -------------------------------> recv()
   close() <------- FIN / ACK -----------> close()

• socket(AF_INET, SOCK_STREAM) creates a TCP socket (SOCK_DGRAM would be UDP).
• bind attaches it to an address and port; listen marks it as a listening socket and sets the backlog, the queue of completed connections waiting for accept.
• The kernel performs the three-way handshake; accept just hands you a new socket for one completed connection. The listening socket keeps listening.
• connect on the client triggers the handshake.

Server

# echo_server.py
import socket
import threading

HOST, PORT = "127.0.0.1", 9000


def handle(conn: socket.socket, addr) -> None:
    with conn:
        while True:
            data = conn.recv(4096)       # returns b"" when the client closes
            if not data:
                break
            conn.sendall(data)           # sendall loops until every byte is sent
    print("closed", addr, flush=True)


def main() -> None:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as srv:
        srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        srv.bind((HOST, PORT))
        srv.listen(128)                  # backlog of fully set-up connections
        print("listening on", PORT, flush=True)
        while True:
            conn, addr = srv.accept()    # handshake already done by the kernel
            threading.Thread(target=handle, args=(conn, addr), daemon=True).start()


if __name__ == "__main__":
    main()

Client

# echo_client.py
import socket

HOST, PORT = "127.0.0.1", 9000


def recv_exactly(sock: socket.socket, n: int) -> bytes:
    buf = b""
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            raise ConnectionError("server closed early")
        buf += chunk
    return buf


with socket.create_connection((HOST, PORT), timeout=5) as sock:
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)  # disable Nagle
    for msg in [b"hello", b"tcp is a byte stream", b"bye"]:
        sock.sendall(msg)
        reply = recv_exactly(sock, len(msg))
        print("sent", msg, "got", reply)

Run the server in one terminal (python3 echo_server.py) and the client in another (python3 echo_client.py). Output:

sent b'hello' got b'hello'
sent b'tcp is a byte stream' got b'tcp is a byte stream'
sent b'bye' got b'bye'

What each detail teaches

• recv returning b"" is how the application sees the peer's FIN. The server then leaves the loop and closes, sending its own FIN. Forgetting to close here is exactly how sockets pile up in CLOSE_WAIT.
• sendall instead of send: send may write only part of the buffer if the kernel's send buffer is full (flow control pushing back). sendall loops until everything is sent.
• recv_exactly: TCP is a byte stream. One sendall of 20 bytes may arrive as two recv results of 12 and 8 bytes, or merged with the next message. Real protocols frame messages, typically with a length prefix or a delimiter, and the reader loops until a full message is in hand.
• SO_REUSEADDR lets the server restart and bind to port 9000 immediately, even if old connections from the previous run are still in TIME_WAIT.
• TCP_NODELAY disables Nagle, so each small message goes out at once. For an echo client that waits for each reply, this avoids the Nagle and delayed-ACK stall described earlier.
• One thread per connection is simple but does not scale to tens of thousands of clients. Production servers use event loops (selectors, asyncio, epoll on Linux, kqueue on BSD and macOS) that watch many sockets from one thread.
• Timeouts: create_connection(..., timeout=5) makes blocking calls raise after 5 seconds instead of hanging forever if the network or server misbehaves.

You can watch the connection states while the client runs with ss -tan '( sport = :9000 or dport = :9000 )' on Linux or netstat -an | grep 9000 on macOS, and you will see LISTEN, ESTABLISHED and, after the client exits, TIME_WAIT on the side that closed first.

Interview questions

Q1. What is the difference between flow control and congestion control?

Flow control protects the receiver: the receiver advertises rwnd, the free space in its buffer, and the sender never exceeds it. Congestion control protects the network: the sender keeps cwnd, inferred from losses and delay, to avoid overloading routers. The sender's limit is the minimum of the two.

Q2. What happens when the receiver advertises a zero window?

The sender stops sending new data and starts the persist timer. It periodically sends zero-window probes, forcing the receiver to reply with its current window, so a lost window update cannot deadlock the connection. Sending resumes as soon as the window opens.

Q3. What is silly window syndrome and how is it fixed?

It is TCP sending a stream of tiny segments with high header overhead, caused by a receiver advertising tiny windows or a sender transmitting tiny writes. Clark's solution makes the receiver advertise zero until it can offer at least an MSS or half its buffer. Nagle's algorithm makes the sender batch small writes while data is unacknowledged.

Q4. Explain Nagle's algorithm and its interaction with delayed ACK.

Nagle holds small writes while any data is unacknowledged, sending them when an ACK arrives or a full MSS accumulates. Delayed ACK holds ACKs briefly to piggyback them. Together, a sender waiting for an ACK and a receiver waiting for more data can stall for the delayed-ACK timeout, typically 40 to 200 ms; the fix is to write whole messages at once or set TCP_NODELAY.

Q5. How does TCP compute the retransmission timeout?

It keeps a smoothed RTT and an RTT variation. On each sample, RTTVAR = 3/4 RTTVAR + 1/4 times the absolute difference between SRTT and the sample, then SRTT = 7/8 SRTT + 1/8 sample, and RTO = SRTT + 4 × RTTVAR, with a minimum (1 s in RFC 6298, 200 ms on Linux). Including variance stops the timer from firing too early on jittery paths.

Q6. What is Karn's algorithm?

It ignores RTT samples from retransmitted segments, because the ACK could belong to either transmission. It also doubles the RTO on every timeout and keeps the backed-off value until a valid sample arrives. The timestamps option removes the ambiguity altogether.

Q7. Describe slow start. Why is it called slow?

cwnd starts at a small value (1 MSS classically, 10 MSS on modern systems) and grows by 1 MSS per ACK, which doubles it each RTT. It is "slow" only compared with sending a full window immediately; its growth is exponential. It ends when cwnd reaches ssthresh or a loss is detected.

Q8. How does TCP react differently to a timeout and to three duplicate ACKs?

Both set ssthresh to half of cwnd. On a timeout, cwnd drops to 1 MSS and slow start begins, because nothing is getting through. On three duplicate ACKs, Reno retransmits immediately (fast retransmit) and sets cwnd to the new ssthresh (fast recovery), since later segments are still arriving.

Q9. Why does AIMD converge to fairness?

Additive increase raises all flows by the same amount, leaving their difference unchanged, while multiplicative decrease cuts every flow by the same factor, shrinking the difference. Over repeated cycles the flows approach equal shares. RTT differences and multiple parallel connections can still skew the outcome.

Q10. What is the difference between Reno and CUBIC?

Reno grows cwnd by one MSS per RTT and halves it on loss, so it recovers slowly on high-BDP paths and favours short-RTT flows. CUBIC grows cwnd as a cubic function of time since the last loss, with a plateau near the previous maximum, and cuts by a factor of 0.7. It is RTT-independent in growth and is the Linux default.

Q11. How does BBR differ from loss-based algorithms?

BBR does not treat loss as the main congestion signal. It estimates bottleneck bandwidth and minimum RTT, paces packets at the estimated bandwidth and keeps about one BDP in flight. This keeps router queues short, reducing bufferbloat, and avoids needless slowdowns on randomly lossy links.

Q12. What is head-of-line blocking in TCP, and how does QUIC avoid it?

Because TCP delivers bytes strictly in order, one lost segment blocks delivery of all later data, even data belonging to unrelated HTTP/2 streams. QUIC runs over UDP and orders data per stream, so a lost packet only delays the streams whose data it carried.

Q13. Why might send write fewer bytes than requested?

The kernel's send buffer may have less free space than the data, because the receiver's window or congestion window is limiting how fast data leaves. A non-blocking or partially completed send returns the count written, and the program must send the rest. Python's sendall does this loop for you.

Q14. What window scale factor is needed for 1 Gbps with 50 ms RTT?

The BDP is 10^9 × 0.05 / 8 = 6.25 MB, about 95.4 times 65,535 bytes. The smallest power of two above that is 128, so a scale shift of 7 is needed. Without scaling, throughput would be capped near 10.5 Mbps on this path (65,535 bytes per 50 ms).

Q15. Starting at cwnd 1 with ssthresh 16, what is cwnd after 6 RTTs with no loss?

Slow start gives 1, 2, 4, 8, 16 over rounds 1 to 5, reaching ssthresh. Round 6 is congestion avoidance, so cwnd is 17 MSS. If the exam convention lets slow start overshoot, say so; under the capped convention used here the answer is 17.

Key takeaways

• The sender's limit is min(rwnd, cwnd): rwnd is flow control (receiver buffer), cwnd is congestion control (network).
• A zero window triggers persist-timer probes; window scaling lifts the 64 KB limit for high-BDP paths.
• Silly window syndrome is fixed by Clark's rule at the receiver and Nagle's algorithm at the sender; Nagle plus delayed ACK can add 40 to 200 ms stalls, fixed by whole writes or TCP_NODELAY.
• RTO = SRTT + 4 × RTTVAR with alpha 1/8 and beta 1/4; Karn ignores samples from retransmissions and backs off exponentially.
• Slow start doubles cwnd per RTT up to ssthresh; congestion avoidance adds 1 MSS per RTT; together with halving on loss, this is AIMD's sawtooth.
• Three duplicate ACKs trigger fast retransmit; Reno's fast recovery sets cwnd to half, while a timeout resets cwnd to 1 in every variant.
• CUBIC (Linux default) grows by a cubic function of time; BBR paces at measured bottleneck bandwidth and keeps queues short.
• AIMD converges to fairness, with caveats for RTT differences, parallel connections and UDP.
• TCP's in-order delivery causes head-of-line blocking; HTTP/3 over QUIC isolates streams.
• In socket code, handle partial sends and receives, close on end-of-file, and frame your own messages.

Next lesson

Continue with DNS.