TCP vs UDP

Article.

Two transport protocols. TCP is reliable and connection-oriented; UDP is best-effort datagrams. Almost every Python backend speaks TCP (HTTP, gRPC, Postgres, Redis), and UDP shows up for DNS, NTP, voice and video, metrics, and underneath QUIC. The question worth answering properly is what TCP actually guarantees, and what each guarantee costs your application: the costs are where the production bugs live. The runs below are Python 3.14.5 on loopback.

Side by side

The mechanism: a byte stream, not messages

TCP numbers every byte, retransmits what is not acknowledged, and hands the receiver an ordered stream. It has no idea where your messages start and end. Three sendall() calls, one recv():

for m in (b"BUY 10\n", b"SELL 5\n", b"BUY 3\n"):
    conn.sendall(m)
...
client.recv(4096)

three sendall() calls, one recv(4096): b'BUY 10\nSELL 5\nBUY 3\n'
one 5 MB sendall(), recv(1 MiB) calls: 8,
  first sizes [1048576, 326604, 196485, 1048576, 457809]

Messages glue together, and one large message arrives in pieces of whatever size the kernel had ready.

Framing is the application’s job

Any protocol over raw TCP needs framing: a length prefix or a delimiter, and a loop that reads until the frame is complete:

def recv_exact(sock: socket.socket, n: int) -> bytes:
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            raise ConnectionError("peer closed mid-message")
        buf += chunk
    return bytes(buf)

def recv_frame(sock: socket.socket) -> bytes:
    (length,) = struct.unpack("!I", recv_exact(sock, 4))   # 4-byte length prefix
    return recv_exact(sock, length)

HTTP, gRPC and every database protocol do this for you; a hand-rolled TCP protocol that calls recv(4096) once per message works on loopback in tests and fails under real network conditions.

UDP keeps boundaries, and has limits

three sendto(), three recvfrom(): [b'BUY 10', b'SELL 5', b'BUY 3']
a 70,000-byte datagram: OSError: A message sent on a datagram socket was larger than ...

One sendto() is one datagram is one recvfrom(). In exchange there is no retransmission, no ordering and no congestion control, and a datagram cannot exceed 65,507 bytes of payload; anything over the path MTU (about 1,500 bytes) is fragmented, and one lost fragment loses the whole datagram. Practical UDP protocols keep datagrams small.

What the guarantees cost

• A round trip before any data. The handshake is 1 RTT; TLS 1.3 adds one more (TLS 1.2 two). At 80 ms between regions, a fresh HTTPS connection costs 160-240 ms before the first request byte, which is why clients pool connections (HTTPS Handshake, mTLS, Keepalive, and the Reverse-Proxy Edge).
• Head-of-line blocking. In-order delivery means one lost packet stalls every byte behind it, including other HTTP/2 streams sharing the connection. That is why HTTP/3 moved to QUIC over UDP: reliability per stream, so a loss stalls only its own stream.
• State on both ends, including after close.

Where it breaks in production

TIME_WAIT and port exhaustion

The side that closes first keeps the connection in TIME_WAIT (60 seconds on Linux) so that delayed packets from it cannot be mistaken for a new connection. A client that opens a new connection per request piles them up:

2,000 short connections: TIME_WAIT sockets 5 -> 2327,
  1965 on the client side (it closed first)

Each one holds an ephemeral port toward that destination. Linux’s default range (32768-60999) is 28,232 ports, so a client making new connections to one ip:port tops out near 470 per second (28,232 / 60 s) before connect() fails with “Cannot assign requested address”.

The fix is connection reuse: a requests.Session, one httpx.AsyncClient, a database pool. SO_REUSEADDR does not help a client; it lets a server rebind its listening port while old connections sit in TIME_WAIT. Linux’s net.ipv4.tcp_tw_reuse lets outgoing connections reuse TIME_WAIT ports, a mitigation rather than a fix.

CLOSE_WAIT means your code never closed

CLOSE_WAIT piling up on a server is the opposite: the peer closed, the kernel acknowledged, and the application still holds the socket. It is a leak in your code (a response object never closed, a connection never returned), and it ends in “too many open files”.

40 ms on small writes

Nagle’s algorithm holds back small segments while earlier data is unacknowledged, and the receiver’s delayed ACK waits up to about 40 ms on Linux. A protocol that writes a request in two small pieces and waits for a reply hits both and gains a steady 40 ms per round trip.

TCP_NODELAY disables Nagle; asyncio sets it on every TCP transport, plain socket does not, so hand-written socket code is where this still appears.

The rest

• Idle connections killed in the middle. A NAT or load balancer drops idle mappings (an AWS ALB after 60 s by default), and the next write on a pooled connection fails. Keepalives or a pool pre_ping handle it.
• DNS over TCP blocked. DNS uses UDP by default, and EDNS(0) lets responses reach about 1,232 bytes, but a truncated answer (large DNSSEC or many records) is retried over TCP port 53. A firewall that allows only UDP/53 breaks exactly those lookups.

The trade not taken: your own protocol on UDP

“TCP is slow, we will use UDP” means reimplementing what TCP provides: retransmission, ordering, congestion control, flow control, and doing it without the decades of tuning in the kernel’s stack. The result is usually either unreliable or a slower TCP.

When TCP’s head-of-line blocking or handshake cost is the real problem, use the protocol that already solved it, QUIC (HTTP/3), and keep raw UDP for traffic that genuinely tolerates loss: voice and video frames, metrics, DNS.

Python sockets

import socket

tcp = socket.create_server(("0.0.0.0", 9000))     # SOCK_STREAM; SO_REUSEADDR on POSIX
conn, addr = tcp.accept()
data = conn.recv(4096)                            # up to 4096 bytes of the stream

udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.bind(("0.0.0.0", 9001))
data, addr = udp.recvfrom(4096)                   # exactly one datagram

TCP states worth recognising

Related

• OSI and TCP/IP models - where TCP and UDP sit in the stack
• Ports and Sockets - ports, sockets and ephemeral ranges
• HTTPS Handshake, mTLS, Keepalive, and the Reverse-Proxy Edge - TLS on top, and keep-alive
• HTTP versions: 1.1, 2 and 3 - HTTP/2, HTTP/3 and head-of-line blocking