Simulate latency and packet loss with tc netem

Your app works on your laptop, where every packet arrives in under a millisecond. Real networks add delay, jitter, loss and limited bandwidth. This post shows how to add each of those between two Docker containers with tc netem, and how to measure the result with ping and iperf3. Every number below is real output from these commands.

What you'll build

 +-------------------+                     +-------------------+
 | netem-a           |  eth0      eth0     | netem-b           |
 | 172.30.0.10       |---------------------| 172.30.0.20       |
 | tc netem on eth0  |   netem-net         | iperf3 -s         |
 +-------------------+   172.30.0.0/24     +-------------------+

Two containers on one Docker network. You apply the impairment to netem-a's eth0 and measure towards netem-b.

What you need

• Docker on Linux, or Docker Desktop on macOS or Windows. On Docker Desktop the containers run in a Linux VM, so the kernel features are there.
• The nicolaka/netshoot image. It includes tc (iproute2), ping and iperf3.

How netem works, in short

Linux sends every outgoing packet through a queueing discipline (qdisc) on its interface. tc is the tool that manages qdiscs. netem is a qdisc that holds, drops, duplicates or reorders packets on purpose. The tc-netem manual page lists every option.

One rule explains most surprises: netem acts on egress. A qdisc on netem-a's eth0 affects packets leaving netem-a, not packets arriving there.

Step 1: Start the containers

docker network create --subnet 172.30.0.0/24 netem-net
docker run -d --name netem-a --hostname a --cap-add NET_ADMIN \
  --network netem-net --ip 172.30.0.10 nicolaka/netshoot:v0.13 sleep infinity
docker run -d --name netem-b --hostname b --cap-add NET_ADMIN \
  --network netem-net --ip 172.30.0.20 nicolaka/netshoot:v0.13 iperf3 -s

--cap-add NET_ADMIN lets processes in the container change the qdisc. Without it, tc fails with RTNETLINK answers: Operation not permitted. netem-b runs an iperf3 server so you can measure throughput.

Step 2: Measure a baseline

docker exec netem-a ping -c 5 172.30.0.20
docker exec netem-a tc qdisc show dev eth0
docker exec netem-a iperf3 -c 172.30.0.20 -t 5

rtt min/avg/max/mdev = 0.044/0.065/0.116/0.026 ms
qdisc noqueue 0: root refcnt 2
[  5]   0.00-5.00   sec  14.0 GBytes  24.0 Gbits/sec                  receiver

Well under a millisecond and 24 Gbit/s, because the traffic never leaves the machine. noqueue means no qdisc is attached yet. Your numbers will differ; what matters is the change in the next steps.

Step 3: Add latency

docker exec netem-a tc qdisc add dev eth0 root netem delay 100ms
docker exec netem-a tc qdisc show dev eth0
docker exec netem-a ping -c 5 172.30.0.20

qdisc netem 8003: root refcnt 17 limit 1000 delay 100ms seed 4975856179656527505

64 bytes from 172.30.0.20: icmp_seq=1 ttl=64 time=100 ms
64 bytes from 172.30.0.20: icmp_seq=2 ttl=64 time=100 ms
64 bytes from 172.30.0.20: icmp_seq=3 ttl=64 time=100 ms
rtt min/avg/max/mdev = 100.194/100.317/100.555/0.123 ms

The round trip is 100 ms, not 200. Only the echo request leaving netem-a is delayed; the reply leaving netem-b is not. To model 100 ms each way, add the same qdisc on netem-b too.

Step 4: Add jitter

Real latency varies. Give netem a second value, the jitter, and a distribution:

docker exec netem-a tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal
docker exec netem-a ping -c 10 172.30.0.20

64 bytes from 172.30.0.20: icmp_seq=4 ttl=64 time=95.2 ms
64 bytes from 172.30.0.20: icmp_seq=5 ttl=64 time=74.1 ms
64 bytes from 172.30.0.20: icmp_seq=6 ttl=64 time=124 ms
64 bytes from 172.30.0.20: icmp_seq=7 ttl=64 time=83.5 ms
rtt min/avg/max/mdev = 72.770/96.412/124.682/17.845 ms

Each packet now gets 100 ms plus or minus a normally distributed amount with a 20 ms spread. tc qdisc show displays delay 100ms 20ms; it does not print the distribution name. change modifies the existing qdisc, while add would fail because one is already attached.

Large jitter reorders packets: a packet with a short delay can overtake an earlier one. That is realistic for some paths and a good test for protocols that care about order.

Step 5: Add packet loss

Use replace, which works whether a qdisc exists or not. This drops the delay and keeps only loss:

docker exec netem-a tc qdisc replace dev eth0 root netem loss 5%
docker exec netem-a ping -q -c 1000 -i 0.01 172.30.0.20

--- 172.30.0.20 ping statistics ---
1000 packets transmitted, 953 received, 4.7% packet loss, time 11027ms
rtt min/avg/max/mdev = 0.018/0.056/0.199/0.023 ms

4.7% measured for 5% set. -i 0.01 sends a ping every 10 ms, which needs root; you are root inside the container. Keep the sample large: in an earlier run with delay, jitter and 5% loss together, 200 pings showed only 2.5% loss. netem drops each packet at random, so short tests are noisy.

Step 6: Limit bandwidth

netem has a rate option that caps how fast packets leave:

docker exec netem-a tc qdisc replace dev eth0 root netem rate 10mbit
docker exec netem-a iperf3 -c 172.30.0.20 -t 5

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.00 MBytes  33.5 Mbits/sec    0   1.13 MBytes
[  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec    0   1.18 MBytes
[  5]   2.00-3.00   sec  1.38 MBytes  11.5 Mbits/sec    0   1.24 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   0.00-5.00   sec  9.38 MBytes  15.7 Mbits/sec    0             sender
[  5]   0.00-6.15   sec  6.88 MBytes  9.38 Mbits/sec                  receiver

Read the receiver line: 9.38 Mbit/s. The sender's first second looks faster because iperf3 counts data handed to the socket, and the first megabytes only fill the queue.

Options combine in one command. Here is a 10 Mbit/s link with 50 ms of delay:

docker exec netem-a tc qdisc replace dev eth0 root netem delay 50ms rate 10mbit
docker exec netem-a iperf3 -c 172.30.0.20 -t 8
docker exec netem-a ping -c 3 172.30.0.20

[  5]   0.00-8.60   sec  9.62 MBytes  9.39 Mbits/sec                  receiver
rtt min/avg/max/mdev = 50.271/50.457/50.730/0.196 ms

Now reverse the test with -R, so netem-b sends and netem-a receives:

docker exec netem-a iperf3 -c 172.30.0.20 -t 5 -R

[  5]   0.00-5.00   sec   345 MBytes   578 Mbits/sec                  receiver

578 Mbit/s, far above 10. The data leaves netem-b, which has no qdisc. It is still well below the 24 Gbit/s baseline, because TCP's acknowledgements leave netem-a and get delayed by 50 ms. This is the egress rule again, and the reason to shape both ends when you model a real link.

Step 7: Impair a container that has no tc

Your own app images rarely include tc or NET_ADMIN. You don't need either. Run tc from a helper container that joins the target's network namespace:

docker run --rm --cap-add NET_ADMIN --network container:netem-b \
  nicolaka/netshoot:v0.13 tc qdisc add dev eth0 root netem delay 30ms
docker exec netem-a ping -c 3 172.30.0.20

rtt min/avg/max/mdev = 80.409/80.463/80.536/0.053 ms

80 ms: 50 ms leaving netem-a plus 30 ms leaving netem-b. The qdisc stays on netem-b's interface after the helper exits.

Remove the impairment

Delete the root qdisc on netem-a and ping again:

docker exec netem-a tc qdisc del dev eth0 root
docker exec netem-a ping -c 3 172.30.0.20

rtt min/avg/max/mdev = 30.282/30.478/30.594/0.139 ms

The 50 ms and the rate cap are gone. The 30 ms left is netem-b's qdisc. Remove it the same way you added it:

docker run --rm --cap-add NET_ADMIN --network container:netem-b \
  nicolaka/netshoot:v0.13 tc qdisc del dev eth0 root

Both interfaces are back to noqueue.

Clean up

docker rm -f netem-a netem-b
docker network rm netem-net

What to test with it

A few cases where these four knobs find real bugs:

• Timeouts. Add a delay larger than your client's timeout and check that it fails cleanly instead of hanging.
• Retries. Add loss and watch whether your app retries, gives up, or sends duplicate requests.
• Chatty protocols. Add 50 ms of delay and count how long a page or a sync takes. Many sequential round trips show up at once.
• Large transfers. Cap the rate and check progress bars, backpressure and memory use while a queue builds up.

Change one thing at a time, and write down the exact tc line with each result.

Quick reference

Going further

• More impairments. The manual page also documents duplicate, corrupt, reorder and correlation values, plus the Gilbert-Elliott model for bursty loss.
• Finer shaping. rate is simple. For queues and burst sizes, look at the tbf and htb qdiscs.
• Test your app. Run a database or API in netem-b and watch timeouts and retries with 200 ms and 2% loss.
• Routed labs. Add netem to the links of the OSPF lab with FRR in Docker and watch hellos and convergence under loss.

iperf3 and netshoot document their own options. Tested with iproute2 6.9.0 and iperf 3.16 in netshoot v0.13, on Docker 29 with a 7.0 Linux kernel.

Set it per cable instead

In Isothrum's editor you can set latency, jitter, loss and bandwidth on each cable or LAN of a lab you draw in the browser, with no account. It exports an isoloom.yml plus Docker Compose and other files, and you run the lab with those tools. It is a quick way to build a network lab with bad links on purpose, without typing a tc line per interface.