A 50 ms egress delay slows llama.cpp RPC, and the logs do not say so

One-way delay on a single RPC worker cuts generation rate from about 22–33 tok/s to about 11 tok/s. Neither the client nor the workers report a bad path.

NoteWhat this note is

A local harness produced these repeats on 23 September 2026. They describe that machine. The full tables stay with the experiment record.

Why this experiment

A split inference request finishes only when the pieces can still talk. llama.cpp RPC keeps the prompt in one process. Two ggml-rpc-server processes run part of the graph (llama.cpp contributors 2026). Every remote call pushes those bytes across a socket.

We did not start from a general complaint about unreliable networks. The question was narrower. The bench was built so we could break exactly one thing:

If packets leaving one worker are delayed by 50 ms, does generation get slower, and does anyone say that the path is bad?

Fifty milliseconds is a long time for a datacenter hop. A person would barely notice it. A system that treats a link as either up or dead should show this delay in the tokens and keep it out of the logs. That gap is the measurement. Gray failure is the usual name for a fault the token stream sees while llama-cli and the workers print nothing (Huang et al. 2017).

flowchart LR
  subgraph host [One laptop, three containers]
    C[llama-cli]
    A[rpc-a]
    B[rpc-b]
  end
  C -->|RPC request| A
  C -->|RPC request| B
  A -->|reply, no queue| C
  B -->|reply through netem| C
  Q[tc netem delay 50ms on rpc-b egress]
  Q -.-> B

Only packets leaving rpc-b enter the queue. Arrivals are untouched. This is not a round trip. rpc-a is the control in the same run. Same binary. No queue.

Why each step

The published full image does not ship ggml-rpc-server. Awkward, if you expected a thin client. We built failure-lab-llama-rpc:local from llama.cpp v0.4.1, build b1-b29c606, with the RPC backend on. The client and both workers use that image. A missing binary cannot pass for a network result.

Qwen2.5-0.5B-Instruct, Q4_K_M, ran on CPU. A small model makes 50 ms large next to the compute. It says nothing about a 70B model on two GPUs.

The prompt stayed The capital of France is. Thirty-two new tokens. The flag was --single-turn. Without that flag, llama-cli opens a chat and waits. Our first attempt sat there until the harness timed out at 180 s. That was a setup bug. It was not the network. Every later repeat exits on its own.

Three repeats ran with no netem before any queue was installed. A delayed run inside that band would have meant no effect. Those repeats generated at 30.1, 24.6, and 33.1 tok/s. A later baseline, taken once worker logs were on, came in at 30.4, 22.4, and 29.9 tok/s. Every answer was “Paris.” The exit code was 0. Short RPC connections on both workers lasted about 0.3 ms.

One knob moved. tc netem delay 50ms went on rpc-b’s egress eth0 (Linux man-pages project 2024). The kernel answered with qdisc netem ... delay 50ms. rpc-a stayed clean. The harness writes that injected delay next to the client’s stdout. The harness knows what it broke. The process does not.

sequenceDiagram
  participant C as llama-cli
  participant A as rpc-a
  participant B as rpc-b
  C->>A: RPC call
  A-->>C: reply, no delay
  C->>B: RPC call
  Note over B: egress queue holds the reply 50 ms
  B-->>C: reply
  Note over A,B: logs say Accepted / Closed, nothing about delay

An empty client stderr only shows that the client was quiet. After the first runs we kept rpc-a.log and rpc-b.log for each repeat. The pair that has those files is the baseline 20260923-172749 and the 50 ms run 20260923-172448.

What the 50 ms run did

Three repeats finished with exit code 0. The answer was still “Paris.” Wall time around the client was about 12 s. With no queue it was about 3 s. Model load sits inside that wall time. The number we compare is generation rate.

Condition	Generation, tok/s
No queue	30.1, 24.6, 33.1
No queue, with worker logs	30.4, 22.4, 29.9
50 ms egress delay	11.4, 11.2, 11.4
Same delay, earlier run	10.8, 11.7, 10.7

The cluster near 11 tok/s does not overlap the 22–33 tok/s baseline. The same queue, run twice, landed in the same place.

Short connections carry the same story in timestamps. The words stay silent. Under the queue, short connections to rpc-b lasted about 106 ms. Short connections to rpc-a in that run lasted about 0.4 ms. Take the queue away. Both workers sit near 0.3 ms again. The extra hold shows up with the queue. It goes away without the queue. We did not count packets. 106 ms is the hold we measured. It is not “exactly two 50 ms delays.”

Neighboring delays still had no worker logs. They moved the same way. The requests still finished. At 10 ms the rates were 21.8, 15.1, and 20.8 tok/s. All of them sit under the first baseline. The gap is smaller than the spread inside the point. At 200 ms the rates were 4.0, 3.2, and 3.3 tok/s. Exit code 0. The answer was complete. Those points are context. They are not the logged pair.

What the system said

Client stderr was empty.

The server banner is in the worker logs. So are Accepted client connection and Client connection closed. Nothing in rpc-a.log or rpc-b.log mentions a slow path, a timeout, or a sick peer. That is true with the queue. It is also true without it.

The worker never names the gap between 0.3 ms and 106 ms. Only the timestamps do.

A health line in the harness event log is not this evidence. The harness writes present: false by itself. The reason is not-sampled-yet. Nobody had read the workers yet.

Conclusion

A live worker behind a 50 ms egress delay still finishes the request. Generation falls from roughly 22–33 tok/s to about 11 tok/s. The client does not hang. The logs from that run do not mention the delay. The path is worse in the token rate. It is also worse in the connection hold. The text of the log matches the healthy run.

Two predictions fail for this setup. The delayed repeats were done in about 12 s. The client does not sit out a 180 s timeout. Neither the client nor the workers announce a bad path.

This is not a result for every degraded link. One direction. One small CPU model. Worker logs only for the 0 ms and 50 ms pair.

What this note does not show

Delay in both directions is absent. Packet loss on the same day did not shift the whole set the way 50 ms did. Some repeats fell to 8–16 tok/s. One stayed near the baseline. At 100% loss the client died in 3.2 s. The exit code was 134. The line was Failed to connect to rpc-b:50052. That is a dead path. The next note takes it up.

Worker logs at 10 ms and at 200 ms were not kept. vLLM was not in the room. Neither were GPUs or a large KV cache.

How to repeat

From the failure-lab tree, after make e001-image and with the model downloaded:

python3 -m harness run --scenario experiments/E-001/scenarios/baseline.json
python3 -m harness run --scenario experiments/E-001/scenarios/d50.json

Each invocation writes a new directory. Inside it: client.stdout, rpc-a.log, and rpc-b.log.

References

Huang, Peng, Chuanxiong Guo, Lidong Zhou, et al. 2017. “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems.” Proceedings of the 16th Workshop on Hot Topics in Operating Systems, HotOS. https://doi.org/10.1145/3102980.3103005.

Linux man-pages project. 2024. Tc-Netem(8) — Network Emulator. Https://man7.org/linux/man-pages/man8/tc-netem.8.html.

llama.cpp contributors. 2026. Llama.cpp. Https://github.com/ggml-org/llama.cpp.