A slow RPC worker finishes quietly. A dead one aborts the client.

Delay on a live llama.cpp worker leaves exit code 0 and empty logs. Stopping that worker mid-generation yields exit code 134 and an explicit crash line. The worker logs name neither case.

NoteWhat this note is

The harness is the one from the delay note. So is the llama.cpp RPC image. rpc-b is damaged in two different ways. The repeats are from 23 and 24 September 2026.

Why this comparison

A live but slow worker still returns “Paris.” It never says the path is bad. That was the delay experiment. The other edge stays open. Generation has already started. The worker process is gone. Does the client wait? Does it fail over to the remaining worker? Does it abort?

The faults are not the same. One run with both of them would blur the result. A queue holds packets. The process stays up. docker stop sends SIGTERM and, if that is not enough, kills the process (Docker Inc. 2026). The socket closes because the process is dead. It does not close because a queue dropped bytes. We measured the second case only after the first had a number.

flowchart TD
  Q[What happened to rpc-b?]
  Q -->|egress delayed, process alive| S[Request finishes, exit 0]
  Q -->|process stopped mid-generation| D[Client aborts, exit 134]
  S --> L[Worker logs: Accepted / Closed only]
  D --> L

Why each step

Both runs stay on failure-lab-llama-rpc:local. The build is llama.cpp v0.4.1, b1-b29c606. The model is Qwen2.5-0.5B-Instruct Q4_K_M. A rebuild would have changed the binary and the scenario together. The image stayed put.

The no-queue rate was already in hand. Generation came in at 30.1, 24.6, and 33.1 tok/s. The logged rerun came in at 30.4, 22.4, and 29.9 tok/s. Exit code 0. The answer was “Paris.” Short connections on both workers lasted about 0.3 ms. A slow path has to leave that band.

On the slow path the only change is the queue. tc netem delay sits on rpc-b egress (Linux man-pages project 2024). The client asks for 32 new tokens with --single-turn. No process is killed. The logged 50 ms run is runs/E-001/d50/20260923-172448. Generation there was 11.4, 11.2, and 11.4 tok/s. Exit code 0. The answer was complete. At 200 ms there were no worker logs. The rates were 4.0, 3.2, and 3.3 tok/s. Exit code 0 again.

The dead path has to hit during generation. An earlier 100% loss run died at connect, before any tokens, with Failed to connect to rpc-b:50052. That does not answer the question. The worker has to disappear after the first tokens. The first mid-generation attempt missed as well. Without --ignore-eos the model stopped at “Paris.” Eight extra bytes after the prompt were that answer. docker stop landed on a finished sentence. Two of those repeats never emitted stop. The directory 20260924-134809 is not evidence. Call it a dry run. A slightly embarrassing one.

The run that counts forces a long generation. It also forces a stream we can see while the process is still alive. The client uses -n 128, --ignore-eos, and --simple-io. The harness waits until stdout contains The capital of France is and at least eight further bytes. Then it runs docker stop on rpc-b. --ignore-eos keeps the model from ending at the first period. --simple-io puts the bytes on the stream before exit. If the bytes never show, the harness records exited-before-stop. That repeat is not this experiment.

sequenceDiagram
  participant C as llama-cli
  participant A as rpc-a
  participant B as rpc-b
  participant H as harness
  C->>A: RPC calls
  C->>B: RPC calls
  Note over C: prompt plus a few generated bytes
  H->>B: docker stop
  B--xC: socket closes
  C->>C: GGML_ABORT, exit 134
  Note over A,B: logs still only Accepted / Closed

What happened when the worker died

The directory is runs/E-002/stop-mid/20260924-135311. Three repeats. Each one carries a stop event, 4.0–5.0 s after the client started. The exit code is 134. This was not a harness timeout. About 5.5 s passed from stop to process exit. Stdout already held a paragraph after “Paris.” The sentence breaks off in the middle. A bit rude of the process. Also exactly what we needed.

Every stderr opened with the same line:

Remote RPC server crashed or returned malformed response

recv failed sits beside it, or send failed. That is the string the RPC client raises when send_rpc_cmd fails on a socket that was already open (llama.cpp contributors 2026). We had read the line in a newer tree than the image. The v0.4.1 binary printed it anyway.

Worker logs did not grow a death line. They hold the startup banner. They hold a warning that the server listens on 0.0.0.0. They hold Accepted client connection and Client connection closed.

Is that a Docker failure?

docker stop did stop the worker. The client saw a closed socket. send or recv failed. The process aborted. A container that was still serving would not have produced that pair.

This is not a graceful shutdown. The RPC server source we read has no SIGTERM handler. docker stop -t 5 sends SIGTERM. It may follow that with SIGKILL after five seconds (Docker Inc. 2026). The client spent about 5.5 s after stop before it died. It was still inside a call. It noticed the dead socket late. The server did not stay up on purpose to drain work.

The workers are not announcing that everything is fine. rpc-b is already dead. A farewell is difficult in that state. rpc-a does not talk to rpc-b. Its log is only its own connections with the client. “Accepted / Closed” is that journal. It is not a health report about the neighbor.

Exit code 134 is what this code does. A failed RPC command becomes GGML_ABORT. There is no retry. There is no plan to continue on rpc-a alone. The surviving worker does not pick up the dead worker’s share of the graph.

Conclusion

The client can tell a dead worker from a slow one. Only once the failure is fatal.

On a live worker, 50 ms of egress delay moves generation from the low-30s or low-20s tok/s band to about 11 tok/s. The request still finishes. Nobody writes that the path is bad. Hold times on the impaired worker move from about 0.3 ms to about 106 ms. The log text does not mention the change.

Kill the worker mid-generation. The client aborts itself. The exit code is 134. The line is Remote RPC server crashed or returned malformed response. It does not wait out the 180 s harness timeout. It does not finish the paragraph. The worker logs still do not name the death.

“The system does not notice” is too coarse. The token stream notices slowness. It stays quiet. The client notices death. It crashes. The workers’ logs notice neither.

What this note does not show

Worker logs at 200 ms were never captured. The same is true under packet loss. Only the client was saved there. A network black hole that leaves the process alive is also missing. In the source we read, recv has no timeout. That case can block instead of aborting. We did not run it.

vLLM, GPUs, and a large KV cache are outside this note. So is the discarded first attempt, 20260924-134809.

How to repeat

From failure-lab:

python3 -m harness run --scenario experiments/E-001/scenarios/d50.json
python3 -m harness run --scenario experiments/E-002/scenario.json

References

Docker Inc. 2026. Docker Stop. Https://docs.docker.com/reference/cli/docker/container/stop/.

Linux man-pages project. 2024. Tc-Netem(8) — Network Emulator. Https://man7.org/linux/man-pages/man8/tc-netem.8.html.

llama.cpp contributors. 2026. Llama.cpp. Https://github.com/ggml-org/llama.cpp.