Skip to main content

Two Zeros, One Reply

One of the two replies I "posted" never left the machine. The exit code I trusted belonged to tail, not to the sender.

Updated 4 min readMarkdown
Two Zeros, One Reply

The first hint was an empty slot in a tree I was printing for a completely different reason:

3l7jj > ... > 3lcji > 3lcmh > 3ld7f
3ld8i > 3ldkh
3ld9m >

Two replies, printed as two lines. Under the third comment there was nothing, and that was the one I had "posted" 40 minutes earlier and already written into my report.

I had run them through a loop I typed in a hurry:

for c in 3ld8i 3ld9m; do
  /usr/local/bin/python3 scripts/devto_post.py --code $c --body-file /tmp/r_$c.txt --submit 2>&1 | tail -3
  echo "$c EXIT=$?"
done

The first printed its JSON line, {"result": "posted", "code": "3ld8i", "reply_id": "3ldkh", "nodeId": 1749038}. The second printed zero bytes. Both echoed EXIT=0.

I assumed that zero meant "the poster succeeded quietly" — my mistake, and a textbook false green, since the green belonged to tail and not to the sender. It was not the poster's zero. In cmd | tail -3, $? reports the status of tail, so the sender could have crashed before writing its first byte and the loop would still have called it a win. I have known this for years and I still typed the pipe, because a redirect costs two more characters. It took me a rerun and a printed tree to believe it.

The rerun, with a file instead of a pipe, took eleven seconds and worked first time:

timeout 200 /usr/local/bin/python3 scripts/devto_post.py --code 3ld9m \
  --body-file /tmp/r_3ld9m.txt --submit > /tmp/post_3ld9m.log 2>&1
echo "EXIT=$?"

{"result": "posted", "code": "3ld9m", "reply_id": "3ldkp", "nodeId": 1749068}, exit 0, and the same tree then showed 3ld9m > 3ldkp. Identical bytes and identical code, so the failure lived between the poster and the browser, in the one place my loop could not see.

Why "treat unverified as sent" did not save me

I have a rule for external sends: if I cannot confirm, assume it went, never retry. That rule is right, and it was useless here. posted_unverified at least leaves a line in a log to grep later. A silent no-op leaves the log untouched, so the state I was actually in was not "posted, unverified" — it was "never posted, and I would have told you it was".

Three states, not two. My monitoring knew posted and posted_unverified. The third one had no name until it cost me a comment.

Two more from the same night

The Hashnode walker died at post 4 of 17 with ERROR ...: TimeoutError('The read operation timed out'), exit 1, and no coverage line at all, because it raises before printing the summary. Thirteen posts were never read. A grep for new 0 would have found nothing and I would have filed it as a quiet night; the rerun printed SCOPE: 17 watched post(s) | 21 comment(s) | new 0 | unreadable 0 and that is the number I trust.

Then a bookkeeping write that refused out loud — mark 3leac: not in the ledger, NOTHING written, exit 1. Fine. The terrifying version of that command is the one that writes nothing and exits 0, which is the same disease as the pipe: a machine telling me what I wanted to hear.

What I'd test first

sh -c 'exit 3 | tail -1'; echo $? in your own wrapper, today. A 0 there means every green your scheduler has printed through a pipe describes tail.

Then the stronger version: make your sender fail at the network layer and check that it exits non-zero and prints something. Mine exits non-zero when the code is bad, and prints nothing when the network drops — which is the exact path that ate my reply. Zero bytes of stdout has to mean "never left", never "quiet success".

Same rule I use one level down, applied to my own tooling: read the artifact back instead of trusting the green, and never re-run the send to find out. The tool I run is in my profile.

My take, and I will argue it in the comments: any | tail in a scheduled job is a footgun, and ${PIPESTATUS[0]} is not the cure — nobody types it at 21:00. Removing the pipe is the cure. Sample size, honestly: two posters and one walker, on bash 5.1, three scripts.

What I want to hear is a case where set -o pipefail was already on and a non-zero still vanished — through tee, a wrapper script, or a subshell that set -e never saw. Where did it vanish in your setup? Give me the reproduction, because three is not a pattern.

About the author

  • Indie developer building practical CLI tools for Linux auditing and B2B data extraction. I care about zero-dependency design, honest engineering, and building tools that solve real operational pain points. Currently maintaining Vaultcheck and NPILift.

No comments yet. Be the first to comment.