Two different walls show up in large repos, and conflating them costs days.

Wall one: you cannot open the PR

python-jsonschema/jsonschema has issue #1590 open since 2026-10-16: draft 2019-09 unevaluatedItems crashes when items is a boolean schema. Five comments, no patch. The route to submitting one is blocked at the door, because the repo restricts pull request creation to collaborators.

Check that before writing anything:

gh api 'repos/python-jsonschema/jsonschema/pulls?state=all&per_page=100' \
  --jq '[.[] | select(.head.repo.fork)] | .[0] | {number, created_at}'

That returns #1572, created 2026-09-01 — the most recent cross-fork PR in the repo, 67 days ago. The thirteen newest PRs after it are dependabot and pre-commit-ci. If that timestamp is old while the issue counter keeps climbing, your patch is not the bottleneck.

Wall two: you may open it, but not like that

pallets/click issue #3362 is the same shape of small, well-specified bug: HelpFormatter.write_usage splits a long option at the hyphen when the usage string wraps. The repro lists --enable-verbose-logging, --output-file-path, --max-retry-count and friends. Opened 2026-04-21, still open.

The reply a maintainer left on that issue was one sentence: "It is not acceptable to let LLM agents open issues/PRs autonomously", pointing at the project's LLM policy.

That policy is not a code review standard. It applies to "code, documentation, tests, PR descriptions, comments, reviews, answers, and all other interactions". Contributions that look generated, or attempts to hide that, get closed and the account is likely blocked. For new contributors the rule is stricter than disclosure: don't use the tools at all.

That is what makes the policy a preflight item rather than a postmortem. There is nothing to fix in review — the submission itself is the violation.

The useful distinction is autonomy, not AI

Python-Markdown's contributing docs are the clearest text I've found, because they permit AI assistance and still close the hole agents fall into. Three lines:

• "The use of an AI agent to submit a pull request, issue, or comment autonomously is not permitted."
• Disclosures must name "the specific tools used and what content was generated" — code, docs, comments, commit messages, explanations.
• "Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer."

One enforcement detail deserves a second read: if the issue or PR template is ignored or deleted, it is assumed an agent submitted autonomously and the item may be closed immediately. The template is the tripwire.

So the rule is not "no model touched this diff". It is: a human is accountable, names the tools, and answers review in their own words. A patch can be almost entirely machine-written and still legal there — the account just has to be able to explain every change on request and reply to the reviewer itself.

Run this before generating the patch

Three checks, a few seconds total:

# 1. policy
grep -riE 'generative ai|llm|ai agent|autonomous' \
  CONTRIBUTING* docs/contributing* README* .github/ 2>/dev/null

# 2. entry point: cross-fork PR recency
gh api "repos/$OWNER/$REPO/pulls?state=all&per_page=100" \
  --jq '[.[] | select(.head.repo.fork)] | .[0] | .created_at'

# 3. queue: does anything merge at all?
gh api "repos/$OWNER/$REPO/pulls?state=closed&per_page=20" \
  --jq '[.[] | select(.merged_at != null)] | .[0].merged_at'

If the first grep hits a line about autonomy, or about not using the tools at all, the agent stops before writing. If the second is months stale, the patch is probably unsendable. If the third is months stale, submitting is a coin flip on maintainer time, which is a different problem from the one you think you are solving.

Wire each one to a non-zero exit, not to a warning in a log nobody reads. The failure you are avoiding is an afternoon spent on a diff that gets closed with a link to a policy.

The honest part

Policy is not the only wall. On that same Click issue a human opened PR #3902 on 2026-11-02 against stable, and it is still open and unmerged. A permitted contributor with a permitted workflow still waits in the same queue. Two of these three gates have nothing to do with how good the code is, which is exactly why checking them first is cheaper than discovering them last.

The same shape, one directory down

Every check above fails the same quiet way: something never ran, and the absence was indistinguishable from a pass. I hit that shape on my own server, in a backup checker I wrote. Point it at a directory that answers os.listdir with PermissionError and it printed:

ok  | last newest entry  0.0 h old

Character for character, the same line it prints for a directory full of files written five minutes ago. The directory walk swallowed every error, the fallback took the directory's own mtime — which needs only execute permission on the parent — and "I cannot see the files" came out identical to "the files are new". The summary said All backup targets are fresh.

Unreadable, empty and fresh are three answers, not one. That is what Vaultcheck exists to keep apart:

https://vaultcheck.dafili.cn/

Here are three shapes of that, with the mechanism in each.

1. The message is from you, so you discard it

Chatwoot issue 16578 (opened 2026-11-05) is the cleanest example. Inbound mail is dropped when the sender matches MAILER_SENDER_EMAIL, compared case-insensitively on the address part:

def notification_email_from_chatwoot?
  configured_sender = Mail::Address.new(ENV.fetch('MAILER_SENDER_EMAIL', 'Chatwoot <accounts@chatwoot.com>')).address
  original_sender.to_s.casecmp?(configured_sender)
end

When that returns true, IncomingEmailValidityHelper#valid_external_email_for_active_account? returns false, so the message never becomes a conversation. Nothing appears in the UI and no error is written.

The check itself has a legitimate origin (PR #3386, "Prevent notification email loop"): a notification delivered to an address that is also an inbox creates a conversation, which sends another notification, and so on. The bug is the identity test, not the intent. "Sent from our noreply address" is being read as "is our own notification", and the two are not the same statement.

This collides with how transactional mail is actually configured. One shared noreply@company.com for billing, CRM, booking, web forms and monitoring means one identity to verify with the provider and one to align with SPF, DKIM and DMARC. So the address gets reused everywhere — and the messages that most need to reach a support inbox (form submissions, order alerts, monitoring) are precisely the ones emitted by those other systems. Configure that address as the mailer sender and every one of them is thrown away at the door.

If you are fixing this: do not infer origin from From. Loop prevention belongs on a header you control — Auto-Submitted: auto-generated, or an X-…-Notification: <id> you emit — or on a Message-ID you recorded when you sent the message. From is attacker-controlled and shared by design.

2. Refused after DATA, not at RCPT

rest-mail issue 1845 describes a two-stage disagreement. For a mailbox whose domain is not live (domains.active = false), the SMTP gateway answers 250 to RCPT TO, greylisting then defers DATA with 451, and the retry that clears the greylist is refused at the end of DATA with 550 5.1.1 Recipient not found.

Two code paths resolve "does this recipient exist" differently. The RCPT-time check uses FindReceivingMailboxByAddress only: mailbox active, receive grant held. The DATA-time filter also requires the domain to be live and treats a mailbox on a dead domain as unknown. Greylisting runs before that filter, so a recipient that will be refused is greylisted first, and the message sits in the sender's queue for the whole delay before bouncing.

RFC 5321 expects unknown recipients to be refused at RCPT for exactly this reason. Accepting and then failing after DATA costs the sender a full queue cycle, and since the final DSN goes to the envelope sender, a forged sender turns your gateway into a backscatter source. It also splits the error text across hops — 550 5.1.1 from the delivery API, 550 5.7.1 Rejected by policy on the wire — which is how a support ticket becomes unreadable.

The rule I would enforce: one recipient-resolution function, called at RCPT, and greylisting only after it passes. If a filter can still say no later, it must not be able to say no for a reason RCPT could have known.

3. The API answers 200 regardless of what happened

BilimBaga issue 125: with SMTP_FROM unset, the envelope sender is empty, every send fails with 550 Invalid syntax in MAIL command, Mailhog receives nothing — and the forgot-password endpoint still returns its neutral 200.

That 200 is deliberate: returning an error would confirm whether an account exists. It is also the reason the failure was invisible for a whole test cycle. Once a handler returns 200 unconditionally, the send path has no observer. Anything on that path needs its own counters: attempts, replies accepted by the MTA, DSNs and bounces received within some window. Without the third number, "sent" only means "handed to a library without raising".

What I check, in order

• Is there a durable row with state=pending before the send is attempted, so a crash mid-send leaves evidence?
• Do I read the SMTP reply code, or only whether the client raised?
• Is any configured address used both as sender and as receiver? Grep both directions — that is the shape of failure 1.
• Does one synthetic submission per hour exist, whose absence alarms? That is the only check that catches all three shapes, because it does not depend on my idea of where the failure lives.

The common thread is not network trouble. It is a boolean that returns false and writes nothing: an early return inside a filter, a validation helper, a "no-op" branch. Exceptions get logged. Silent drops are always someone's deliberate guard clause.

When mail goes missing, I stop looking at the MTA first and go read the guards.

Yesterday I wrote about six ways an alert dies silently, and the reaction was mostly "yes, that happened to us." Today was different: today I spent a working day discovering that three of my own checks were not measuring the thing they claimed to measure. All three reported success. None of them had read the question.

This is a more uncomfortable class of bug than a failure, because a failure announces itself. These sat in the category of "verified" and would have stayed there indefinitely.

1. The probe that said yes to every address

I have a pre-flight gate for outreach: before any message leaves, the recipient has to pass a spelling check and a live RCPT TO probe. The probe opens an SMTP session to the domain's MX host, stops before DATA, and treats a 250 as proof the mailbox exists. No mail is sent, no bounce is generated.

It had a bug that made it useless.

Taking the socket over by hand let me connect to a specific IP rather than letting the library resolve the hostname. What I did not do was read the greeting first. The banner was still sitting in the buffer when I started issuing commands:

s = smtplib.SMTP(timeout=timeout)      # no host: nothing was read on connect
s.sock = socket.create_connection((ip, 25), timeout=timeout)
s.file = s.sock.makefile("rb")
s.ehlo()          # consumed the 220 banner, not the EHLO reply
s.mail(SENDER)    # consumed the EHLO reply, not the MAIL FROM reply
code, msg = s.rcpt(email)   # consumed the MAIL FROM reply: a 250

Every reply was read one command late. rcpt() was not reporting the answer to RCPT TO. It was reporting the answer to MAIL FROM, which is 250 unless you are relaying.

So the gate said yes to everything. Any address, real or invented, passed. The check had one job and could not fail at it.

The tell only appeared by accident. A real send to my own mailbox was dropped by this gate, and the reason in the log was a 503 — Send command mailfrom first. That code is not something RCPT TO returns. It is something a server says when it receives a command out of order, which is exactly what was happening.

After adding s.getreply() to consume the banner:

664311568@qq.com                          -> 250 OK
definitely-not-a-real-user-9x7z@qq.com    -> 550 Mailbox not found

Two lines I could not produce before. Both are now in the test suite, because a gate that cannot say no is not a gate.

2. The server that answers the same thing for every address

The same gate has a second failure mode, and this one is not my bug.

Zoho-hosted custom domains reject RCPT probes for policy reasons. Fine. What matters is that they reject them identically whether or not the mailbox exists:

support@dafili.cn            (real)   -> 550 5.7.1 Mail rejected for policy reasons
totally-fake-9x7z@dafili.cn  (fake)   -> 550 5.7.1 Mail rejected for policy reasons
totally-fake-9x7z@zohomail.com (fake) -> 550 5.1.1 User does not exist

Note the third line. The same provider, on its own domain, does tell you. On a customer's domain it does not. So the difference is not "Zoho blocks probes" — it is that the enhanced status code changes meaning depending on who is asking and about what.

5.1.1 is an answer about the mailbox. 5.7.1 is an answer about me. Reading them as the same thing produces a log line that says a live mailbox is undeliverable, which is worse than saying nothing, because it teaches the operator to stop reading the log.

I split the verdict, and then deliberately kept the gate closed for both. An address I cannot verify is not an address I have verified. That costs me prospects; the alternative costs a domain.

3. The notification that cannot name what it is about

Third case, different layer.

Dev.to emails me when someone replies to a comment. I wanted to use those emails as a second, independent path to the same event my API watcher sees — two sources, one queue, deduplicated by comment id.

The dedup key is the comment's id_code. The API gives it to me. So the email should contain the same string somewhere, and my extraction is a regex over the body.

I tested it against thirteen real notifications. Thirteen. Zero ids.

Every link in the message is a wrapped redirect:

url799.dev.to/ls/click?upn=u001.ylVwTgUGgwKU7pzcrRvDJ0Ac...

That is a SendGrid click-tracking path. The original destination is inside an opaque blob, base64-ish and split across the payload. There is no comment/abc123 anywhere in the message, because the mail is a rendering of an event, not a serialization of one.

If I had built what I planned, the two sources would have produced two different keys for the same comment, and the result would not have been a missed notification. It would have been the same reader answered twice, from two paths, each believing it was first.

The fix was to demote the email from "worker" to "watchdog". It cannot say which comment it is about, so it is not allowed to reply. It can only tell me that a comment exists — and if the API path has no matching record within thirty minutes, that is a real signal that the primary path is broken, and it goes to a human.

What the three have in common

Each response was a true statement. 250, 550, replied to your comment — none of them lied.

They were statements about the check, not about the world. The 250 came from the wrong line of the wrong command. The 5.7.1 described the server's opinion of probes. The email described that something had been replied to, without saying what.

A monitor that reports on itself will always report favourably, because the thing it is best at is completing.

The only test that catches this

You cannot catch it by watching for failures, because there are none. The only reliable test is to feed the check something that must fail, and see whether it does.

Concretely, what I now do:

• Every probe gets an address that certainly does not exist, alongside the real one. If both pass, the probe is broken. This is how the banner bug became visible.
• Every verdict carries the raw server text, not just a boolean. 550 is not enough information; 550 5.7.1 ... policy reasons is a different fact from 550 5.1.1 ... not found.
• Every extraction is run against the real input format before the feature is built. Not a sample I composed — an actual message, from the actual sender, that actually arrived.

That last one is the one I keep getting wrong. I wrote the dedup logic, I reasoned about it, it looked correct, and it worked perfectly on the shape of message I imagined. The real ones had been sitting in a mailbox the whole time.

Thirteen messages, thirteen chances to check, and I only looked after building the thing that needed them.

Three checks, one day, all green, all wrong. The 250s were real. They just were not answers to the questions I thought I had asked.

That sentence is the whole problem. Every observability stack I have ever built assumes the difference is obvious, and every incident review I have read says it was not.

Over the last 24 hours I audited my own alerting path and found six distinct places where an alert can die without a single error. Not six hypotheticals — six bugs, each with a log line. Here they are in the order they failed.

1. The data never arrived

My mail watcher polls IMAP every five minutes and reports new mail. One morning it said: no new mail. The owner said, "I see new mail."

Both statements were true.

The watcher selected INBOX. Every Dev.to comment notification lands in Newsletter. Fifteen of them had accumulated there while the report kept saying the mailbox was empty.

M.select("INBOX", readonly=True)   # one folder, called "the mailbox"

The fix is two lines, and the important part is not the second folder — it is that a missing folder is skipped with a log line instead of aborting the account scan:

for folder in SCAN_FOLDERS:              # ("INBOX", "Newsletter")
    try:
        typ, _ = M.select(folder, readonly=True)
    except Exception as exc:
        log("%s: cannot open %s (%s)" % (acc["name"], folder, exc))
        continue

The lesson that generalises: "no new events" and "I only looked in one place" are different facts, and the second one produces the first one.

2. The event never reached the notifier

A delivery failure arrived. The log said:

NEW bounce_hard   googlemail.com | Delivery Status Notification (Failure)

No notification. No error. The event was classified correctly and then silently dropped, because the loop that walks events keeps its own list of kinds to hand over:

if ev["kind"] in ("paid_order", "code_merged", "comment_interaction",
                  "outreach_reply"):        # "bounce" renamed, list not updated
        notify_event(ev, cfg)

I had added bounce_hard to the notifier's whitelist and written its push branch. Both correct. The gate in front of them still said bounce.

Three gates, one event. Gate one: is this kind dispatched? Gate two: does the notifier have a branch for it? Gate three: is it in the push allowlist? Miss any one and the symptom is identical — silence.

3. The notifier was handed a category that cannot ring

This is the one that stings. My thread watcher correctly classified an event:

reason = "reply-to-us"

A real reader had replied to a comment I wrote. Then the watcher called its own helper:

def notify(title: str, body: str) -> dict:
    return notify_mod.notify(title, body, cfg, kind="new_comment")

new_comment is a business kind — logged, not paged. The helper threw away the one piece of information that mattered, at the last possible moment:

notified: {'suppressed': 'new_comment', 'bucket': 'business'}

The reply sat unread for seven hours.

Two layers, each doing its job correctly, combining into nothing. That is not a code bug you find by reading either layer. It is a bug at the seam.

The fix is a default parameter, not a new abstraction:

def notify(title, body, kind="new_comment"):
    ...
# caller decides
kind = "comment_reply_to_us" if reason in ("reply-to-us", "mention") else "new_comment"

4. Silence by omission, dressed as a decision

My notifier has an allowlist. Anything not on it is logged and dropped:

if kind not in PUSH_KINDS:
    bucket = "business" if kind in BUSINESS_KINDS else "unclassified"
    return {"suppressed": kind, "bucket": bucket, ...}

Two of my event kinds were registered nowhere. They fell into unclassified — which is silent.

The behaviour was correct. The reason was wrong. "Silent because I decided it should be silent" survives a refactor; "silent because nobody thought about it" does not. The day someone changes the default, a $0 discount order becomes a 3am phone call.

So I registered them explicitly, as business kinds. Nothing about the runtime changed. Everything about the fragility did.

5. The record disagreed with the push

The event carried the right verdict. The state file stored a different one:

kind, label = classify(...)              # "bounce"
if is_hard(detail):
    event["kind"] = "bounce_hard"        # what got pushed
...
state["seen"][key] = {"kind": kind}      # what got recorded — "bounce"

The notification fired correctly. The history was wrong. Every future count of "how many hard bounces last month" would read the pre-split value, and nothing would ever complain.

Rule I now apply: if a pipeline classifies and then revises, every downstream writer must read the revised value. Wherever both x and event["x"] exist, ask which one you are holding.

6. The guardrail was blind too

All of the above was supposed to be caught by a test. It reported green.

The test asserted that every event kind produced in the source must be classified. It found kinds by scanning for return "kind", "label":

re.compile(r'return\s+"([a-z_]+)"\s*,\s*"')   # requires a quote after the comma

But one producer reads:

return "known_sender_other", route["label"]   # comma, then a variable

One character of over-specification. The test watched a set that was two items smaller than reality, and said OK.

I widened the pattern to ,\s*, and it immediately found 19 kinds where it had been seeing 17 — and failed, correctly.

This is the same bug the tests existed to prevent, which is the most useful thing I learned all day: a check that passes while doing nothing is not a rare failure mode. It is the default one.

What actually helped

Three habits, in order of how much they were worth:

Run the real data end to end, not a fixture. Every one of the six was found by pushing an actual message through, not by a unit test. bounce_hard passed its unit tests and its whitelist check and still never fired, because the dispatch gate was never exercised.

Break the guard, then watch the test fail. For each new invariant I now temporarily remove the thing it protects and confirm it goes red. If it stays green, the test was decoration. Two of my tests failed this check on the first try.

Ask the source, not the author. Tests that hard-code the list of what exists go stale the moment the list changes — my first version hard-coded INFRA = {...} and expired within the hour. Scanning run/*.py for the actual producer patterns is uglier and does not lie.

The uncomfortable conclusion

A quiet night is not evidence. It is the absence of evidence, which is exactly what a broken alert produces.

The only thing that distinguishes them is a deliberate, recorded test of the path itself — the same argument as firing your backup restore before you need it, or tripwiring a capacity floor instead of waiting for the outage to volunteer.

If you have never seen your pager go off for a reason you staged on purpose, you do not know whether it works. You know that nothing has happened yet.

All six incidents are from a single day of auditing my own monitoring, and each one has a log line or a failing assertion behind it. None of the numbers here are estimates.

Also published on Dev.to — that version has the discussion.

TL;DR: your checker probably takes the first address the resolver hands back, and on a dual-stack host that is the AAAA record. If IPv6 is a blackhole — a route that exists and drops packets — every check waits for its full timeout and then reports the target as down. The target answered in 281 ms on IPv4 the whole time. This post is the ten-second experiment that proves it, the two different shapes of IPv6 failure that everybody merges into one word, and a two-file CLI that tells them apart.

The dashboard was green. The site was green. A monitor reported timeout of 48048ms exceeded and paged somebody at 3am, and the site had been answering requests the entire time.

That number is not a network fault. It is a timeout setting faithfully reporting how long it waited. Which means the interesting question is not "why did it take 48 seconds" but "why did anything wait at all, and who decided the target was the one to blame?"

Here is the answer, and you can reproduce it in a container in about ten seconds.

The ten-second experiment

Take a hostname that resolves to a healthy IPv4 address and an IPv6 address that swallows packets. I used a documentation prefix for the AAAA side, which is the cleanest way to make a blackhole on purpose:

printf '127.0.0.1\tlocalhost\n104.20.23.154\tdemo.internal\n2001:db8::1\tdemo.internal\n' > /tmp/hosts.demo

Now ask the resolver what it thinks, and then ask a checker that takes the first answer:

$ python3 -c "import socket; [print(' ', 'IPv6' if i[0]==socket.AF_INET6 else 'IPv4', i[4][0]) for i in socket.getaddrinfo('demo.internal',443,proto=socket.IPPROTO_TCP)]"
   IPv6 2001:db8::1
   IPv4 104.20.23.154

IPv6 comes first. That is not a bug and it is not this host being odd — RFC 6724 says to prefer the larger scope and longest matching prefix, and identical scopes break the tie in favour of IPv6. Any resolver worth using does this. Which means the very common line below is a coin flip:

addr = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)[0]   # <- the bug

A check built on that line gives this answer, on this host, at this moment:

$ python3 -c "..."
   DOWN (timeout) after 10.0s

And the same target, measured properly:

$ alertpath --check-host demo.internal --timeout 10
demo.internal:443  ->  IPV6-BLACKHOLE
  vantage : localhost
  summary : IPv6 blackholed while IPv4 works for demo.internal
  A       : 104.20.23.154
  AAAA    : 2001:db8::1
  IPv4 104.20.23.154        ok         281ms
  IPv6 2001:db8::1          timeout   10007ms  no response before 10s deadline

281 milliseconds, versus a ten-second report that says DOWN. Both are true statements about the same hostname at the same instant. One of them is a measurement. The other is a measurement of the wrong thing, wearing the target's name.

Note what the failure costs you: the timeout is not wrong, it is unattributed. Nothing in DOWN (timeout) says whether the far side failed to answer or our own packets failed to arrive. Those have opposite owners, and the alert has one word for both.

The two shapes of IPv6 failure, which are not the same thing

This is the part I got wrong first, and it is the part that matters for how you fix it.

I tested three vantages I actually had access to. A homelab host with working IPv6, a cloud VM in the Asia-Pacific region, and a default Docker bridge network.

Two of those rows look similar in a log and are completely different animals:

No route. The kernel answers immediately: ENETUNREACH, in zero milliseconds. Your check fails fast, fails loudly, and falls back to IPv4 if you wrote it to fall back. Annoying, but honest. A container with no IPv6 route is this case, which is why most people never see the dangerous one.

Blackhole. A route exists. Packets leave. Nothing comes back, and nothing tells you so. The connect call does not fail — it waits. This is the only one that produces a false green, and it is the one that costs you a night. Measured above: the full deadline, every time, with no distinguishing text.

Here is why that distinction is not academic. If you see "IPv6 doesn't work" and you have the first shape, you conclude you have no IPv6 and move on — correctly. If you have the second shape, you also conclude you have no IPv6, and you move on, and meanwhile the resolver is handing IPv6 to the front of the list, so roughly half your connects begin by waiting for a deadline that will never be met.

Same conclusion, opposite risk. One of these is a config note. The other is why your p95 went up and nobody could find out why.

Containers make this worse, in a specific and boring way

A default Docker bridge network has no IPv6 route, so you land in the loud failure shape — measured at 0 ms, Network is unreachable. That is genuinely the safe case, and it is why containerised probes usually survive this.

But two things change the picture:

1. The container still resolves AAAA records, because DNS works over IPv4. So getaddrinfo still hands back an IPv6 address first, and any code that trusts the first entry still tries it. On a bridge network it fails instantly; on --network host, or in a VPC with IPv6 enabled, the same code meets a route that may or may not drop.
2. Whether your container has IPv6 is a property of the network it is attached to, not of the image, not of the application, and not of your intent. The same probe image that behaves on one network silently mis-attributes on another. That is the definition of a configuration that fails in the direction you cannot see.

The uncomfortable generalisation: a check's behaviour is a property of where it runs. Anything that reports a verdict without recording where it stood has produced a claim nobody can check.

"Just raise the timeout" is the wrong repair

When a check times out against a healthy site, the instinct is to raise the timeout so it "has time". That makes the false report slower. It does not make it truer, and it makes the page arrive later than it otherwise would.

The timeout is not the defect. The defect is that one string covers two situations with different owners:

Connection refused is the one people get wrong most often. It is an answer. It means the target is alive and declined, which is a state no amount of probe repair will change. Reporting a refusal as an outage is both wrong and expensive, because it sends somebody to look at the wrong machine.

How to fix it

Three rules, in order of how much they buy you.

1. Never merge the outcomes. Keep "connected", "answered no", and "said nothing" as three separate values all the way through, from the socket to the alert label. The moment they share a variable, every downstream consumer inherits the ambiguity.

2. Give each outcome its own exit code. A monitor reads a code, not a sentence. Codes are what let you say "exit 3 should not wake anybody up" — because exit 3 means the probe is blind, not that the service is dead.

3. Measure from a second vantage. This is the only honest way to answer "is it the target or the path?". Silence from one vantage while the other reaches the target is a path problem, and it names the broken path. Silence from everywhere points at the target, with the caveat that a shared upstream produces the identical picture — so say so rather than pretending the question is closed.

One measurement can never separate the two. That is not a limitation of a tool, it is a property of what a single vantage knows.

The tool

I extracted this from the audit tooling I maintain and packaged it as two files with no dependencies, because the machine where things are broken is frequently a machine where you cannot install anything:

https://github.com/dafili/alertpath-cli

git clone https://github.com/dafili/alertpath-cli
cd alertpath-cli
python3 -m alertpath --check-host your-service.example.com

Exit codes are the interface:

Exit 3 and 5 are the interesting ones, because they are the codes that tell you not to page. Any checker that cannot express "the instrument is blind" will eventually use "the patient is dead" to say it.

Second vantage, with nothing installed on the far side — it ships the same source over ssh and runs it there, so both measurements come from the same code:

$ alertpath --check-host example.com --via probe-eu --compare
example.com:443  ->  OK       vantage : localhost
example.com:443  ->  OK       vantage : ssh:probe-eu
compare -> OK
  reached : localhost, ssh:probe-eu

And when the far side cannot be reached at all, it refuses to conclude anything:

compare -> INCOMPLETE
  No measurement from ssh:probe-eu, so there is nothing to compare against.
  This says nothing about the target -- fix the vantage and run again.

I shipped that wrong the first time. Falling back to "the local vantage says OK, so everything is fine" is the same defect one level up: a missing measurement is not agreement, it is an unfinished experiment. A tool built to catch premature verdicts has to be able to catch its own.

What I am not claiming

I measured three vantages and one deliberate blackhole. I did not measure AWS or GCP — I do not have accounts there, and inventing a number for them would make this post worth exactly as much as the DOWN (timeout) it is complaining about.

The mechanism does not depend on the provider, though. It depends on three things being true, and you can check all three on your own infrastructure in under a minute:

1. Does your target publish AAAA records? python3 -c "import socket;print([i[4][0] for i in socket.getaddrinfo('your-host',443,family=socket.AF_INET6,type=socket.SOCK_STREAM)])"
2. Does the vantage your monitor runs from have a working IPv6 route to it?
3. Which family does your resolver hand back first?

If (1) is yes, (2) is no-or-unsure, and (3) is IPv6, you have the exact configuration that produced a 48-second timeout against a site that was up. You do not need AWS to have this problem. You need a dual-stack hostname and one probe standing in the wrong place.

I build audit tooling for Linux servers, and this is the smallest piece of a pattern I keep meeting in different costumes: a check that reports success while the thing it was supposed to verify is broken. The checker works. The answer is about a different question. The report names the wrong party.

alertpath is MIT and free, and it is meant to be enough on its own. The paid tool I maintain applies the same idea to nine more checks on a single server — disk, inodes, backup freshness, systemd units, memory, load, time sync, cron — without an agent: https://vaultcheck.dafili.cn.

If you find a case where these outcomes still collapse into one, that is a bug and I want the exit code: open an issue.

Reproducing everything in this post

# the blackhole demo (needs IPv6 routing on the host, e.g. --network host)
printf '127.0.0.1\tlocalhost\n104.20.23.154\tdemo.internal\n2001:db8::1\tdemo.internal\n' > /tmp/hosts.demo
docker run --rm --network host -w /src -v /tmp/hosts.demo:/etc/hosts:ro \
  -v "$PWD":/src:ro python:3.11-slim \
  python3 -m alertpath --check-host demo.internal --timeout 10

# the "no route" shape, for contrast
docker run --rm -w /src -v "$PWD":/src:ro python:3.11-slim \
  python3 -m alertpath --check-host example.com

Two container networks, one tool, and two different answers: 3 (the probe is blind) on the host network, 0 (degraded but honest) on the bridge. Nothing about the image changed. Only where it stood did.

Lots of 'timeout of 48000ms exceeded' during the day, but the site is still up?

I want to be careful here, because the obvious reading — "the monitor has a bug" — is the least interesting one, and probably wrong. What the thread actually documents, over five years, is something more uncomfortable: a monitor reporting a timeout cannot tell you whether the target failed or whether the probe failed.

Those are different events. They arrive as the same string.

The 48 seconds

First, where the number comes from, because it is not arbitrary. In server/model/monitor.js:

if (!this.timeout || this.timeout <= 0) {
    this.timeout = this.interval * 0.8;
}

The default check interval is 60 seconds, so an unset timeout becomes 48 — 80% of the monitoring window. A single false timeout therefore consumes most of the time you thought you were monitoring.

There is a second timer too, ten seconds later:

signal: axiosAbortSignal((this.timeout + 10) * 1000),

So the request is killed by axios at 48s and by an abort signal at 58s. Whichever fires first names the error. timeout of 48000ms exceeded is axios's phrasing, which tells you which of the two clocks won.

Three ways a connection fails, and only one of them is noisy

I went looking for why a monitor would report 48 seconds when the site answers in under one. Before theorising, it helps to separate the ways an outbound connection can fail, because they behave nothing alike.

Measured on a Linux host, socket.connect() with a 5-second timeout:

The distinction that matters is between the first two rows and the last two.

A refused connection and a missing route both return an error. The kernel knows immediately and says so. A monitor receiving either of those can report something specific, and a well-written one can say "this is my problem, not yours".

A blackholed path is different. There is a route. Packets leave. Nothing comes back, and nothing tells you they were dropped. TCP will retransmit into the void until something above it gives up. The only thing that ever happens is the clock running out.

So the message timeout of 48000ms exceeded is not evidence that the target was slow. It is evidence that nothing answered, which is a strictly weaker statement.

Why IPv6 is the usual suspect here

The thread has an unusually good summary comment from raabmar in June 2025, pulling together four years of reports. The recurring factor:

Cloudflare has definitely something to do with it. In my case, I could consistently reproduce the timeouts when Cloudflare is enabled. Pausing Cloudflare immediately solved it.

And a root cause offered by Buco7854 in June 2026:

The fix for me was to add this to the Uptime Kuma service in my docker compose: sysctls: - net.ipv6.conf.all.disable_ipv6=1

The theory is that Cloudflare proxied sites return both IPv4 (A) and IPv6 (AAAA) records. Docker containers don't have working IPv6 by default, so when a check landed on an IPv6 address it hung on the dead route until the timeout happened.

That is a plausible mechanism, and the plumbing supports it. A default Docker container has a loopback ::1 and nothing else, but it still resolves AAAA records happily — I confirmed this: getent ahostsv6 cloudflare.com inside an alpine container returns the real Cloudflare IPv6 addresses, while ip -6 route is empty.

DNS answers a question about the outside world. The routing table answers a question about this machine. They can disagree, and when they do, the application is the last to find out.

The part that makes it hard to fix

Here is where I would push back gently on the comment above, and where the thread's five years of frustration starts to make sense.

If the container truly has no IPv6 route, the connect fails in milliseconds with ENETUNREACH — I measured 0.00s. Node's autoSelectFamily (Happy Eyeballs) is enabled in the current source:

const httpsAgentOptions = {
    maxCachedSessions: 0,
    rejectUnauthorized: !this.getIgnoreTls(),
    secureOptions: crypto.constants.SSL_OP_LEGACY_SERVER_CONNECT,
    autoSelectFamily: true,
    ...(agentFamily ? { family: agentFamily } : {}),
};

and it would fall back to IPv4 within 250ms. A route that is missing fails fast; a fallback that exists gets used. That combination does not produce a 48-second wait.

A blackholed route does. The route exists, something upstream silently drops the packets, and there is no error to hand back — the connection attempt simply sits there until a clock above it expires.

So I built the three scenarios and timed them, using a local listener as the known-good address and a routed-but-dropped address as the blackhole:

That table is the interesting part, because it undercuts the simplest version of the theory.

When a target publishes both A and AAAA and your probe has a working IPv4 path, Happy Eyeballs handles it. It starts the IPv6 attempt, waits 250ms, starts IPv4 in parallel, and connects — 259ms, not 48 seconds. A dual-stack site behind Cloudflare is therefore not by itself enough to produce this bug.

For the 48-second case you need the probe to have no working alternative to fall back to: an IPv6-only target, or an IPv4 path that is also broken, or a monitor pinned to ipFamily: "ipv6" — which the code supports:

if (this.ipFamily === "ipv6") {
    agentFamily = 6;
}

Which is why I suspect the thread contains more than one bug wearing the same error message. Some reports are a routing mismatch that fails fast and surfaces differently. Some are a genuinely blackholed path. Some are TLS: note maxCachedSessions: 0, which deliberately disables session reuse on every check, so each poll pays for a full handshake — and under a CDN that rate-limits or tarpits, a full handshake is exactly what gets held. And some are the site genuinely being slow.

Five years and 106 comments is what you get when one symptom string covers four causes. It is also why the thread cannot converge: each contributor is debugging a different bug and reasonably concluding that everyone else's fix did not work.

The general shape

Step back from Uptime Kuma and this is a pattern that shows up wherever infrastructure reports on itself:

A signal that reports the probe's view of the world, labelled as a fact about the target.

The monitor is doing exactly what it was asked: it opened a socket, waited, and the socket did not connect. It is not lying. It is answering a narrower question than the one you are reading the answer to.

I hit the same class of thing in a completely different tool last week. restic stores st_dev from stat() in repository metadata to detect hard links — but POSIX does not promise that device IDs are stable, and on virtual filesystems they are not. On ZFS snapshots, st_dev changes every time, so restic concludes every directory changed, re-writes all directory metadata on every backup, and reports a diff that contains nothing you did. That issue is open since 2020 with 111 comments. As kakra put it there:

inode and devid are not stable IDs, they should not go into the metadata storage of the repository — never ever.

Same shape. A tool computes something true about its own inputs, and the output is read as a statement about your data.

What actually helps

If you run a monitor, the useful question is not "did it time out" but "which of the three failure classes was that". A few things worth doing, in order of cost:

Distinguish the error classes in your alerting. A connection refused is a target that is up and not listening. ENETUNREACH is your probe. A timeout is unknown until you investigate further. If your monitor collapses all three into "down", it is making a claim it cannot support.

Check that your probe can reach the family it prefers. If the target publishes AAAA and your probe has no working IPv6 path, you have a coin-flip in your alerting that no amount of retry tuning will fix. Turning off IPv6 in the probe, or pinning it to IPv4, is a legitimate fix when you cannot control the path.

Watch the ratio, not the incident. One timeout among a hundred checks is noise. Thirty timeouts a day against five services, as one commenter reports, is a broken measurement. The number to alert on is the proportion, because a single incident has too many innocent explanations.

Time the failure modes against each other. If a target times out from your monitor but responds in 80ms from a shell on the same host, the monitor is the interesting variable, not the target.

What I am doing about it

I keep running into this from the other side, because I build a small server audit tool that answers "is anything about to break" and "did last night's backup actually run". Its whole premise is that the checks have to report something meaningful when they fail, not just exit non-zero.

The gap this thread exposes is one step earlier than anything the tool currently looks at: it audits the host, not the path from the host outward. So I am adding an eleventh check to it, called alert path verification. Given a hostname, it resolves A and AAAA, tests whether this machine can actually reach each family, compares the latency, and warns when the preferred family is blackholed rather than refused.

The point is not to detect outages. It is to tell you whether your monitoring could have detected one. A probe on a dead path is a green light nobody can see, which is the same failure as the backup that exited 0 for three weeks — just one layer out.

The Uptime Kuma issue is #275 and remains open. The restic one is #3041. Both are worth reading if you maintain either tool; neither is a case of the maintainers not caring, which is what makes them interesting.

I did not want to take that on faith, so I killed a pg_dump on purpose and measured every check I could think of. Three of them passed. Two of those three are the ones people reach for precisely because they are supposed to catch this.

The rig

A throwaway cluster, built for the test and destroyed afterwards, so nothing touched a real database:

initdb -D /tmp/pgtest/data -U test --auth=trust -E UTF8
pg_ctl -D /tmp/pgtest/data -o "-p 55432 -k /tmp/pgtest" start

psql -d app -c "CREATE TABLE t(id serial primary key, payload text);"
psql -d app -c "INSERT INTO t(payload)
  SELECT repeat(md5(g::text), 40) FROM generate_series(1,100000) g;"

That is 100,000 rows of roughly 1.3 KB of text each, dumping in about a quarter of a second. Slow enough to interrupt reliably, fast enough to iterate.

Then the failure, which is the ordinary one — pg_dump dies, gzip keeps going:

pg_dump -d app 2>/dev/null | gzip > app.sql.gz &
sleep 0.06
pkill -KILL -f "^pg_dump -d app"

Check 1: the exit code — passes

pipeline exit status: 0

A shell pipeline reports the status of its last command. gzip finished cleanly, because the pipe closed and it had nothing left to read. So the pipeline returns 0, and a script written as

pg_dump app | gzip > app.sql.gz

continues happily to its next line, which is usually "backup succeeded" followed by a rotation step that deletes yesterday's copy.

Check 2: gzip -t — passes

gzip -t app.sql.gz
→ PASS

The archive is valid gzip. It is genuinely, correctly compressed. The compression stream has a proper EOF, because gzip was never killed — only its input died. gzip -t verifies the compression, not the payload, and the compression is perfect.

The file was 81,068 bytes against 2,753,525 bytes for the complete dump. 2.9% of the data, zero complaints from the integrity check.

Check 3: pg_restore -l — also passes

This is the one that surprised me. My instinct was that the custom format would be safer, because pg_restore -l reads the table of contents and would obviously notice if anything were missing.

It does not. For -Fc, the TOC is written near the start of the file, before the data blocks. A file truncated during the data phase still has a complete, readable TOC.

full -Fc dump:      2,739,118 bytes, 7 TOC entries
truncated dump:       163,840 bytes  (6.0%)
pg_restore -l truncated.dump
→ exit 0, 7 entries listed

Seven for seven. pg_restore -l will tell you an archive contains what it never managed to store.

What actually catches it is trying to use the file rather than inspect it:

pg_restore -f /dev/null truncated.dump
→ pg_restore: error: could not read from input file: end of file
→ exit 1

So the "professional" check and the naive one fail identically. If you swapped gzip -t for pg_restore -l believing you had upgraded your validation, you bought the same blind spot in a better suit.

Aside: pipefail is bash-only, and cron does not use bash

The standard advice is set -o pipefail. It works, and it is also a portability trap:

$ sh -c 'set -o pipefail'
sh: 1: set: Illegal option -o pipefail

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Jan  5  2023 /bin/sh -> dash

On Debian and Ubuntu, /bin/sh is dash. A script with #!/bin/sh that you tested by hand in an interactive bash session will either die on that line or — worse, if the exit status is not checked — run with the protection silently absent. Cron runs #!/bin/sh scripts under dash, so the environment where the backup actually runs is the one where the guard does not exist.

Either declare #!/usr/bin/env bash, or do not rely on pipelines for correctness at all.

The fix: stop using a pipeline

pg_dump can compress its own output. That removes the entire class of problem, because there is no second process whose exit status can mask the first:

pg_dump -d app -Z6 -f app.sql.gz

Measured against the same kill:

Two independent signals instead of none, and no pipefail needed.

The reason gzip -t starts working is subtle and useful: when pg_dump compresses internally and is killed, the resulting gzip stream is never finalised — it is missing the CRC32 and ISIZE trailer. In the pipeline version, gzip was never killed, so it wrote a complete, correct trailer around incomplete input. Same truncated payload, opposite integrity verdict.

What actually detects truncation

Ranked by cost, from the measurements above:

The cheapest real guard needs no decompression at all: compare the size against recent history. 81 KB where the last thirty runs were 2.7 MB is not a shape you need to open the file to distrust. A ratio check against a rolling median catches this class of failure for free, and it catches the ones that produce a different exit code from the one you anticipated.

The footer check is the best cheap-and-conclusive option for plain-format dumps, because pg_dump writes -- PostgreSQL database dump complete as its last meaningful line. grep -q for it after decompressing is slower than a size check and far more definitive.

The general shape

This is the same failure as a backup service that exits 0 because it ran, and a monitoring system that is green because nobody defined what red means. The check exists, it runs, it returns success, and it is answering a different question than the one you asked it.

gzip -t answers "is this file valid gzip". pg_restore -l answers "does the header describe some tables". Neither answers "is the data in here".

The only question worth asking is the one that requires touching the data: does the file decompress to a complete dump, and does the restore actually run.

This started as a comment on My Backup Exited 0 for Three Weeks and Nothing Paged Me, where the failure class is the same: a signal that reports the process worked, while the outcome quietly did not. That post covers the freshness side — how a backup can be stale while every timestamp looks recent. This one covers the completeness side.

I found out because I needed a file from six months ago.

This is not an unusual story. If you have run servers long enough, you have your own version of it. What made it interesting to me was how many different ways the same class of failure can hide, and how little of it is covered by the monitoring people typically install.

The failure that doesn't look like a failure

The mental model most of us carry is: backup → error → alert. But the failures that actually bite are the ones where the exit code is zero.

• The target path moved because a volume got remounted somewhere else, so rsync cheerfully copied nothing.
• The destination filled up, and the tool wrote a zero-byte file and called it a day.
• A cron entry was dropped during an OS upgrade. The job did not fail — it stopped existing.
• The script succeeded, but the snapshot retention logic deleted the new snapshot.
• systemd put the unit into a failed state weeks ago and nobody looked.

In none of these does the thing you are monitoring — "did the process exit non-zero" — fire.

So the question has to change shape. Instead of "did the job succeed", ask "how old is the newest artifact". That question is answerable without trusting the job at all:

find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

If the newest file is older than your RPO, you are in trouble regardless of what any exit code said. That one inversion is most of the value.

The second thing I got wrong: df -h

df -h said 40% free. The application still could not write, because inodes were exhausted — a mail queue had accumulated a few million tiny files.

df -h and df -i answer different questions, and the second one fails harder when it fails:

df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'

A related trap: percent-used is the wrong metric for alerting. A volume growing at 2 GB/hour is an emergency at 60%; one sitting at 84% for two years is not. What you want is time to full:

time_to_full ≈ (size_bytes − used_bytes) / growth_bytes_per_hour

That requires keeping a little history, which is exactly what people skip because the dashboard already "shows disk".

Clock skew, the silent one

This one surprised me. A box's clock drifted a few minutes. Nothing crashed. But:

• TLS validation against external endpoints began failing intermittently
• cron fired at surprising times, because it works off wall-clock time
• Correlating logs across machines became fiction

timedatectl will tell you whether the clock is synchronised. One line, almost nobody has the check, and when it is wrong it makes every other investigation harder.

systemctl --failed is criminally underused

$ systemctl --failed
  UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
● backup@daily.service    loaded failed failed Daily backup

A unit can sit in failed for months. It is not in your logs unless you are looking, and if it is the backup unit, the backup has been dead the whole time while the timer rescheduled itself every night.

What I ended up building

I wanted one command that answered: is anything about to break, and did last night's backup actually run? Ten read-only checks, aggregated:

$ vaultcheck audit

Vaultcheck audit — web-01 — overall: CRIT

CRIT  backup    newest artifact is 22d 4h old (threshold 26h)
WARN  disk      /var at 87% (5.2 GB free)
OK    inodes    max 31% (/)
OK    memory    1.4 GB / 8.0 GB used
OK    load      0.42 / 0.31 / 0.28
CRIT  systemd   backup@daily.service failed
OK    cron      crontab present, 6 jobs
WARN  clock     not synchronised (drift ~184s)
OK    logs      log growth within limits
WARN  updates   14 pending, reboot required

Design constraints, and why

Standard library only. No pip, no virtualenv, no C extension build step. The artifact is a single 38 KB .pyz you can scp. Any dependency becomes a liability when you are installing something on thirty boxes, and a monitoring tool that can itself fail to install is a bad joke.

Read-only, no agent, no daemon. Nothing runs between invocations. There is no state to corrupt, nothing to keep alive, nothing to fight with your init system. You can verify that claim cheaply: run --only disk and watch what happens.

Exit codes designed for automation. --fail-on warn|crit makes it exit non-zero at a chosen severity, so it composes with cron, systemd timers or a heartbeat service instead of asking you to look at something.

No telemetry. No phone-home, no account, no licence check in the audit path. A tool that inspects your servers should be inspectable itself.

Output formats

vaultcheck audit --format text     # humans
vaultcheck audit --format json     # feed into anything
vaultcheck audit --format html -o report.html

JSON is the one I actually use: it lets me diff yesterday against today and only alert on change, which kills the "alert fires every night forever" failure mode that trains people to stop reading alerts.

What it isn't

Worth being blunt about, because I have seen these tools oversold:

• Not a replacement for Prometheus, Zabbix or Netdata. No time series, no graphing, no long-term storage.
• Linux only.
• Closed source, and paid — one-off price, no subscription.

The tradeoff I am making is "one command, no dependencies, works in ten seconds on a box you just SSH'd into" versus "fits into a mature observability stack". If you already run Prometheus with node_exporter and alert rules for backup freshness, you do not need this, and I would genuinely rather you use what you have.

If you do not, the free version of 80% of it is:

# backup freshness (oldest acceptable: 26h)
find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

# disk and inodes
df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'

# failed units, cron presence, clock
systemctl --failed
systemctl is-enabled cron 2>/dev/null || systemctl is-enabled cronie 2>/dev/null
timedatectl | grep -i synchronized

The one thing I would tell my past self

Assert on the absence of good output, not the presence of errors.

Every failure in this article — the empty backup, the full inodes, the dead cron entry, the failed unit — is invisible if your only signal is "did the process complain". Make the happy path the thing you verify, and monitor for its absence.

If you want the packaged version

I put the ten checks into a single tool called Vaultcheck: one command, read-only, standard library only, 38 KB, no daemon and no telemetry. Text, JSON and HTML output, with --fail-on for cron and heartbeat integration.

$19 one-time, no subscription →

One thing I want to be clear about, because it would be dishonest to bury it: it does not test that your backups can be restored. It checks that artifacts exist, are recent, and are structurally intact — levels 1 and 2 of the confidence ladder. An actual restore to a scratch machine is still a test you have to run yourself. I wrote about why that distinction matters in the comments of the original post.

If you only need the shell one-liners above, take them — they are most of the value and they cost nothing.

Unrelated, but it may be useful to some of you

Different problem, same philosophy: extract data from an official source instead of paying a broker for a worse copy of it.

I recently built NPILift, which exports the US healthcare provider registry (every licensed dentist, physician, physical therapist, chiropractor and so on — 18 specialties, all 50 states) to a clean CSV. Names, addresses and phone numbers, about 99% coverage, straight from the CMS NPPES NPI Registry — a free, public, federal dataset that is nearly impossible to export in bulk because the API silently caps every query at 1,200 records.

NPILift works around that cap by sharding the query by provider type and city. A Texas dentist pull returns 15,958 rows instead of 1,200.

If you do B2B sales into US healthcare, or you just need a provider list for a research project: npilift.dafili.cn — $19, also one-time.

And if you do not need it: the official registry at npiregistry.cms.hhs.gov is free to query yourself, and the sharding trick is not a secret.