Twelve Green Tests, One Missing Key

The most expensive bug I shipped this quarter was not a logic error. It was a deleted key, and the test file that covered it agreed with the deletion.

Here is the whole failure in two payloads. This is what a tool should look like on the wire:

{ "type": "function", "function": { "name": "find", "parameters": {} } }

This is what left the gateway:

{ "name": "find", "parameters": {} }

Same name. Same schema. No type, no function. Every server on the other end that reads tools[].function.name found nothing where it looked, and did the reasonable thing with an empty tool list: nothing. No error, no warning. HTTP 200, an ordinary assistant message, zero tool calls.

Two branches, three lines apart

A provider adapter rewrites outgoing tool names, and while it was in there it also normalised regex patterns inside the parameter schemas. Two sibling branches:

if (fn !== undefined) {
  const fnOut = { ...fn, name, parameters: completeRequiredSchema(fn["parameters"]) };
  return normalizeRe2Declaration(fnOut);                    // wrapper gone
}
if (custom !== undefined) {
  const customOut = { ...custom, name, format: completeRequiredSchema(custom["format"]) };
  return { ...rawTool, custom: normalizeRe2Declaration(customOut) };  // wrapper kept
}

The custom branch reassembles the outer object and nests the normalised declaration back inside it. The function branch returns the declaration itself and loses everything that wrapped it. A one-line asymmetry that is completely invisible until you print the outbound bytes.

Why the tests could not catch it

Two tests covered that exact rewrite. Both of them read the result like this:

const tool = (payload["tools"] as Array<Record<string, unknown>>)[0]!;
expect(tool["name"]).toBe("find");
const params = tool["parameters"] as Record<string, unknown>;

tool["name"]. The flattened shape. The shape the bug produces.

So the assertions were not neutral about the envelope — they encoded its absence. Anyone who read that test learned that the wire format is { name, parameters }, because the test says so twice, in a file that runs twice on every push.

Measured on a clean checkout of main at 05ffff40, bun 1.4.4:

$ bun test test/providers/buddy-donor-gaps.test.ts
 12 pass
 0 fail
 23 expect() calls

Green. On the commit where tool calling through that provider does not work.

Fix the test first, then the code

I resisted touching the source. I changed the two assertions to read through the wrapper, added an explicit shape test, and reran the same file:

$ bun test test/providers/buddy-donor-gaps.test.ts
 10 pass
 4 fail
 24 expect() calls

Four red. Two are the old RE2 assertions now reading tool["function"]["parameters"]. Two are the new envelope tests — one against the normaliser, one through the CN pre-payload path. The failure text is the entire post-mortem:

expect(tool["type"]).toBe("function")
Expected: "function"
Received: undefined

That undefined is what upstream had been receiving for however many releases.

The fix is the shape of the custom branch, applied to function:

-      return normalizeRe2Declaration(fnOut);
+      return { ...rawTool, function: normalizeRe2Declaration(fnOut) };

$ bun test test/providers/buddy-donor-gaps.test.ts
 14 pass
 0 fail
 32 expect() calls

Fourteen, not twelve. The two envelope tests are new; the two RE2 tests still assert the same pattern rewriting, just read from the place the wire actually reads it from.

"Probably pre-existing" is not a finding

One file green means one file green. The full provider directory:

$ bun test test/providers
 1644 pass
 18 skip
 13 fail
 1675 tests across 174 files

Thirteen failures, none about tools — database-backed suites calling getTestPool, with no Postgres in this environment. I did not take that on faith. I stashed the change, ran the directory again, and diffed the two failure lists: same thirteen test names, before and after. That diff is the difference between "pre-existing" and "probably pre-existing", and it took one command.

$ bun run typecheck
$ tsc --noEmit          # exit 0

The shape, generalised

Three conditions, and they co-occur more often than they should:

1. A transform returned the inner thing where the outer thing was expected.
2. The consumer treats a missing key as "feature not requested" rather than as a protocol violation — 200, no tool_calls, no log line.
3. The test covering the transform asserted the transformed value, not the value's position in the contract.

Condition 2 is why it survived. Condition 3 is why nobody found it. Health checks, quota counters and telemetry all measured the request; all of them were telling the truth about a request that arrived and was answered.

Four habits I'd keep

• Assert on the serialised string. JSON.stringify the outbound body once and check that the literal "type":"function" appears in it. No amount of surrounding shape-drift can make that assertion agree with a bug.
• Read through the contract, never around it. Wire format tools[].function.name means the test reads tools[0].function.name.
• Put a negative assertion beside every positive one. expect(tool["name"]).toBeUndefined() is one line, and it is the line that stops the flattened shape from returning quietly.
• Diff failure lists, not failure counts. Counts are a vibe; names are evidence.

I have only seen this from one side, so I'll ask: if you maintain an outbound payload transform, do you assert on the serialised body or the parsed object — and has a dropped wrapper ever slipped past the parsed version in your tree?