Five crash logs I couldn't reproduce

A customer sent five crash reports. His copy of a respected commercial plugin took our player down every time he opened its editor. The same plugin, in the same app, on my machine: not a single crash, in either hosting mode.

This is the worst kind of bug report, and the most common kind that matters. Anyone can fix a crash they can watch. The crashes that decide whether customers trust you are the ones that only exist on hardware you will never touch.

What five stack traces agree on

MetricKit crash diagnostics arrive as JSON: no symbols, just binary UUIDs and offsets. Unglamorous, but honest — and the honesty starts with verifying that the dSYMs you symbolicate against match the UUIDs in the report, byte for byte. Rebuild-from-tag is a layout-drift lottery; the archive from the exact CI run is the only truth.

Symbolicated, all five crashes told one story. The same null pointer — an offset of 0x28 into address zero, which is to say forty bytes into an object that never existed — in the same legacy UI framework call, on the main thread, during the plugin’s editor-view construction. Two different paths reached it: four crashes through the modern view-controller request, one through our fallback path that asks the plugin for its old-style UI factory directly. Two doors, one haunted room.

So the plugin’s editor initialization was dying on his machine and not mine. The question every debugging session eventually reduces to: what is different?

The tell

We eliminated the obvious candidates. Same app version — actually two app versions, months apart, both crashing identically, so not a regression. Same macOS line. Same plugin, which runs clean in every DAW on earth.

Then the boring difference: his copy was licensed. Mine was a trial.

That reframes everything. Licensed plugins run code that trial copies never execute — license files get read, verified, cached. And our player is sandboxed, because it ships through the Mac App Store. A plugin loaded in-process inherits our sandbox, and our sandbox is not the environment the vendor tests in. A license check that works in every DAW can take a path, inside our container, that the vendor has never seen fail. I want to be careful here: this is a hypothesis, reported upstream to the vendor with the evidence, not a verdict. The stacks end in their code; the environment is ours; the truth is probably a handshake between the two.

What was not a hypothesis: every user with a licensed copy and default settings was one editor-open away from the same crash, and none of them had reported it yet.

The fix took a day because the architecture took a year

The rebuilt Fidelia hosts Audio Units out of process by default when a pre-flight scanner flags them as risky — copy-protection schemes that allocate unsigned executable memory, embedded UI toolkits that hook the host’s run loop. Out of process, a plugin crash kills a helper; the music keeps playing.

This plugin’s binary is clean. I know because I scanned it against every signature we have: nothing. There was no mechanism to detect, which is exactly why it sailed through the scanner and ran in-process. So the fix was a second mechanism the scanner had been quietly waiting for: an explicit denylist keyed by the component’s exact identity, checked before the byte scan, with the full crash evidence written into a code comment so the entry never becomes folklore. One entry. The user’s explicit in-process override still wins, because a tool that overrides its user is worse than a crash.

Shipped the same day, inside the release that was already going out the door. The fix was one line of data. The reason it could be one line is that the containment architecture already existed and had already paid for itself twice with other plugins.

The part that almost went wrong

Halfway through, the investigation produced a tidy story: the crash timeline lined up with a user switching hosting modes, so the fifth crash must be the fallback path betraying him after he took our advice. Compelling. Coherent. And built on an assumption nobody had checked — he had sent all five crashes before we ever exchanged a word.

This is the failure mode I watch for hardest in an agent-assisted shop: a verified fact and a plausible inference, told together, sound exactly like two verified facts. The stack traces were real. The symbolication was real. The story stitched over them was fiction, and it was my job to ask which parts had actually been checked before that fiction shipped inside a support email.

The crash was fixed by a denylist entry. The debugging was fixed by the question.

Fidelia is on the Mac App Store with a free trial and no subscription — bring your rack, including the weird plugins.

And if your product has a crash that only happens on machines you don’t own: that is the day job. audiofile.engineering/engineering.