How do you alert on something that's supposed to be quiet?

Most monitoring is built on an assumption that's easy to forget you're making: that the thing you're watching produces a roughly steady signal, so an absence of that signal means something broke. A web server should answer requests every second. A batch job should write a row every hour. When it stops, you page someone.

That assumption falls apart for event-driven pipelines, and I ran into exactly this problem building the monitoring for a transcript-ingestion pipeline that pulls meeting recordings from Zoom, Microsoft Teams, and Google Meet and links them back to the calendar events they belong to. A quiet week for this pipeline is often completely normal — advisors take holidays, calendars empty out, meetings just don't happen. An earlier monitoring effort had actually excluded transcripts from alerting entirely, for exactly this reason: there was no way to tell a quiet week from a broken integration, so the choice was between constant false alarms and no coverage at all. They picked no coverage. That's a reasonable call when your options are that narrow, but it also means the pipeline had a real blind spot — and it turned out to be a costly one. Three customers silently lost transcript capture for weeks before anyone noticed. A folder moved, a setting flipped somewhere upstream, and nothing was watching. The alerting that did exist could only say "transcripts aren't flowing," never which meetings were actually missing, so every one of those incidents turned into a manual call to the customer just to reconstruct what had happened.

Stop measuring the signal, measure it against an expectation

The reframe that got me out of the "quiet vs. broken" bind was to stop asking "did anything arrive?" and start asking "did what we expected arrive?" Those sound similar but they're answering completely different questions. The first has no baseline — an empty result is indistinguishable whether it's Tuesday-normal or something-is-broken. The second has a built-in baseline, because calendar events already tell you how many meetings should have produced a transcript in a given window. Once you have that number, a shortfall stops being an unquantified silence and becomes a concrete gap: nine of twelve meetings captured, not "nothing happened, maybe that's fine."

I built this as a BigQuery view that derives the expected meeting count from calendar data, which a classifier then reads to decide, per customer and per platform, whether coverage looks healthy, degraded relative to that customer's own baseline, or stalled outright. It runs as a scheduled Cloud Run job, with log-based metrics and alert policies sitting on top of the classifier's output rather than on raw transcript counts.

The expectation is only useful if it excludes what was never expected

An expected-count baseline solves the "is this actually a gap" problem, but it introduces a new failure mode if you're not careful: calendar events exist for platforms a customer never actually connected. If a customer has Zoom linked but not Teams, their calendar still has Teams-labeled meetings on it sometimes, and a naive expected-count model would flag every one of those as a missing transcript, forever, for a platform that was never wired up to produce one. That's the second half of the idea, and it matters as much as the first: an expectation is only useful if it excludes cases that were never expected in the first place. I added a connected-platform gate that checks what a customer has actually linked before the classifier ever counts a meeting as "should have produced a transcript." Without that gate, the alerting would have traded one kind of noise (arbitrary silence) for another (permanently flagging things that were never going to happen).

The diagnostic found a real bug while I was still building it

The strongest evidence that this framing was right came before the system was even finished. While I was building and testing the expected-count view, it surfaced a real outage nobody knew about: one customer's Google Meet transcripts had quietly stopped landing two weeks earlier, and nothing existing had flagged it. Closing that case wasn't trivial — it needed a correction to the view so a transcript that had just arrived wasn't miscounted as still-missing, plus a change to the classifier so this specific failure pattern could get through the gate without opening the door to false alarms on unrelated accounts. But finding a live, unreported outage during development, using the exact mechanism I was building to prevent that outage, is about as good a validation as a monitoring system can get.

Shipping it slower than I wanted to

The pipeline itself is live in dev now, verified against real production log output rather than unit tests alone — event-driven systems have edge cases that synthetic test data doesn't reliably surface. But the alert policy stays deliberately disabled for now, until we've watched roughly a week of real runs and confirmed the thresholds behave as predicted. That's the part I'd get wrong if I optimized for looking finished: a monitor that pages people on its own false positives in week one destroys its own credibility before it's earned any trust, and nobody re-enables an alert they've learned to ignore.