Empty Stream ID Policy¶

Verified by tests

EmptyStreamIdPolicyTests, EmptyStreamIdGuardTests, EmptyStreamIdExceptionTests, StreamIdCoalescerTests, DeadLetterRecoverySqlTests — library CI run #31657041675 (2026-08-13)

EmptyStreamIdPolicy controls how Whizbang handles outbox / inbox / perspective rows whose stream_id is Guid.Empty (the all-zeros UUID, distinct from NULL).

Background¶

Whizbang's stream_id column is nullable. NULL is the documented marker for singleton-stream messages — typically event-store-only writes that don't need per-stream FIFO ordering. The coordinator handles NULL by falling back to WorkId as the stream identity at drain time, so each singleton-stream row becomes its own "stream of one."

Guid.Empty (00000000-0000-0000-0000-000000000000) looks like a real UUID to C# null-checks but represents nothing. It's almost always a producer bug: passing default(Guid) or an uninitialized field instead of null when there is no stream.

Pre-v0.657, the coordinator's stream-id coalesce — r.StreamId ?? r.WorkId ?? Guid.Empty — only caught NULL. The ?? operator treats Guid.Empty as "valid stream," so it skipped the WorkId fallback. The subsequent .Where(g => g != Guid.Empty) filter then dropped the row from the drain channel entirely.

The result: rows with stream_id = Guid.Empty were claimed by ClaimWorker every cycle (attempts incremented in claim_orphaned_outbox) but never reached OutboxDrainWorker. No publish attempt, no DLQ promotion, no error captured, no log emitted. The bug was silent.

EmptyStreamIdPolicy is the structural fix — defense-in-depth across the producer, drainer, and DLQ recovery surfaces.

Policy Values¶

Value	Storage behavior	Drain behavior
`Reject` (default)	Throws `EmptyStreamIdException` at `StoreOutboxMessagesAsync` / `StoreInboxMessagesAsync` time	Coordinator-side recovery is unconditional — Empty → `WorkId` fallback + Warning per row
`FallbackToMessageId`	Storage accepts the row	Same as Reject (always-recover at drain)
`DeadLetter`	Storage accepts the row	Designed: coordinator moves to `wh_dead_letters` with `MessageFailureReason.EmptyStreamId` instead of attempting to drain
`Purge`	Storage accepts the row	Designed: coordinator DELETEs the row + emits Error with the `EmptyStreamId` reason code

The drainer's Empty → WorkId recovery runs independently of the policy. It recovers rows that already landed before the policy was tightened. The policy gates what the producer can write; the drainer always heals.

Updated

Shipped behavior: the storage-time guard (for Reject) and the unconditional Empty → WorkId drain recovery are implemented. The DeadLetter and Purge drain-time behaviors described above are the design intent but are not yet wired in the coordinator — under those policy values, storage accepts the row and the drainer's always-on recovery processes it exactly like FallbackToMessageId. The MessageFailureReason.EmptyStreamId code (value 11) and the recover_dead_letter Empty→NULL normalization are already in place for when the drain-time policies land.

Default: Reject¶

Default Policy

services.AddWhizbang(options => {
  // No-op — Reject is already the default.
  // options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.Reject;
});

Under Reject, calling StoreOutboxMessagesAsync with a message whose StreamId == Guid.Empty throws EmptyStreamIdException:

Whizbang.Core.Messaging.EmptyStreamIdException: Producer attempted to write
  MyApp.Contracts.RemoveUserCommand (message_id=019e92b2-1bbb-708d-...)
  with stream_id=Guid.Empty (00000000-0000-0000-0000-000000000000). Empty
  stream_id is rejected under EmptyStreamIdPolicy.Reject — pass null for
  singleton-stream messages or a real stream identity. See
  operations/configuration/empty-stream-id-policy for migration guidance.

The exception's MessageId and MessageType properties carry the offending row's identity so producers can locate the bad call site without grepping logs.

Lenient Mode: FallbackToMessageId¶

Lenient Policy for Migrations

services.AddWhizbang(options => {
  options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.FallbackToMessageId;
});

FallbackToMessageId is for deployments that have legacy producers writing Guid.Empty and can't be patched immediately. The row is accepted at storage time; the drainer's always-on recovery uses WorkId as the singleton-stream identity, emits a Warning naming the row, and processes it normally.

Use the Warning count to track the rate of "still-bad" producer writes. When the rate hits zero, flip the policy back to Reject to close the surface.

Forensic Preservation: DeadLetter¶

Forensic DLQ Policy

services.AddWhizbang(options => {
  options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.DeadLetter;
});

DeadLetter is designed to preserve every offending row in wh_dead_letters with failure_reason = MessageFailureReason.EmptyStreamId (value 11) — a forensic trail of producer bugs plus the option to replay rows after the producer is fixed. Not yet wired at drain time — see the shipped-behavior callout above; today this value behaves like FallbackToMessageId.

Recovery is self-healing: recover_dead_letter (migration 051) normalizes Guid.Empty → NULL on the INSERT back into the source table, so a recovered row doesn't immediately re-stick. This normalization is implemented and applies to any wh_dead_letters row with an Empty stream_id, however it got there.

Hard Purge: Purge¶

Hard Purge Policy

services.AddWhizbang(options => {
  options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.Purge;
});

Purge is designed to DELETE the row without publishing and emit an Error log with the MessageFailureReason.EmptyStreamId code. Data loss is permanent — intended only for known-spammy producers you've already given up on (e.g., decommissioned services still emitting bad messages). Not yet wired at drain time — see the shipped-behavior callout above; today this value behaves like FallbackToMessageId.

Defenses in Depth¶

Three independent surfaces close the silent-stuck pattern:

Empty Stream ID Defenses

flowchart LR
  P[Producer call site]
  S{Storage<br/>EmptyStreamIdGuard}
  W[wh_outbox / wh_inbox]
  D{Drainer<br/>StreamIdCoalescer}
  Drain[Drain channel]
  DLQ{DLQ recovery<br/>recover_dead_letter}

  P -->|Reject ?| S
  S -->|throw EmptyStreamIdException| P
  S -->|accept| W
  W --> D
  D -->|Empty -> WorkId<br/>+ Warning| Drain
  W -->|move_to_dead_letters| DLQ
  DLQ -->|normalize Empty -> NULL| W

Surface	Defense	When it fires
Producer	`EmptyStreamIdGuard.ThrowIfAnyHasEmptyStreamId`	INSERT time under `Reject`
Drainer	`StreamIdCoalescer.Coalesce` — `Empty → WorkId` + Warning per row	Every `ClaimWorkAsync` call (unconditional)
DLQ replay	`recover_dead_letter` normalizes `Empty → NULL`	On every `RecoverAsync` of an outbox/inbox source row

The structural canary in Stuck Row Sentinel catches any future bug of the same shape ("row claimed but never drained") regardless of root cause.

Migration Guide¶

Existing deployment with legacy producers¶

If you're on <= v0.656 and have producers writing Guid.Empty today, upgrade to v0.657 with FallbackToMessageId first to avoid breaking the producer's commit:

Phase 1: Recover then observe

services.AddWhizbang(options => {
  options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.FallbackToMessageId;
});

Watch the Warning rate (Empty stream_id detected on outbox row {MessageId}). Each one names a producer call site you need to fix. When the rate hits zero for a meaningful window:

Phase 2: Tighten to Reject

services.AddWhizbang(options => {
  options.EmptyStreamIdPolicy = EmptyStreamIdPolicy.Reject;
});

Net-new deployment¶

Stay on the default Reject. New producers will fail loud at INSERT time, surfacing bugs at the call site before the silent-stuck pattern can ever develop.

Stuck rows already in the table¶

The coordinator backstop (StreamIdCoalescer) auto-recovers existing Guid.Empty rows on the next claim tick — no manual DELETE needed. The recovery emits a Warning per row so you can audit what cleared.

Operator Telemetry¶

When the drainer recovers an Empty-stream row, StreamIdCoalescer emits a Warning through the work coordinator's logger (EFCoreWorkCoordinator<TDbContext> category):

Warning: Empty stream_id (00000000-0000-0000-0000-000000000000) detected on
  outbox row 019e92b2-1bbb-708d-... — falling back to WorkId as
  singleton-stream identity. Producer-side fix needed; see
  operations/configuration/empty-stream-id-policy.

Grep the Warning rate to track: - The pace of legacy-producer cleanup (if running FallbackToMessageId) - Drift after a Reject deploy — every Warning means a producer bypassed the storage path (e.g., raw SQL INSERT)

If MessageFailureReason.EmptyStreamId rows accumulate in wh_dead_letters (once the DeadLetter drain-time policy ships), group by message_type to find the producer:

SELECT message_type, COUNT(*) AS empty_count
FROM wh_dead_letters
WHERE failure_reason = 11  -- MessageFailureReason.EmptyStreamId
GROUP BY message_type
ORDER BY empty_count DESC;