Database Readiness¶

Verified by tests

DatabaseAvailabilityMiddlewareTests, DatabaseAvailabilityMiddlewareExtensionsTests, HeartbeatWorkerTests, SchemaInitializationOptionsTests, WhizbangDatabaseInitializerServiceTests — library CI run #31657041675 (2026-08-13)

Whizbang coordinates workers with database availability through the ISchemaReadyGate — a signal-based gate that workers await before issuing any SQL. The schema initializer marks the gate ready exactly once, after migrations succeed; until then, every database-touching worker blocks at the top of its ExecuteAsync.

Updated

Earlier designs used a polling IDatabaseReadinessCheck interface that each worker invoked on every cycle. That interface has been removed. The shipped mechanism is the signal-based ISchemaReadyGate described on this page: workers wait once at startup instead of re-checking readiness per poll, and readiness is driven by the migration runner rather than by table-existence probes.

Overview¶

Why a Readiness Gate?¶

Without a gate, workers race the migration runner:

Workers registered before the driver's initializer can fire SQL against an unmigrated database
Startup exceptions are indistinguishable from runtime failures
Every worker needs its own retry/backoff for the "schema not there yet" window

With ISchemaReadyGate:

Workers hold off on all SQL until migrations have completed
Hosted-service registration order stops mattering — a worker whose StartAsync runs before the initializer still waits on the gate
Migration failure keeps the gate closed, so workers never run against a broken schema
One signal, many waiters — no polling, no per-worker readiness logic

ISchemaReadyGate Interface¶

ISchemaReadyGate Interface

/// <summary>
/// Signal-based gate that workers await before issuing SQL against the database. The schema
/// initializer (typically WhizbangDatabaseInitializerService in the EFCore Postgres
/// driver) calls MarkReady after migrations succeed. Workers call
/// WaitForReadyAsync at the top of their ExecuteAsync so they hold off
/// on any SQL until the schema is provisioned.
/// </summary>
public interface ISchemaReadyGate {
  /// <summary>
  /// Awaits the schema-ready signal. Returns immediately when ready; otherwise blocks until
  /// MarkReady is called or the cancellation token fires.
  /// </summary>
  Task WaitForReadyAsync(CancellationToken cancellationToken);

  /// <summary>True once MarkReady has been called; pure synchronous query.</summary>
  bool IsReady { get; }

  /// <summary>
  /// Signals all waiters that the schema is provisioned. Idempotent — subsequent calls are
  /// no-ops. Called by the initializer in its StartAsync after migrations complete.
  /// </summary>
  void MarkReady();
}

Contract:

WaitForReadyAsync returns immediately once ready; otherwise blocks until MarkReady or cancellation
MarkReady is idempotent and sticky — waiters that arrive after the signal return immediately
IsReady is a synchronous, allocation-free query (used by the HTTP middleware)

The default implementation, SchemaReadyGate, is a single TaskCompletionSource created with RunContinuationsAsynchronously; any number of waiters can await it.

Who Marks the Gate Ready¶

The EFCore Postgres driver registers WhizbangDatabaseInitializerService — a plain IHostedService (not a BackgroundService). How its StartAsync behaves depends on SchemaInitializationOptions.NonBlockingSchemaInit:

Non-blocking (NonBlockingSchemaInit = true, the turnkey default): StartAsync returns immediately and initialization runs in the background. The host binds its port and can answer liveness probes while the gate stays closed until migrations succeed. An optional SchemaInitializationOptions.MigrationTimeout (default: none) treats a hung migration as failed so the pod doesn't sit alive-but-wedged forever.
Blocking (NonBlockingSchemaInit = false, opt-out): initialization runs inline in StartAsync, so the host does not finish starting (no HTTP port, no workers) until it completes.

Either way, the same initialization sequence runs:

WhizbangDatabaseInitializerService initialization sequence

private async Task _runInitializationAsync(CancellationToken cancellationToken) {
  // ISchemaInitializationRunner.RunAsync — delegates to
  // DbContextInitializationRegistry.InitializeAllAsync (with MigrationTimeout as a ceiling, if set)
  await _runMigrationsAsync(cancellationToken);

  // Best-effort: recompute partition_number columns that may have drifted across a
  // PartitionCount change. NEVER blocks MarkReady — workers can run on a stale partition
  // map (next claim cycle picks them up correctly via the live PartitionCount).
  await TryRecomputePartitionsAsync(cancellationToken);

  _schemaReadyGate.MarkReady();
}

Ordering guarantees:

Migrations run first (ISchemaInitializationRunner.RunAsync, which delegates to DbContextInitializationRegistry.InitializeAllAsync)
Partition recompute is best-effort — a failure logs a warning but does not block readiness
MarkReady is called last — only after the schema is provisioned

On migration failure: the gate is not marked ready (fail-closed), in either mode. In blocking mode StartAsync throws, host startup aborts, and workers never enter their main loops. In the non-blocking default the host stays alive (liveness green) but never becomes ready — the pod stays out of traffic rotation and the rollout fails cleanly instead of the pod being killed mid-migration; the failure also drives the managed run-control lifecycle into its fault path (when registered) so health reporting surfaces the failure. Either way, nothing runs against a broken schema.

How Workers Use the Gate¶

Database-touching workers await the gate once, at the top of ExecuteAsync, before their first SQL call. From ClaimWorker:

ClaimWorker gate usage

// Hold off on any SQL until the schema is provisioned. The driver's initializer
// (WhizbangDatabaseInitializerService) signals the gate after migrations succeed.
// This decouples worker startup from hosted-service registration order — even if
// this worker's StartAsync runs before the initializer, we still wait here.
try {
  await _schemaReadyGate.WaitForReadyAsync(stoppingToken);
} catch (OperationCanceledException) {
  return;
}

Workers that gate on schema readiness include ClaimWorker, HeartbeatWorker, MaintenanceWorker, LeaseRenewalWorker, the inbox/outbox drain and flush workers (InboxDrainWorker, InboxDispatchWorker, InboxHandlerWorker, OutboxDrainWorker, OutboxPublishWorker, OutboxCompletionFlushWorker, PerspectiveCompletionFlushWorker, FailureFlushWorker), and DeadLetterRecoveryWorker.

The PerspectiveWorker itself consumes work channels fed by ClaimWorker (see Perspective Worker) — since the upstream claimer is gated, perspective processing implicitly starts only after the schema is ready.

Key difference from polling designs: readiness is checked once, not per cycle. After the gate opens, transient database failures during runtime surface as ordinary exceptions with retry/backoff in each worker's loop — they are not conflated with "schema not ready yet."

HTTP Availability Middleware¶

The ASP.NET hosting package includes DatabaseAvailabilityMiddleware, which returns 503 Service Unavailable for gated requests until the gate signals ready — then becomes a pass-through. A configurable set of exempt path prefixes (default: /alive, /health, /version via DatabaseAvailabilityMiddleware.DefaultExemptPaths) always passes through, so liveness/readiness probes keep working while the schema initializes under non-blocking init:

DatabaseAvailabilityMiddleware

public class DatabaseAvailabilityMiddleware {
  // Probe endpoints are never gated
  public static readonly IReadOnlyList<string> DefaultExemptPaths = ["/alive", "/health", "/version"];

  private static readonly byte[] _responseBody = Encoding.UTF8.GetBytes(
    """{"error":"Service temporarily unavailable","reason":"schema_initializing"}""");

  public DatabaseAvailabilityMiddleware(
      RequestDelegate next, ISchemaReadyGate schemaReadyGate, IReadOnlyList<string>? exemptPaths = null,
      AvailabilityGateMode mode = AvailabilityGateMode.AllNonExempt) { /* ... */ }

  public async Task InvokeAsync(HttpContext context) {
    if (!_schemaReadyGate.IsReady && !_isExempt(context.Request.Path) && _isGated(context.Request.Method)) {
      context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
      context.Response.Headers.RetryAfter = "30";
      context.Response.ContentType = "application/json";
      await context.Response.Body.WriteAsync(_responseBody, context.RequestAborted);
      return;
    }

    await _next(context);
  }
}

Clients receive a JSON body with "reason": "schema_initializing" and a Retry-After: 30 header while migrations run. AvailabilityGateMode selects what gets gated: AllNonExempt (default) 503s every non-exempt request; MutationsOnly 503s only unsafe methods (POST/PUT/PATCH/DELETE) so safe reads (GET/HEAD/OPTIONS) against read-model tables keep working during an event-store migration.

Health Checks Integration¶

ISchemaReadyGate.IsReady composes naturally with ASP.NET Core health checks for Kubernetes readiness probes:

Health Checks Integration

public class SchemaReadyHealthCheck : IHealthCheck {
  private readonly ISchemaReadyGate _gate;

  public SchemaReadyHealthCheck(ISchemaReadyGate gate) {
    _gate = gate;
  }

  public Task<HealthCheckResult> CheckHealthAsync(
    HealthCheckContext context,
    CancellationToken ct = default
  ) {
    return Task.FromResult(_gate.IsReady
      ? HealthCheckResult.Healthy("Schema is provisioned")
      : HealthCheckResult.Unhealthy("Schema is still initializing"));
  }
}

// Program.cs
builder.Services.AddHealthChecks()
  .AddCheck<SchemaReadyHealthCheck>("schema", tags: new[] { "ready" });

app.MapHealthChecks("/health/ready", new HealthCheckOptions {
  Predicate = check => check.Tags.Contains("ready")
});

Kubernetes Integration:

Health Checks Integration (2)

apiVersion: v1
kind: Pod
metadata:
  name: whizbang-worker
spec:
  containers:
  - name: worker
    image: whizbang:latest
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

Benefits:

Container orchestrator knows when the pod is ready
Traffic routing is delayed until migrations complete
The DatabaseAvailabilityMiddleware covers direct HTTP traffic in the same window

Testing¶

Because the gate is a simple signal, tests wire workers with a pre-marked gate (or hold it closed to assert blocking behavior):

Testing with SchemaReadyGate

[Test]
public async Task Worker_WithReadyGate_ProcessesWorkAsync() {
  // Arrange — gate ready, worker may issue SQL immediately
  var gate = new SchemaReadyGate();
  gate.MarkReady();

  var worker = new HeartbeatWorker(
    _scopeFactory,
    _instanceProvider,
    gate,
    Options.Create(new HeartbeatWorkerOptions { IntervalSeconds = 1 }),
    NullLogger<HeartbeatWorker>.Instance);

  // Act / Assert — worker enters its main loop without blocking
  await worker.StartAsync(CancellationToken.None);
}

[Test]
public async Task Worker_WithClosedGate_DoesNotTouchDatabaseAsync() {
  // Arrange — gate NEVER marked ready
  var gate = new SchemaReadyGate();
  var worker = new HeartbeatWorker(
    _scopeFactory,
    _instanceProvider,
    gate,
    Options.Create(new HeartbeatWorkerOptions()),
    NullLogger<HeartbeatWorker>.Instance);

  // Act
  await worker.StartAsync(CancellationToken.None);

  // Assert — no SQL was issued; the worker is parked on WaitForReadyAsync
  await Assert.That(gate.IsReady).IsFalse();
}

Best Practices¶

DO ✅¶

✅ Await the gate before any SQL in custom database-touching hosted services
✅ Register the driver initializer (done automatically by AddWhizbang().WithDriver.Postgres)
✅ Use DatabaseAvailabilityMiddleware (or an equivalent health check) so HTTP traffic waits for the schema
✅ Let migration failures abort startup — a closed gate is the safety mechanism, not a bug
✅ Mark the gate ready in test fixtures that bypass the real initializer

DON'T ❌¶

❌ Poll IsReady in a loop from workers — WaitForReadyAsync is the intended wait primitive
❌ Call MarkReady from application code — that is the initializer's job (tests excepted)
❌ Treat post-ready database outages as a readiness concern — after the gate opens, failures are handled by each worker's retry/backoff
❌ Rely on hosted-service registration order for startup sequencing — the gate exists precisely so order doesn't matter

Troubleshooting¶

Problem: Workers Never Start Processing¶

Symptoms: No worker log output beyond startup lines; no SQL activity; HTTP returns 503 with "reason": "schema_initializing".

Causes: 1. Migrations failed or hit MigrationTimeout — the gate was never marked ready (host startup aborts in blocking mode; the host stays alive-but-never-ready in the non-blocking default) 2. The driver initializer is not registered (custom DI setup that bypasses AddWhizbang().WithDriver.Postgres) 3. Test fixture constructed workers with a SchemaReadyGate that was never marked ready

Solution: Problem: Workers Never Start Processing

# Check startup logs for migration errors (host aborts on initializer failure)
grep -i "migrat\|initializ" logs.txt

# Verify the database is reachable
psql -h localhost -U postgres -d whizbang -c "SELECT 1;"

# Verify Whizbang tables exist after a successful start
psql -h localhost -U postgres -d whizbang -c "\dt wh_*"

Problem: SQL Fired Against Unmigrated Database¶

Symptoms: "relation does not exist" errors on startup.

Causes: 1. A custom hosted service issues SQL without awaiting ISchemaReadyGate 2. Application code runs queries during ConfigureServices/startup before the host starts

Solution: inject ISchemaReadyGate and await it first: Problem: SQL Fired Against Unmigrated Database

protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
  await _schemaReadyGate.WaitForReadyAsync(stoppingToken);
  // ... safe to issue SQL from here
}