Ga naar inhoud

Introduce optional Hub 24/7 alert routing with dedicated action group

Context and Problem Statement

Managed landing zones require reliable after-hours support for Hub infrastructure incidents. Existing alert processing in the platform uses shared action groups and broad routing patterns, which does not provide a clean customer-specific 24/7 webhook path for selected Hub alerts.

Decision Drivers

  • Customer-specific webhook routing is required for support integrations.
  • The feature must be optional per customer and disabled by default.
  • Hub alert filtering should target relevant resource types and selected activity log alert rules.
  • Daily support window must be configurable without code changes.
  • Existing non-Hub alert routing should remain unchanged.

Considered Options

  • Reuse the existing shared connectivity action group for Hub 24/7 routing.
  • Implement Hub 24/7 routing in foundation layer for all subscriptions.
  • Implement Hub 24/7 routing in connectivity layer with a dedicated action group.

Decision Outcome

Chosen option: "Implement Hub 24/7 routing in connectivity layer with a dedicated action group", because it keeps Hub operations close to the owning layer, enables customer-specific webhook isolation, and avoids coupling after-hours Hub support with shared platform-wide alert channels.

Consequences

  • Good, because customer webhook integrations are isolated from shared alert channels.
  • Good, because rollout is controlled by a per-customer feature flag (hub_247_support.enabled).
  • Good, because filters reduce noise by targeting selected alert names and Hub resource types.
  • Good, because schedule configuration supports local support windows (default 18:00-08:00).
  • Bad, because an additional action group resource is created when enabled.
  • Bad, because alert-name curation must be maintained over time if upstream naming changes.

Validation

This decision is validated by:

  • Terraform plan/apply showing no changes when hub_247_support.enabled = false.
  • Terraform plan showing dedicated action group and APR creation when enabled.
  • PR review confirming module-level backward compatibility of alert_processing_rule.
  • Customer-specific parameter updates in the params template repository.

Pros and Cons of the Options

Reuse the existing shared connectivity action group for Hub 24/7 routing

  • Good, because no extra action group is needed.
  • Good, because implementation is smaller.
  • Bad, because shared routing increases blast radius and coupling.
  • Bad, because customer-specific webhook separation is harder to guarantee.

Implement Hub 24/7 routing in foundation layer for all subscriptions

  • Good, because foundational monitoring is centralized.
  • Neutral, because it can be reused by multiple layers.
  • Bad, because Hub support scope is connectivity-specific.
  • Bad, because it increases cross-layer dependencies for a feature used by a subset of customers.

Implement Hub 24/7 routing in connectivity layer with a dedicated action group

  • Good, because scope aligns with Hub infrastructure ownership.
  • Good, because dedicated action group supports clean per-customer webhook routing.
  • Good, because the feature can be enabled gradually per customer.
  • Bad, because there is additional configuration surface to document.