Introduce optional Hub 24/7 alert routing with dedicated action group¶

Context and Problem Statement¶

Managed landing zones require reliable after-hours support for Hub infrastructure incidents. Existing alert processing in the platform uses shared action groups and broad routing patterns, which does not provide a clean customer-specific 24/7 webhook path for selected Hub alerts.

Decision Drivers¶

Customer-specific webhook routing is required for support integrations.
The feature must be optional per customer and disabled by default.
Hub alert filtering should target relevant resource types and selected activity log alert rules.
Daily support window must be configurable without code changes.
Existing non-Hub alert routing should remain unchanged.

Considered Options¶

Reuse the existing shared connectivity action group for Hub 24/7 routing.
Implement Hub 24/7 routing in foundation layer for all subscriptions.
Implement Hub 24/7 routing in connectivity layer with a dedicated action group.

Decision Outcome¶

Chosen option: "Implement Hub 24/7 routing in connectivity layer with a dedicated action group", because it keeps Hub operations close to the owning layer, enables customer-specific webhook isolation, and avoids coupling after-hours Hub support with shared platform-wide alert channels.

Consequences¶

Good, because customer webhook integrations are isolated from shared alert channels.
Good, because rollout is controlled by a per-customer feature flag (hub_247_support.enabled).
Good, because filters reduce noise by targeting selected alert names and Hub resource types.
Good, because schedule configuration supports local support windows (default 18:00-08:00).
Bad, because an additional action group resource is created when enabled.
Bad, because alert-name curation must be maintained over time if upstream naming changes.

Validation¶

This decision is validated by:

Terraform plan/apply showing no changes when hub_247_support.enabled = false.
Terraform plan showing dedicated action group and APR creation when enabled.
PR review confirming module-level backward compatibility of alert_processing_rule.
Customer-specific parameter updates in the params template repository.

Pros and Cons of the Options¶

Reuse the existing shared connectivity action group for Hub 24/7 routing¶

Good, because no extra action group is needed.
Good, because implementation is smaller.
Bad, because shared routing increases blast radius and coupling.
Bad, because customer-specific webhook separation is harder to guarantee.

Implement Hub 24/7 routing in foundation layer for all subscriptions¶

Good, because foundational monitoring is centralized.
Neutral, because it can be reused by multiple layers.
Bad, because Hub support scope is connectivity-specific.
Bad, because it increases cross-layer dependencies for a feature used by a subset of customers.

Implement Hub 24/7 routing in connectivity layer with a dedicated action group¶

Good, because scope aligns with Hub infrastructure ownership.
Good, because dedicated action group supports clean per-customer webhook routing.
Good, because the feature can be enabled gradually per customer.
Bad, because there is additional configuration surface to document.