告警目录
每一条都有 firing 的 promtool 用例。除 Watchdog 外，每一条还有 exp_alerts 为空的用例。数字来自构建时记录。
| 告警 | severity | team | for | firing | non-firing | passed |
|---|---|---|---|---|---|---|
| OrdersApiErrorBudgetBurnFast | critical | orders | 2m | 1 | 3 | true |
(
slo:sli_error:ratio_rate1h{service="orders-api",slo="availability"} > (14.4 * 0.001)
and
slo:sli_error:ratio_rate5m{service="orders-api",slo="availability"} > (14.4 * 0.001)
)
or
(
slo:sli_error:ratio_rate6h{service="orders-api",slo="availability"} > (6 * 0.001)
and
slo:sli_error:ratio_rate30m{service="orders-api",slo="availability"} > (6 * 0.001)
) | ||||||
| OrdersApiErrorBudgetBurnSlow | warning | orders | 2m | 3 | 2 | true |
(
slo:sli_error:ratio_rate1d{service="orders-api",slo="availability"} > (3 * 0.001)
and
slo:sli_error:ratio_rate2h{service="orders-api",slo="availability"} > (3 * 0.001)
)
or
(
slo:sli_error:ratio_rate3d{service="orders-api",slo="availability"} > (1 * 0.001)
and
slo:sli_error:ratio_rate6h{service="orders-api",slo="availability"} > (1 * 0.001)
) | ||||||
| OrdersApiLatencySLOBurn | warning | orders | 2m | 1 | 1 | true |
(
slo:sli_latency_error:ratio_rate1h{service="orders-api",slo="latency"} > (14.4 * 0.01)
and
slo:sli_latency_error:ratio_rate5m{service="orders-api",slo="latency"} > (14.4 * 0.01)
)
or
(
slo:sli_latency_error:ratio_rate6h{service="orders-api",slo="latency"} > (6 * 0.01)
and
slo:sli_latency_error:ratio_rate30m{service="orders-api",slo="latency"} > (6 * 0.01)
) | ||||||
| PaymentsQueueBacklog | warning | payments | 5m | 1 | 1 | true |
service:payments_queue_oldest_age_seconds:max > 300 and max by (service) (deriv(payments_queue_depth[10m])) > 0 | ||||||
| ServiceHighErrorRate | warning | platform | 10m | 1 | 1 | true |
service:http_requests:rate5m > 0 and ( service:http_requests_errors:rate5m / service:http_requests:rate5m ) > 0.05 | ||||||
| ServiceLatencyP95High | warning | platform | 10m | 1 | 1 | true |
max by (service) (service_route:http_request_duration_seconds:p95_5m) > on (service) service:http_latency_p95_threshold_seconds:constant | ||||||
| BlackboxProbeFailed | critical | platform | 5m | 1 | 1 | true |
probe_success == 0 | ||||||
| InstanceDown | critical | platform | 2m | 1 | 1 | true |
up == 0 | ||||||
| NodeFilesystemFillingUp | warning | platform | 30m | 1 | 1 | true |
instance:node_filesystem_avail:ratio < 0.2
and
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0 | ||||||
| NodeMemoryHigh | warning | platform | 15m | 1 | 1 | true |
instance:node_memory_utilisation:ratio > 0.90 | ||||||
| PostgresConnectionsNearLimit | warning | platform | 10m | 1 | 1 | true |
pg_stat_activity_connections / pg_settings_max_connections > 0.85 | ||||||
| PostgresDown | critical | platform | 2m | 1 | 1 | true |
pg_up == 0 | ||||||
| ScrapeTargetMissing | critical | platform | 5m | 1 | 1 | true |
absent(up{job="storefront"}) | ||||||
| TLSCertExpiringSoon | warning | platform | 1h | 1 | 1 | true |
(probe_ssl_earliest_cert_expiry - time()) < (14 * 24 * 3600) | ||||||
| AlertmanagerNotificationsFailing | critical | platform | 15m | 1 | 1 | true |
( sum by (service) (rate(alertmanager_notifications_failed_total[5m])) / sum by (service) (rate(alertmanager_notifications_total[5m])) ) > 0.10 and sum by (service) (rate(alertmanager_notifications_total[5m])) > 0 | ||||||
| PrometheusRuleEvaluationFailures | warning | platform | 10m | 1 | 1 | true |
increase(prometheus_rule_evaluation_failures_total[5m]) > 0 | ||||||
| Watchdog | info | platform | none | 1 | 0 | true |
vector(1) | ||||||