Incidents
INC-2045 — Notification delivery backlog
Push notification queue depth exceeded 40k messages.
lowresolvednotification-service48m
Timeline
Every state change, automatically captured
18:02Alert triggered
Queue depth above threshold.
18:20Workers scaled
Batch workers scaled 2 → 6.
18:50Resolved
Queue drained to normal depth.
AI analysis
Correlated from metrics, logs and deployments
A marketing campaign produced a 5x send spike. Autoscaling thresholds were too conservative for burst traffic.
Metrics
p95 latency around the incident window
Related logs
0 matching entries
No error or warning logs correlated to this incident.
Incident notes
Shared with everyone on the response
No notes yet.
Incident IDINC-2045
Servicenotification-service
AssigneeDana Whitfield
Createdyesterday
Duration48m
Related deployments
deploy-1838Successful
v1.19.7 · 0ea41f9 · yesterday
Resolution summary
Scaled batch workers and lowered the autoscaling trigger from 20k to 8k queued messages.