INC-2048 — Payment API latency
Checkout and payment requests are timing out for a subset of production traffic in us-east-1.
Timeline
Every state change, automatically captured
12:30Alert triggered
p95 latency above 500ms for 3 consecutive minutes.
12:32Engineer assigned
Paged Sarah Chen via on-call rotation.
12:36Investigation started
Correlated latency with orders-db-primary CPU.
12:41Database issue identified
Connection pool exhausted after deploy-1842.
12:48Fix deployed
Increased pool size and enabled statement timeout.
12:53Service recovered
p95 latency back to 190ms, error rate under 0.3%.
AI analysis
Correlated from metrics, logs and deployments
Latency rose from 180ms to 742ms within 15 minutes, matching a 38% increase in orders-db-primary CPU that began 4 minutes after deploy-1842. The new settlement retry budget triples concurrent queries per request. Rolling back deploy-1842 or raising the connection pool ceiling should restore baseline latency.
Metrics
p95 latency around the incident window
Related logs
2 matching entries
Incident notes
Shared with everyone on the response
No notes yet.
Related deployments
v4.12.0 · 9f3ac21 · 12 min ago
v4.11.9 · aa72e15 · yesterday
Resolution summary
Available once the incident is resolved.