Skip to content

Runbook: Database Failover

When to use: AWS RDS has triggered a failover (planned or unplanned) for one of the Chakra Commerce PostgreSQL instances. Services may be returning 5xx errors during the failover window (typically 30–60 seconds for RDS Multi-AZ).

Databases covered: | Database | RDS Identifier | Service | Namespace | |---|---|---|---| | Orders Event Store | Not RDS — EventStoreDB on EKS | Orders | chakra-orders | | Inventory PostgreSQL | chakra-inventory-postgres | Inventory | chakra-inventory | | Customers PostgreSQL | chakra-customers-postgres | Customers | chakra-customers |

Step 1: Confirm Failover is in Progress

# Check RDS events for the affected instance
aws rds describe-events \
  --source-identifier chakra-{service}-postgres \
  --source-type db-instance \
  --duration 60 \
  | jq '.Events[] | {Message, Date}'

# Common event messages indicating failover:
# "Multi-AZ instance failover started"
# "DB instance restarted"
# "Multi-AZ instance failover completed"

A Multi-AZ failover is typically resolved within 60 seconds. If the event shows "failover completed" and services are still unhealthy, proceed to Step 2.

Step 2: Verify DNS Resolution After Failover

RDS Multi-AZ uses a CNAME that automatically updates to the new primary after failover. Services use the CNAME endpoint — not a direct IP. However, the application connection pool may be holding stale connections.

# Check the CNAME resolves to the new primary
nslookup chakra-{service}-postgres.cluster-xxxxx.{region}.rds.amazonaws.com

# Force connection pool refresh by rolling the pods
kubectl rollout restart deploy/{service} -n chakra-{service}
kubectl rollout status deploy/{service} -n chakra-{service}

Step 3: Verify Service Health After Pod Restart

# Check pods are running
kubectl get pods -n chakra-{service}

# Check database connectivity from the pod
kubectl exec -it deploy/{service} -n chakra-{service} -- \
  node -e "require('./dist/health').checkDb().then(console.log)"

# Or use the health endpoint
curl -s https://internal.chakracommerce.internal/v1/{service}/health | jq .

Step 4: Check for Data Loss (EventStoreDB)

EventStoreDB (Orders) is not RDS and does not use Multi-AZ in the same way. It uses a 3-node cluster with leader election.

# Check EventStoreDB cluster state
kubectl exec -it eventstore-0 -n chakra-orders -- \
  curl -s http://localhost:2113/gossip | jq '.members[] | {state, instanceId}'

# Expected: one member with state "Leader", others "Follower"
# If no leader for > 30 seconds: escalate to platform team

Step 5: Check for Outstanding Saga Compensation

If the Orders service lost connectivity to EventStoreDB or PostgreSQL during a saga (place-order flow), check for orphaned sagas:

# Find orders in AwaitingPayment state older than 5 minutes
# (These may have had payment captured but confirmation not yet appended)
kubectl exec -it deploy/orders -n chakra-orders -- \
  node -e "require('./dist/tools/find-orphaned-sagas').run()"

Orphaned sagas require manual review by the orders-team lead.

Step 6: SLA Impact Assessment

Database failovers are the most common cause of latency spikes that exceed SLA targets. After recovery:

  1. Check the error budget remaining for the affected service in Grafana.
  2. If budget is < 20%: notify engineering VP and consider a feature freeze.
  3. Add a note to the post-incident review about the failover cause (AWS infrastructure event vs. operator error vs. known instance issue).

Prevention

  • All services use RDS Multi-AZ (both primary and standby in separate AZs).
  • Connection pools use keepAlive: true with a maximum connection age to avoid stale connections surviving a failover.
  • RDS Proxy is a future consideration — it maintains a warm connection pool and reduces failover client impact.