Runbook: Kafka Consumer Lag¶

When to use: The KafkaConsumerLagHigh alert fires, or the SLA breach runbook identifies consumer lag as a contributing cause (e.g., Redis projection falling behind, Fulfillment Gateway delayed dispatches).

Alert thresholds: - Warning: consumer lag > 1,000 messages for > 2 minutes - Critical: consumer lag > 10,000 messages for > 5 minutes

Step 1: Identify the Affected Consumer Group¶

# List all consumer groups and their lag
kafka-consumer-groups.sh \
  --bootstrap-server $KAFKA_BOOTSTRAP \
  --describe --all-groups \
  | grep -E "GROUP|LAG" \
  | sort -k6 -rn

# Or check the Grafana Kafka Overview dashboard
# Panel: "Consumer Group Lag by Group and Topic"

Common consumer groups: | Consumer Group | Service | Topic | |---|---|---| | inventory-reservation-consumer | Inventory | chakra.orders.placed | | inventory-redis-projection | Inventory (Redis) | chakra.inventory.reserved, chakra.inventory.released | | fulfillment-gateway-consumer | Fulfillment Gateway | chakra.orders.confirmed |

Step 2: Diagnose the Root Cause¶

Consumer is running but slow¶

# Check consumer pod CPU/memory
kubectl top pods -n chakra-{service}

# If CPU-throttled → increase resource limits or scale out
kubectl scale deploy/{service}-consumer -n chakra-{service} --replicas=3

Consumer is not running (pod down)¶

kubectl get pods -n chakra-{service} | grep consumer
# If CrashLoopBackOff → check logs
kubectl logs -n chakra-{service} deploy/{service}-consumer --previous

Topic partition count too low¶

# Check partition count vs consumer instances
kafka-topics.sh --bootstrap-server $KAFKA_BOOTSTRAP \
  --describe --topic chakra.{domain}.{event}
# If partitions < consumer instances → consumers sit idle
# Increasing partitions requires coordinated topic admin — escalate to platform team

Step 3: Remediate¶

Root cause	Remediation
Consumer pod down	Fix crash cause; let pod restart; lag will drain
Consumer CPU-throttled	Scale horizontally; update HPA min replicas
Temporary traffic spike	Monitor drain rate; if draining, no action needed
Partition count too low	Escalate to platform team for partition increase (rolling)
Poison pill message	See below

Handling a Poison Pill¶

If the consumer is crashing on a specific message:

# Find the offset of the bad message
kubectl logs -n chakra-{service} deploy/{service}-consumer | grep "deserialize\|parse error"

# Skip the offset (use with caution — only for non-critical consumers)
kafka-consumer-groups.sh \
  --bootstrap-server $KAFKA_BOOTSTRAP \
  --group {consumer-group} \
  --topic chakra.{domain}.{event} \
  --reset-offsets --to-offset {bad_offset+1} \
  --execute

Do not skip offsets for inventory-reservation-consumer or fulfillment-gateway-consumer without engineering lead approval — these affect business correctness.

Step 4: Verify Recovery¶

Lag decreasing in Grafana Kafka Overview dashboard
Alert resolves within 15 minutes of remediation
inventory_projection_lag_seconds metric normalizing (for Redis projection consumers)

Impact by Consumer Group¶

Consumer Group	Impact if lagged
`inventory-reservation-consumer`	New orders are not triggering stock reservations — saga stalls at `OrderPlaced`
`inventory-redis-projection`	Read model is stale — stock levels shown to customers are behind, but write consistency is maintained
`fulfillment-gateway-consumer`	Confirmed orders are not being dispatched to WMS — fulfillment delay, no data loss