The pod was failing. DynamoDB queries timed out. Logs blurred into noise, and your cluster metrics spiked like a warning siren. You need the right runbook now—not after the incident review.
Kubernetes access to DynamoDB can break for several reasons: misconfigured IAM roles, expired service account tokens, sudden network policy changes, or query throttling. Without a clear runbook, engineers waste minutes chasing the wrong variable.
Direct Kubernetes Access Controls
Before touching queries, verify your pod’s IAM role bindings. In EKS or GKE, confirm service accounts are mapped to roles with correct DynamoDB permissions. In RBAC, ensure the namespace allows the pods to read their configuration secrets. Restrict and document these bindings in your runbook.
DynamoDB Query Diagnostics
Tracking the lifecycle of a query is key. Use CloudWatch to inspect latency patterns, throttling events, and consumed capacity. Query logs should feed into a centralized tool—Grafana, Kibana, or equivalent—for fast correlation. Your runbook must include:
- Command to run
aws dynamodb describe-table for provisioned throughput data. - Steps to run test queries from within the cluster using
aws-cli and temporary creds. - Guidelines to adjust client query batch sizes and retry logic.
Runbook Automation
Static markdown in a wiki is not enough. Integrate runbook triggers into your alerting stack. A failing DynamoDB query alert should link directly to Kubernetes commands for:
- Inspecting pod status (
kubectl get pods --namespace X) - Reviewing mounted secret versions (
kubectl describe pod) - Restarting deployments (
kubectl rollout restart deployment Y)
Incident Recovery
Your runbook should hold a minimal set of shell commands, AWS CLI calls, and Kubernetes pod actions that restore baseline service. Every step must be tested in staging before going live. Keep the runbook small, current, and executable under pressure.
Tools and documentation decay when left alone. Build your Kubernetes access DynamoDB query runbooks to live alongside the code, triggered as code, versioned as code. The next failure will come without warning. The runbook is your fastest path back.
See how to generate, execute, and test these runbooks inside Hoop.dev—live and running in minutes.