The alert hit at 2:13 a.m. and the AWS console showed nothing obvious. The cost graph was flat. No spikes. No drops. Yet the gut feeling said something was wrong. Opening CloudTrail logs told the real story—hidden in pages of JSON, embedded in obscure event fields.
CloudTrail is AWS’s best log of truth, but the raw data is a swamp. Thousands of events pile up per minute. Finding the right one is like hunting a single packet in the open sea. This is why CloudTrail queries matter. And why version-controlled query runbooks in Git turn chaos into a repeatable workflow.
A Git CloudTrail Query Runbook is a simple but brutal tool. It’s a repository holding saved CloudTrail queries, instructions, and scripts that anyone on the team can run. The SQL lives in plain sight. Filters, parameters, and known false positives stay documented. Change history is tracked. The team never starts from scratch when the fire hits.
Building these runbooks starts with organizing queries in the same way you version code. Commit every useful query. Link each query to a specific incident if possible. Use clear filenames and README files to describe intent. Store helper scripts—whether for Athena, CloudWatch Logs Insights, or your own Lambda CLI tooling—right next to the query files.