Step Functions Logging
To turn on step functions logging, attach an execution role that writes to CloudWatch Logs. Then choose a log level on your state machine to record each run. Teams often grant wide write permissions so workflows can ship data right away. Broad roles make setup fast, but they can expose sensitive payloads to too many readers. How step functions logging works in AWS State machines run as a sequence of steps that pass JSON data forward. Each step takes input, does work, and sends output to the n
K8s Audit Logging + Cloud Functions IAM: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
To turn on step functions logging, attach an execution role that writes to CloudWatch Logs. Then choose a log level on your state machine to record each run. Teams often grant wide write permissions so workflows can ship data right away. Broad roles make setup fast, but they can expose sensitive payloads to too many readers.
How step functions logging works in AWS
State machines run as a sequence of steps that pass JSON data forward. Each step takes input, does work, and sends output to the next state. Standard workflows store run history in the service by default. Express workflows do not keep that history unless you turn on step functions logging.
First, you create a log group in CloudWatch Logs to hold the output. Next, you update the IAM role attached to your state machine. That role needs permission to create log streams and put log events. Without these exact rights, the workflow cannot send records to the destination.
After the role is ready, you pick a log level for the workflow. You can log nothing, only errors, fatal stops, or every single event. You also decide whether to include the full execution data in those records. Turning on execution data puts the raw input and output into every log entry.
What native logging solves and where gaps remain
Native logs help engineers see why a step failed or stalled during a run. When a task throws an error, the log shows the exact state and payload. Teams use these records to fix broken transitions and track run times. It gives a clear view of how the state machine moves from start to finish.
Still, this setup only controls where logs go and who can open the log group. It does not inspect what individual steps or tasks do while they run. If a step queries a database, the state machine log just prints the returned payload. Any email addresses, phone numbers, card numbers, or national IDs land in the log text.
Also, engineers often need to run tasks or check databases to fix a failed step. IAM roles decide who can log in to those backend systems. Once inside, a user with a broad role can run any query or command. Native workflow logs do not filter live queries or mask fields in query results.
K8s Audit Logging + Cloud Functions IAM: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Managing sensitive data in workflow runs
When you include execution data in logs, every payload becomes part of your log storage. A payment step might pass card numbers from one function to the next. A user signup step might carry email addresses and phone numbers in plain text. Anyone with read rights to CloudWatch Logs can query those records.
Teams try to fix this by turning off execution data in production workflows. That keeps private fields out of the logs, but it makes errors hard to fix. Engineers cannot see which input caused a task to crash. They end up logging in to production databases to check the state by hand.
Direct database sessions bring their own risks during an incident. Shared admin roles let engineers read whole tables instead of single rows. You need controls that check each command and mask sensitive fields in results. Data masking rewrites matched fields like card numbers or emails before returning output.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
Why are my Step Functions logs not showing up in CloudWatch?
The state machine role usually lacks permission to write to CloudWatch Logs. Check that the IAM role allows creating log streams and putting log events.
Should I include execution data in production state machine logs?
Only include execution data if your payloads do not hold private user details. Raw payloads often contain emails, phone numbers, or keys that should not sit in logs.
What is the difference between Standard and Express workflow logging?
Standard workflows keep visual run history in the console even without CloudWatch. Express workflows rely on CloudWatch logs for you to inspect past runs and errors.