How to Fix MongoDB Error 356 (TxnRetryCounterTooOld)
You get mongodb error 356 when a transaction message carries an old retry counter. You fix it by retrying the whole transaction from the client app with a new counter. The database engine names this failure TxnRetryCounterTooOld. It occurs during multi-shard transactions when nodes coordinate commit or abort decisions. A participant node rejects a request if its local retry counter is already higher. This safeguard stops old network messages from corrupting active two-phase commit workflows.
Your coding agents run with your access.
Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.
You get mongodb error 356 when a transaction message carries an old retry counter. You fix it by retrying the whole transaction from the client app with a new counter.
The database engine names this failure TxnRetryCounterTooOld. It occurs during multi-shard transactions when nodes coordinate commit or abort decisions. A participant node rejects a request if its local retry counter is already higher. This safeguard stops old network messages from corrupting active two-phase commit workflows.
Root causes of mongodb error 356 in clusters
The official database spec puts code 356 in two error groups. These groups are VoteAbortError and TwoPhaseDecisionAckError. Both groups tie directly to how sharded clusters run two-phase commits across nodes.
In a two-phase commit, a coordinator node collects votes from all participant shards. Network lag or packet drops can make the client or coordinator retry a step. Each retry bumps the transaction retry counter up by one.
If a delayed message from an earlier attempt arrives late, its counter is too old. The receiving shard spots the lower number and throws TxnRetryCounterTooOld right away. It votes to abort the stale attempt or rejects the old decision acknowledgment.
High network jitter between shards often triggers this condition in busy clusters. Slow disk writes on replica set primaries can also delay commit acknowledgments. Primary stepdowns during an active transaction also force retry counter updates across nodes. When old messages cross paths with new attempts, code 356 stops data drift.
How to diagnose and fix TxnRetryCounterTooOld in production
Start by checking your cluster logs for slow network paths or replica set elections. Look for spikes in transaction abort rates around the time code 356 appears. Check if a shard primary stepped down while the coordinator waited for votes.
You do not need wide admin permissions to inspect collection stats during diagnosis. MongoDB grants access through roles that apply down to the collection level. The built-in read role lets engineers run dbStats, collStats, find, and listIndexes safely. It leaves out write actions like insert, update, remove, and dropCollection.
MongoDB Authentication & Authorization + End-to-End Encryption: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Using a read role keeps production data safe while your team checks cluster health. If an engineer needs to test writes, the readWrite role adds data changes. Even then, keep role scope tight on a single database or collection.
To fix the error in your app, use the driver retry loop for transactions. Modern drivers catch transient transaction failures and restart the whole workflow with fresh counters. Keep your transactions short so network blips have less time to cause lag.
Break large batch updates into smaller writes whenever your app logic allows it. Shorter transactions hold locks for less time and finish before network timeouts hit. Also verify that all shard nodes sit in low-latency network zones with stable links.
Preventing retry counter mismatches at scale
Good cluster sizing stops many two-phase commit delays before they start. Make sure your shard primaries have enough disk speed and memory for peak loads. When nodes process votes fast, coordinators rarely need to bump retry counters.
Review how engineers and apps run ad-hoc queries against live shards as well. Heavy unindexed reads from manual sessions can starve transaction threads of disk bandwidth. Give analysts read roles on secondary nodes or limit query weight during peak hours. Masking sensitive fields like emails or card numbers in query results also helps safety. It rewrites matched fields in results while engineers inspect failed transaction records.
Add command guardrails and field masking beside MongoDB
Collection roles and IP rules control who connects to MongoDB. Yet an active client or agent session can still run a pipeline that writes data or reads raw personal fields. In practice, hoop.dev sits inline in the wire protocol beside your database as a sidecar for your MongoDB. For every command an agent or human sends in production, hoop.dev decides to pass, mask, block, or escalate it for review.
Using hoop.dev MongoDB controls, guardrails refuse banned commands before they run or hold risky calls for approval, and data masking rewrites matched fields in returned documents in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and generates evidence for GDPR, HIPAA, and PCI DSS programs. View the project on GitHub.
Frequently asked questions
What does TxnRetryCounterTooOld mean in MongoDB?
TxnRetryCounterTooOld is the official name for error code 356 in the database engine. It means a shard received a transaction vote or decision message with an outdated counter. The node rejects the stale message to protect two-phase commit safety.
Does error 356 cause data loss in sharded transactions?
No, this error actually prevents data loss and split-brain state across shards. It belongs to the VoteAbortError and TwoPhaseDecisionAckError categories, which halt stale requests. Your app just needs to retry the aborted transaction from the start.
What database role do I need to check collection stats during error spikes?
You only need the built-in read role on the affected database to check stats. This role allows dbStats, collStats, listCollections, and find actions without permitting data edits. It lets you inspect cluster state without risking accidental writes in production.
Short role checks help teams work fast. Simple rules make each review clear. Good logs keep every team safe. Scoped keys stop bad changes early.
Related guides
In this post