How to Fix MongoDB Error 127 (UnrecoverableRollbackError)
You get mongodb error 127 when a former primary node cannot undo diverged writes. You fix this UnrecoverableRollbackError by resyncing the broken node from a healthy replica. Replica sets keep multiple copies of your data across different servers. When a network split happens, a primary node can lose contact with the rest. Another node takes over as the new primary and accepts fresh writes. If the old primary also took writes before stepping down, those writes conflict. When the old primary rej
Your coding agents run with your access.
Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.
You get mongodb error 127 when a former primary node cannot undo diverged writes. You fix this UnrecoverableRollbackError by resyncing the broken node from a healthy replica.
Replica sets keep multiple copies of your data across different servers. When a network split happens, a primary node can lose contact with the rest. Another node takes over as the new primary and accepts fresh writes. If the old primary also took writes before stepping down, those writes conflict. When the old primary rejoins the group, it tries to undo its isolated writes. If the gap is too large or oplog history is gone, the rollback fails. The database then raises code 127, known as UnrecoverableRollbackError, and stops replication.
Root causes of mongodb error 127 in replica sets
This failure starts when two nodes act as primary at different times. The old primary accepts client writes that never reach a majority of members. Meanwhile, the new primary moves ahead with a new timeline of operations. When the network heals, the old primary must roll back its local changes. It looks for a common point in the oplog shared with the new primary.
If the new primary already overwrote that common oplog point, the node cannot recover. The rollback also fails if the isolated node accepted too much data during the split. A sudden drop in oplog size or a long network outage makes this worse. Client apps that send writes without majority write checks also increase this risk. When the engine sees it cannot safely revert the local state, it halts.
How to diagnose and fix the failure in production
First, check the server logs on the broken member to confirm the error code. Look for the exact timestamp when the node rejoined the replica set. Before anyone inspects the data, make sure you follow strict role rules. MongoDB grants access through roles that define actions on specific resources. A role applies to its home database and can control access down to one collection.
MongoDB Authentication & Authorization + End-to-End Encryption: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Give your on-call responders the read role while they check the affected database. The read role lets them run find, listCollections, listIndexes, dbStats, and collStats. It blocks insert, update, remove, and dropCollection so nobody changes data by mistake. Never hand out full admin keys or broad readWrite rights just to read status.
To fix the halted node, you must rebuild its data from a healthy member. First, back up any isolated local files if you need to check lost writes. Next, stop the database service on the broken secondary node. Clear its data directory or move the files to a safe archive folder. Start the node again so it runs a clean initial sync from the primary. Wait for the sync to finish before routing read traffic back to that server. To stop future splits, tell client apps to wait for majority write acknowledgment. Also size your oplog window to cover your longest expected maintenance outage.
Check MongoDB commands and documents on the wire
MongoDB roles and SCRAM login decide who opens a session. They do not check single commands or redact sensitive fields inside returned documents. With hoop.dev, you run an open-source proxy, a sidecar for your MongoDB, inline in the MongoDB wire protocol. On every command an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Before a command runs, hoop.dev guardrails check its effect and refuse banned drops or writes with MongoDB error code 13, or hold risky commands for approval. On the way back, hoop.dev data masking rewrites sensitive fields in returned documents in memory, and every verdict is logged. See the open-source code on GitHub.
Frequently asked questions
What does UnrecoverableRollbackError mean in plain terms?
It means a replica set node took local writes that conflict with the main cluster. The node tried to undo those writes, hit a limit, and stopped syncing.
Do I lose data when fixing mongodb error 127?
The cluster keeps all writes that reached a majority of nodes. Any isolated writes on the broken node must be saved manually before you resync it.
Which role should engineers use to inspect collections after a rollback failure?
Use the built-in read role on the target database. It allows find and stats checks while blocking insert, update, remove, and dropCollection actions.
Related guides
In this post