How to Fix MongoDB Error 106 (DistributedClockSkewed)

Published
Length
3 min read

Luis Felipe Dias

All posts

You see mongodb error 106 when clocks across cluster nodes drift too far apart. You fix this DistributedClockSkewed failure by syncing host time with a network time service. Distributed databases rely on shared timestamps to order writes and track cluster state. When one server falls behind or jumps ahead, the cluster rejects distributed requests. This check protects your data from out-of-order writes and broken transaction logs. Teams must fix the host clocks before normal cluster traffic can

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You see mongodb error 106 when clocks across cluster nodes drift too far apart. You fix this DistributedClockSkewed failure by syncing host time with a network time service.

Distributed databases rely on shared timestamps to order writes and track cluster state. When one server falls behind or jumps ahead, the cluster rejects distributed requests. This check protects your data from out-of-order writes and broken transaction logs. Teams must fix the host clocks before normal cluster traffic can resume.

What causes mongodb error 106 in a cluster

The database engine maps code 106 to the DistributedClockSkewed error name. This error fires when logical or physical clocks between nodes exceed safe drift limits. Replica sets and sharded clusters pass cluster time metadata with every internal message. If a node receives a timestamp too far in the future, it stops the operation.

Several common host issues cause clocks to drift across your database servers. A stopped time sync daemon is the most frequent culprit on virtual machines. Host pause events during live virtual machine migration can also freeze a system clock. Blocked network ports may stop servers from reaching upstream time servers. High CPU load or kernel bugs can also slow down local clock ticks.

When the clock skew grows too large, nodes cannot agree on write order. Read operations that depend on causal consistency or cluster time will also fail. Shard routers and replica set members then return the DistributedClockSkewed error to client apps.

How to diagnose and fix clock skew safely

Start by checking the system time on every replica set member and shard host. Compare the current UTC time across all nodes at the exact same moment. Check the status of your host time service on each server. Make sure the time daemon is running and can reach its upstream time pool.

Avoid jumping a live database server clock backward by a large amount at once. Sudden backward time jumps can confuse internal timers and stall replication threads. Instead, step a drifted secondary node down or take it out of rotation first. Sync the host clock, verify the time offset is near zero, and restart the node.

MongoDB Authentication & Authorization + End-to-End Encryption: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

During an incident, engineers often inspect cluster state and database stats to check recovery. You do not need to grant full admin rights for these diagnostic checks. MongoDB grants privileges through role-based rules down to a collection level of granularity. A built-in read role lets an engineer run database and collection stats safely. The read role blocks data changes like insert, update, remove, and collection drops. When sensitive fields appear in query results, data masking rewrites emails and phone numbers.

How to prevent future clock drift in production

Good host monitoring prevents clock skew from taking down production traffic again. Add an alert that measures the time offset between each node and your time server. Trigger a warning long before the offset approaches the database error limit. Check that every host startups script enables the network time service at boot.

Keep all nodes in a cluster pointed to the same internal time source. Using a single trusted time pool keeps relative drift between database nodes very low. Review firewall rules after network changes so time sync traffic never gets dropped.

Add command guardrails and field masking beside MongoDB

Collection roles and IP rules control who connects to MongoDB. Yet an active client or agent session can still run a pipeline that writes data or reads raw personal fields. In practice, hoop.dev sits inline in the wire protocol beside your database as a sidecar for your MongoDB. For every command an agent or human sends in production, hoop.dev decides to pass, mask, block, or escalate it for review.

Using hoop.dev MongoDB controls, guardrails refuse banned commands before they run or hold risky calls for approval, and data masking rewrites matched fields in returned documents in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and generates evidence for GDPR, HIPAA, and PCI DSS programs. View the project on GitHub.

Frequently asked questions

What does DistributedClockSkewed mean in MongoDB?

DistributedClockSkewed is the official name for error code 106 in the database engine. It means the system clocks on two or more cluster nodes are out of sync.

Can I fix error 106 from inside the database shell?

No, you cannot fix physical clock drift with a database query or role change. You must fix the operating system clock on the host server using a time daemon.

Does fixing the system clock require a cluster outage?

You can usually fix clock drift without full downtime by updating one node at a time. Fix secondary nodes first, then step down the primary node to sync its clock safely.

The query runs. The emails stay hidden.