Monad Testnet Outage Report — Host Connectivity Loss and Recovery
Overview
At 10:33 CET, BitCtrl monitoring detected loss of communications with the Monad testnet host. The server was fully unreachable over the network, preventing normal operator access and blocking immediate remediation.
Initial response began at 10:39 CET. Multiple connection attempts failed and remote reboot capabilities were not available, indicating an infrastructure-level issue beyond the node software stack. At 10:48 CET, the incident was escalated to datacenter support with a request for a manual restart.
Context
Datacenter support completed the manual reboot at 11:09 CET, restoring host availability. At 11:11 CET, post-reboot inspection began. The Monad service processes were running but the node was not syncing, indicating a stalled execution state after the host recovery. The Monad services were restarted at 11:15 CET, after which the node resumed syncing and returned to normal operation.
Datacenter support confirmed the root cause in follow-up: "The issue was caused by a faulty cable responsible for handling proper server restarts. The cable has been successfully replaced and the issue should no longer occur." By 11:18 CET, the validator was fully online and produced its first post-incident block. Reference block: https://testnet.monadvision.com/block/14881342
Datacenter Response
The issue was caused by a faulty cable responsible for handling proper server restarts. The cable has been successfully replaced and the issue should no longer occur.
Sources
- Duration: 10:33 -> 11:18 CET (45 minutes total impact window)
- Primary failure mode: host became unreachable; remote reboot unavailable, requiring datacenter intervention
- Secondary effect: node services were up after reboot but syncing stalled until services were restarted
- Resolution: manual reboot (DC) + Monad service restart restored syncing and block production