Polygon Blockchain Nodes Briefly Went Out of Sync, Affecting Explorer, Sowing Confusion

CORRECTION (Feb. 23, 4:42 UTC): Corrects story to say that Polygon nodes briefly went out of sync on Wednesday. An earlier version of this story stated that speculation over a possible blockchain malfunction was unfounded.

The Polygon blockchain ran into trouble on Wednesday after some of its nodes fell out of sync, causing a cascade of problems that hamstrung performance and knocked popular blockchain explorer PolygonScan offline.

The interruption on PolygonScan created the appearance that no transactions were taking place, sowing confusion and ultimately prompting Polygon’s leader, Sandeep Nailwal, to tweet that there were “issues” with the blockchain explorer.

“About 8:26 UTC, a few nodes went out of sync. This caused a reaction where some nodes could not validate blocks for a very brief period of time,” a spokesperson said, referring to the process by which decentralized networks such as Polygon add information to their permanent record.

Dapp developers reliant on impacted nodes struggled to get services back online, according to the founder of one infrastructure company, Rivet.

“While I would hesitate to call it a catastrophe, it is likely more people were impacted than were talking about it,” Rivet’s Greg Lang said of the incident.

Degraded performance

“There could have been a degradation in network performance temporarily," the spokesperson said, explaining the ramifications of some nodes falling behind. All nodes have come back in sync, the spokesperson said, and Polygon is investigating what caused the issue.

Polygon’s troubles appear to stem from an “unusually large” block reorganization that occurred two minutes before Polygon said nodes went out of sync, according to Lang. Rivet is a node infrastructure provider on Polygon.

When operators of a network publish two blocks at once, it creates a “fork” in the chain in which there are dueling accounts of the chain’s permanent record. A block reorganization is how the network resolves the discrepancy: It excludes one of the records in favor of what the network deems the correct, “canonical” record.

Small block reorganizations are commonplace on Polygon and other networks based on the Ethereum Virtual Machine (EVM), Lang said, and they’re not necessarily cause for concern. What was different on Wednesday was the size of the reshuffle: 157 blocks.

“The reorg was definitely outside the norm in size” for Polygon reorganizations, Lang said. “From Rivet’s standpoint it overran our ‘no-touch’ reorg correction mechanisms by at least a couple dozen blocks.”

Infrastructure providers handle reorgs differently, Lang said, but Rivet can automatically handle shuffles of roughly 125 or less. Beyond that, Lang said Rivet has to manually restore the chain history from backups. He said that likely happened on Wednesday.

Generally speaking, Lang said node operators can get thrown off kilter by blockchain reorganizations and could even “serve stale data” (data from excluded blocks) if they don’t get back on track fast.

“We certainly were in ‘all hands on deck’ mode,” Lang said of Rivet’s efforts to fix the issue.

Perhaps the most notable public-facing outcome of the blockchain's period of instability was an outage of popular block explorer PolygonScan. The Polygon spokesperson said the network issues “caused Polygonscan to get stuck” because the block explorer utilizes nodes that had fallen out of sync.

Polygon's Nailwal tweeted that PolygonScan was experiencing problems, and that the OKLink explorer was available as an alternative.