Solana Validators, Engineers Grapple With Blockchain Slowdown on Public Call

Solana’s superfast blockchain slowed to a relative crawl Thursday morning due to a mysterious congestion event that spurred debates about centralization, communication and transparency at the ecosystem’s critical stakeholder, Solana Labs.

Those tensions were on display midday after validators, peeved by slowdowns to Solana’s claimed Nasdaq-like speed, joined an impromptu roundtable. The problem was simple: A network that usually processes more than 2,000 transactions per second was stumbling along at speeds below 500. (Ethereum processes around 15 transactions per second.)

The issues had been mostly resolved by Thursday evening, but not before about 35 community members attended a public video chat, including a CoinDesk reporter and at least two Solana Labs engineers.

Solana’s 1,000-plus validators are the computing power behind a nearly $12 billion ecosystem of lending, trading and other decentralized finance (DeFi) platforms. Only a handful of them came to Thursday’s call. They weren’t happy.

The 50-minute conversation started with tough questions – Could Solana benefit from a centralized observation system? – ended with a surreal one – Can the next breakpoint be staged on yachts in the Mediterranean? – and left unanswered what exactly would happen next.

Read more: Solana Throws a Three-Day Party for Itself

“All of the apps running on top of Solana are the customers of the validators, right?” Brian Long, who operates the data site Validators.app and who convened the call, told the roundtable. “It’s just super urgent that we get this sorted out.”

Failing to do so could stunt the growth of a burgeoning financial system.

Daffy Durairaj, co-founder of popular Solana-based trading service Mango Markets, said Solana’s past outages have been pain points for the decentralized exchange’s outreach efforts.

“I’m trying to bring in these market-makers and traders and whatnot,” he said. “They just lose a lot of confidence when the nodes go down, and then they don’t know why.”

Past problems

Solana suffered an 18-hour outage in September that was resolved only after a hard restart. Thursday’s event, which was not an outage, was nowhere near as severe.

Still, it provided some validators a chance to air grievances to Solana Labs engineers.

They asked why the startup that has been critical to Solana’s development and that continues to be an omnipresent force in its growth remained so tight-lipped about the goings-on of an ecosystem that is supposed to be decentralized.

“We see the GitHub changes,” said the pseudonymous Zantetsu of the validator Shinobi Systems, who commanded much of Thursday’s conversation. “But, man, it just feels like a big black hole.”

Dan Albert, executive director of the Solana Foundation, denied that the project keeps unflattering information under wraps. But he acknowledged a “conservative stance” toward sharing sensitive details that Solana’s critics could spin against the network.

“The tech is hard, and I want you guys, really invested community members, to continue to feel ownership and feel like you’re being engaged by the developers and those who are working behind the scenes,” Albert said.

Read more: Grayscale Launches New Trust Dedicated to Solana

Behind the scenes, there’s “a fair bit of chaos,” Will Hickey, one of Solana Labs’ newest engineers, said on the call.

“My impression in my first three weeks is that there’s a fair bit of chaos,” Hickey said. “The amount of chaos that you might expect within a startup, but maybe an amount of chaos that’s a bit scary, given how much the Solana ecosystem has grown in the last year.”

Lingering questions

Although there was no clear reason for the slowdown, which had mostly resolved by Thursday evening, a handful of voices seemed comfortable with at least experimenting with a new layer of centralization that would aim to give Solana Labs more insight into the network.

Hickey polled the room on an “observability system.” He acknowledged such a system would be centralized by design. He said it could help the engineers pinpoint problems and prepare patches faster.

The room was skeptical but not to a fault. It would have to be a voluntary system, Zantestu said. Validators.app’s Long agreed. Solana Labs couldn’t mandate such a system without pushback.

“The crazy thing is we need them as a part of this effort, though, right? They have the insight to help us figure out where we’re going to pull the metrics from. They have the skill required to actually write the code without effing it up. And so we’ll have to pull them in at some point,” Long said.

UPDATE (Dec. 10, 1:40 UTC): Corrects job title of Dan Albert.