What to Do After Your First Cyber-Attack – So There’s No Second One

Phemex shares its lessons learned from its unprecedented incursion.

Even thought leaders are susceptible to the risks of running a business online. Phemex, a hybrid exchange that features best-of-breed processes of both centralized and decentralized platforms, suffered an attack from a serious threat actor at the end of January.

Rather than shy away from this unwelcome event, the Phemex team decided to be forthright and transparent about it – and maybe that’s the most important lesson they could take away from it.

“We want to use this piece to address the incident, talk about how we handled it, and explain what we’ve done to prevent such incidents in the future,” says Phemex CEO Federico Variola.

He stressed that, while the attack came from a highly sophisticated threat actor, the vast majority of user funds were never at risk and the exchange covered all users’ losses.

“We also resumed core operations as quickly as possible and immediately revamped our hot wallet security infrastructure to greatly minimize these security risks in the future,” he continued.

Attack and defense

The attacker has a history of crypto hacks and is considered to be extremely sophisticated, so the nature of the cyber-attack was complex and difficult to prevent. These perpetrators have not been publicly identified by law enforcement, but likely reside in a state that supports this kind of action and are probably insulated from any prosecution or other legal action.

Bybit's recent hack seems to be related to the same group, according to Variola, but the incidents differ.

“In our case, a hot wallet was targeted,” he said. “In Bybit's case it was their main ETH cold wallet.”

Phemex employs separate hot- and cold-wallet systems in order to minimize risk of loss during “edge cases” – cybersecurity jargon for a problem or situation that happens at the extreme limit of what's normal or expected.

“Only funds from our hot wallet were stolen, and that loss minimization is exactly why we have separate hot and cold wallets in the first place,” according to Variola. “What happened is undoubtedly a negative event, but it’s within the limits of what’s acceptable for our exchange to handle.”

The attack was perpetrated via social engineering, targeting Phemex employees via Telegram. A full incident report appears on Medium.

Estimates put the total value of the stolen funds at $85 million, so transparency and trustworthiness to users were among Phemex’s highest priorities as the attack was occurring.

“We immediately notified users,” Variola continues, “and could assure them that their funds were safe by encouraging them to check for themselves using our self-proving Merkle Tree Proof-of-Reserves Tool.”

Once Phemex contained and assessed the damage, the next step was to minimize the harm. Some funds have already been recovered. Stolen funds popping up on other exchanges were immediately frozen.

“Recovery of funds is currently still ongoing and we are hopeful to recover a decent amount of stolen assets,” Variola says. Even so, “we still have resources to operate at full performance.”

In the meantime, Phemex is working with law enforcement, cybersecurity firms and other crypto platforms with the recovery process. The exchange was able to restore core functionality to users within 24 hours – possibly one of the fastest recoveries from a hack by any established crypto exchange. Following that, Phemex implemented a strict, manual review of deposit and withdrawal transactions to reinforce security and ensure no malicious transactions were being made in the immediate aftermath.

Lessons learned, actions taken

Immediately after the breach, Phemex’s technical team designed and implemented a new, more robust hot-wallet security infrastructure.

“A major lesson we’ve learned and reflected on is that Phemex has grown very fast during the latest bull market and some of our operating procedures lagged behind our growth,” Variola says. “This cyber-attack showed that the kind of security measures that may have been serviceable for our previous size are now no longer acceptable for our current scale.”

Phemex’s new structure is designed with a zero-trust architecture in mind and leverages cutting-edge Enclave technology. This includes AWS Nitro to achieve robust, chip-level security for hot wallets.

While that solves the immediate problem, it wouldn’t put Phemex ahead of the hackers. So the team made moves to protect all wallets which any of its users might hold.

“We plan to employ a tiered-wallet system with cold wallets,” Variola says. “It would also apply to hot wallets – which will hold a much smaller proportion of our funds moving forward.”

The tiered system also applies to warm wallets, which combine hot wallets’ internet connection, speed and efficiency with cold wallets’ enhanced security and manual control.

Phemex is also increasing the workforce dedicated to infrastructure security, with different teams overseeing separate elements and fewer individuals having access to the entire system. From end to end, Variola promises, every task will be reviewed by industry-leading third parties.

That could slow down the pace of Phemex’s service delivery by a step, but Variola’s team is convinced it must be done.

“The operations of our exchange will be more complex using the new system, but this cannot be avoided because security is of highest priority,” Variola says. “We are extremely confident in the new system and we’re applying for third-party certifications on these security standards.”