Tim Brown will remember 12 December 2020 for ever.
It was the day the software company SolarWinds was notified it had been hacked by Russia.
Brown, the chief information security officer at SolarWinds, immediately understood the implications: any of the company’s more than 300,000 global clients could be affected too.
The exploit allowed the hackers remote access to the systems of customers that had installed SolarWinds’ network software Orion, including the US treasury department, the US department of commerce’s National Telecommunications and Information Administration, along with thousands of companies and public institutions.
Brown says he was “running on adrenaline” in the first few days after the attack.
It was during the early stages of the Covid pandemic when full-time work-from-home was the norm, but the company’s email was compromised and couldn’t be used to communicate with staff.
“We gave up on the phones and just everybody came into the office and we got Covid testing,” Brown says. “I lost 25 pounds in about 20 days … just going, going, going.”
He appeared on CNN and 60 Minutes, and in every major newspaper.
“The world’s on fire. You’re trying to get information out and trying to have people understand what’s safe and what’s not safe.”
The company switched to Proton email and Signal while its email was compromised, Brown says. He was taking calls from companies and government agencies across the globe, including the US army and the Covid vaccine program Operation Warp Speed.
“You get the world wanting verbal communication not written communication. And that is a kind of an important lesson: you can write things down, but they want to talk to the [chief information security officer],” says Brown, who spoke at Melbourne’s CyberCon on Friday.
“They want to be able to hear colour around the outside of it, so very important to be prepared for that kind of response.”
How the cyber-attack unfolded
The notification about the hack came in a phone call from Kevin Mandia, the founder of the cybersecurity firm Mandiant, to SolarWinds’ then CEO Kevin Thompson.
Mandia told Thompson that SolarWinds had “shipped tainted code” to its Orion software, which helps organisations monitor outages on their computer networks and servers.
The exploit in Orion was being used to attack government agencies, Mandia told Thompson.
“We could see in that code [it] was not ours, so when we got that, it was ‘all right, this is real’,” Brown recalls.
The Texas-based SolarWinds determined that 18,000 people had downloaded the tainted product, which the hackers, later attributed to the Russian Foreign Intelligence Service, were able to insert into Orion in the build environment where source code is turned into software.
The news broke on the Sunday. SolarWinds notified the stock market before it opened on Monday.
The original estimate that up to 18,000 clients could be affected was later revised down to about 100 government agencies and companies that actually were.
“It would have been nice to know that on day one, but that was the truth of the matter, right?” Brown says. “We weren’t really the target. We were just a route to the target.”
SolarWinds called in CrowdStrike, KPMG and the law firm DLA Piper to deal with the response and investigation.
Aftermath: the heart attack
SolarWinds stopped work on new features for the next six months and its team of 400 engineers focused on systems and security to get the company back on its feet.
“We really took transparency to heart – how can we make sure people realise [what] threat actor models [are out there], what they do, how they do reconnaissance, how they then do an attack [and] how they then leave.”
Brown says the company’s customer renewal rate fell into the 80% range in the first few months after the incident, but has since returned to more than 98%.
But then came the legal implications.
The Biden administration imposed sanctions and expelled Russian diplomats in 2021, partly in response to the attack.
SolarWinds settled a class action lawsuit over the attack in 2022 for US$26m. The Securities and Exchange Commission (SEC) then filed a lawsuit against SolarWinds and Brown personally in October 2023, accusing the company and Brown of misleading investors over its claims about cybersecurity protections, and failing to disclose known vulnerabilities.
Brown was in Zurich when he found out he was being charged.
“When I walked up a hill, I would lose my breath. My arms would get heavy, my chest would get tight. I was just not getting enough oxygen,” he says. “I did a silly thing. I flew home … I couldn’t walk from the terminal to my car without stopping. That’s a walk I had done thousand of times.”
He was having a heart attack. When he got home, his wife took him to the hospital, where he underwent surgery. He has since recovered.
“Stress keeps building up and I thought I was managing it well and I didn’t proactively go to a doctor,” he says.
Brown says he now advocates for companies going through similar incidents to employ psychiatrists to help staff process the stress.
“The stress level was pumped up, and then it just went over the edge, but stress was building up all the time.”
A confidential jointly proposed settlement with the SEC was announced in July, but has yet to be approved. The US government shutdown has delayed the finalisation of the agreement.
Brown has remained with SolarWinds throughout the process.
“It happened on my watch, that’s how I look at it. There are reasons why it occurred, nation state attack, et cetera, but still it happened on my watch,” he says.
“I guess I’m stubborn. But it was just very important for us to get through this whole cycle, so leaving wasn’t an option until it was done.”
