The SolarWinds supply chain compromise part 1

The SolarWinds saga started when FireEye revealed a breach compromised their internal red team tools on December 8th. They provided IDS rules to assist others with detecting similar breaches and investigated how they got compromised.

In the meantime, right before the holiday break, the world scrambled to evaluate their risk and mitigate or fix the situation. FireEye provided new IDS rules related to the SolarWinds’ breach.

I imported both sets of rules into our IDS but detected nothing which wasn’t all that surprising. A sophisticated attacker targeting an organisation like FireEye won’t burn their tools on just anyone even if ISPs are one of the top targets of cyber threats.

What’s remarkable that organisational security levels have risen to the point that attackers have to target trust vectors like ISPs. Healthcare is under attack because of the pandemic so LARG*net is a bigger target than normal because we serve healthcare.

We were using SolarWinds’ NCM (Network Configuration Management) tool at the time for configuration backups and auditing. We followed Fireeye’s mitigation recommendations from their blog post and took our server offline, evaluated hashes and other indicators of compromise, and determined that we were not compromised.

Ensure that SolarWinds servers are isolated / contained until a further review and investigation is conducted. This should include blocking all Internet egress from SolarWinds servers. - WHERE DID THIS COME FROM? FIREEYE BLOG, SOLARWINDS POST?

When SolarWinds recommended patching their software we complied. Once the server was patched, we brought it back online. Unfortunately the problem with a supply chain compromise is that patching can compromise your system if it wasn’t already. Given the circumstances, we were relatively confident that SolarWinds would not push compromised patches.

The bad guys never compromise so we followed recommendations on how to fix an admittedly extraordinary situation. The damage is done though. SolarWinds’ reputation is now in dispute and their products will attract the attention of security researchers for the foreseeable future. I expect we will hear of more vulnerabilities in their software as it undergoes intense scrutiny.

Trustwave responsible disclosure

Trustwave wasn’t involved in the original Fireeye/SolarWinds story but it inspired them to research and evaluate the SolarWinds product for vulnerabilities. They found a number of new 0-days independently.

Discovery of these vulnerabilities started in December and were responsibly disclosed to SolarWinds yet questions about the fundamental security of the product linger four months later.

At this point the already discovered vulnerabilities are almost irrelevant as many more researchers are evaluating SolarWinds products and it is very likely that many more exploits will be discovered.

SolarWinds blames an Intern?

To make matters worse current and former SolarWinds’ executives blamed an intern for the 2019 'solarwinds123' password leak that exposed their file server. We’re not saying this is how SolarWinds was backdoored but it makes you wonder when blame gets pushed all the way to the bottom.

"They violated our password policies and they posted that password on an internal, on their own private Github account." -former SolarWinds CEO Kevin Thompson

Obviously I have questions… An intern has the power to set a weak password widely used in production? And the ability to post it to a public Github account? Leading to a massive exposure because no one oversees an intern? This literally screams blame culture at full volume and that type of environment is not conducive to security efforts. It seems SolarWinds has some corporate culture work to do along with everything else.

Moving Forward

Software with full access to fundamental network equipment needs to be trusted given the sensitivity of access. The way this story has unfolded really doesn’t inspire much confidence.

I may have supported Zoom and their approach to fixing security flaws in previous blogs and fully admit that SolarWinds is doing well in quickly addressing their own flaws. But there’s one fundamental difference: Zoom doesn’t have high security requirements or the kind of sensitive access that SolarWinds does. For this reason we are moving on from SolarWinds.

The next few blogs will be a series documenting our search for a new NCM.