The U.S. Spent $2.2 Million on a Cybersecurity System That Wasn’t Implemented — and Might Have Stopped a Major Hack

ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for The Big Story newsletter to receive stories like this one in your inbox.

NEW YORK—As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.

The massive breach, which U.S. intelligence agencies say was “likely Russian in origin,” penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.

The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors”) that granted them access to an untold bounty of government and company secrets.

The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.

This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole”), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life’s work. In 2013, Popular Science named him as one of its “Brilliant Ten” scientists under 40.

Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn’t happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.

“In security, you almost never go from making something possible to impossible,” Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack.” Although the SolarWinds breach was a “really sneaky” approach, Cappos said, “in-toto definitely can protect against this. It’s very possible to catch it.”

In-toto’s system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto’s federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent.” Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure” and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it’s impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately,” said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies.”

Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams, head of Microsoft’s initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world.” (After Williams’ initial response, Microsoft declined to comment further.)

One senator blasted the government’s failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains.”

The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what’s done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn’t tampered with in transit.

Cappos and a team of colleagues have worked to develop the in-toto approach for years. It’s been up and running since 2018. The project received a three-year grant from the National Science Foundation that year, aimed at promoting “widespread practical use” of in-toto. (Later in 2018, President Donald Trump signed the Federal Acquisition Supply Chain Security Act, aimed at protecting government secrets from software supply-chain threats.)

In-toto could block and reveal countless cyberattacks that currently go undetected, according to Cappos, whose team includes Santiago Torres-Arias, an assistant electrical and computer engineering professor at Purdue University, and Reza Curtmola, co-director of the New Jersey Institute of Technology’s Cybersecurity Research Center. In an August 2019 paper and presentation to the USENIX computer conference, titled “in-toto: Providing farm-to-table guarantees for bits and bytes,” Cappos’ team reported studying 30 major supply-chain breaches dating back to 2010. In-toto, they concluded, would have prevented between 83% and 100% of those attacks.

“It’s available to everyone for free, paid for by the government, and should be used by everyone,” said Cappos. “People may still be able to break in and try to hack around it. But this is a necessary first step and will catch a ton of these things.” The slow pace of adoption is “really disappointing,” Cappos added. “In the long game, we’ll win. I just don’t know that we want to go through the pain that it’ll take for everyone to wise up.”

One of in-toto’s earliest adopters, starting in 2018, was Datadog, a SolarWinds competitor that provides monitoring software for internet cloud applications. Now a publicly traded company with 2020 revenues of nearly $600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog uses in-toto to protect the security of its software updates. In an NYU press release, Datadog staff security engineer Trishank Kuppusamy, who worked on the program’s design and implementation, said that what distinguishes in-toto is that it “has been designed against a very strong threat model that includes nation-state attackers.” (Datadog did not reply to ProPublica’s requests for comment.)

The General Services Administration, which provides access to software for federal government agencies, still lists SolarWinds products available for purchase. But it said in a statement that “compromised versions” of SolarWinds programs identified by DHS are no longer available.

SolarWinds itself declined to weigh in on whether its hack could have been prevented. “We are not going to speculate on in-toto and its capabilities,” a spokesman said in an emailed statement. “We are focused on protecting our customers, hardening our security and collaborating with the industry to understand the attack and prevent similar attacks in the future.”

Previously little known to the general public, SolarWinds is a public company based in Austin, Texas, with projected 2020 revenues of just over $1 billion. It boasts of providing software to 320,000 customers in 199 countries, including 499 of the Fortune 500 companies. In a recent SEC filing, the company said its flagship Orion products, the vehicle for the cyberattack, provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy.”

After the hack was discovered, SolarWinds’ stock plunged, and it is now facing shareholder lawsuits. The company has shifted aggressively into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite Washington lobbyists; a crisis-communications advisor; and the newly formed consulting team of Christopher Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (who was famously fired for contradicting Trump’s claims of mass voting fraud) and Alex Stamos, former security chief at Facebook.

News of what’s now known as the SolarWinds attack first came on Dec. 8. That’s when FireEye, perhaps the nation’s preeminent hack-hunter, announced that it had itself fallen victim to a “highly sophisticated state-sponsored adversary” that had broken into its servers and stolen its “Red Team tools,” which FireEye uses to try to hack into the computer networks of its clients as a test of their cyber-defenses. FireEye soon discovered the attackers had gained access through corrupted updates to the SolarWinds Orion network-monitoring software that it used.

On the evening of Dec. 13, CISA issued an emergency directive, identifying SolarWinds as ground zero for the hack and alerting federal agencies using Orion products to disconnect them immediately. Over the following weeks, investigators discovered that SolarWinds had been targeted back in early September 2019, when hackers started testing their ability to inject code into its software updates. After remaining undetected for months, they inserted malware in new updates between February and June 2020. SolarWinds estimated these infected updates affected “fewer than 18,000 of its customers.”

Precisely what the hackers saw, and stole, has yet to be determined and is under investigation. But the full impact of the breach is becoming clearer, as we now know it touches several tech companies, including Microsoft. The software giant has also labored to limit the damage by helping seize an internet domain in the U.S. that the hackers used to siphon data from some SolarWinds customers.

Stamos told the Financial Times, in an interview after being hired to help SolarWinds, that he believed the attackers had embedded hidden code that would continue to give them access to companies and government agencies for years. He compared the situation to Belgian and French farmers going out into their fields where two world wars were fought and discovering an “iron harvest” of unexploded ordnance each spring.

Dmitri Alperovitch, who co-founded CrowdStrike (the cybersecurity firm SolarWinds has hired to investigate the hack) before leaving last year to start a nonprofit policy group, said he thinks that, in theory, the in-toto system could work. But he warned that software is so complex, with many products and companies in the supply chain, that no one defense is a panacea. Still, he agrees that in-toto could provide protection, and said “it’s always a good thing to have more protection for supply chains.”

Russian intelligence services have clearly identified supply-chain attacks “as a much better way to get in,” offering “a much bigger set of targets,” Alperovitch said. “This is an indictment of the entire cybersecurity industry, as well as the intelligence community, that they were able to orchestrate such a broad, sweeping attack right under our noses.”

Justice Department Sues Google For Violating Antitrust Laws

WASHINGTON—Today, the Department of Justice — along with eleven state Attorneys General — filed a civil antitrust lawsuit in the U.S. District Court for the District of Columbia to stop Google from unlawfully maintaining monopolies through anticompetitive and exclusionary practices in the search and search advertising markets and to remedy the competitive harms. The participating state Attorneys General offices represent Arkansas, Florida, Georgia, Indiana, Kentucky, Louisiana, Mississippi, Missouri, Montana, South Carolina, and Texas.

“Today, millions of Americans rely on the Internet and online platforms for their daily lives.  Competition in this industry is vitally important, which is why today’s challenge against Google — the gatekeeper of the Internet — for violating antitrust laws is a monumental case both for the Department of Justice and for the American people,” said Attorney General William Barr. “Since my confirmation, I have prioritized the Department’s review of online market-leading platforms to ensure that our technology industries remain competitive.  This lawsuit strikes at the heart of Google’s grip over the internet for millions of American consumers, advertisers, small businesses and entrepreneurs beholden to an unlawful monopolist.”

 “As with its historic antitrust actions against AT&T in 1974 and Microsoft in 1998, the Department is again enforcing the Sherman Act to restore the role of competition and open the door to the next wave of innovation—this time in vital digital markets,” said Deputy Attorney General Jeffrey A. Rosen.

As one of the wealthiest companies on the planet with a market value of $1 trillion, Google is the monopoly gatekeeper to the internet for billions of users and countless advertisers worldwide. For years, Google has accounted for almost 90 percent of all search queries in the United States and has used anticompetitive tactics to maintain and extend its monopolies in search and search advertising.  

As alleged in the Complaint, Google has entered into a series of exclusionary agreements that collectively lock up the primary avenues through which users access search engines, and thus the internet, by requiring that Google be set as the preset default general search engine on billions of mobile devices and computers worldwide and, in many cases, prohibiting preinstallation of a competitor. In particular, the Complaint alleges that Google has unlawfully maintained monopolies in search and search advertising by:

  • Entering into exclusivity agreements that forbid preinstallation of any competing search service.
  • Entering into tying and other arrangements that force preinstallation of its search applications in prime locations on mobile devices and make them undeletable, regardless of consumer preference.
  • Entering into long-term agreements with Apple that require Google to be the default – and de facto exclusive – general search engine on Apple’s popular Safari browser and other Apple search tools.
  • Generally using monopoly profits to buy preferential treatment for its search engine on devices, web browsers, and other search access points, creating a continuous and self-reinforcing cycle of monopolization.

These and other anticompetitive practices harm competition and consumers, reducing the ability of innovative new companies to develop, compete, and discipline Google’s behavior. 

The antitrust laws protect our free market economy and forbid monopolists from engaging in anticompetitive practices. They also empower the Department of Justice to bring cases like this one to remedy violations and restore competition, as it has done for over a century in notable cases involving monopolists over other critical industries undergirding the American economy like Standard Oil and the AT&T telephone monopoly. Decades ago the Department’s case against Microsoft recognized that the antitrust laws forbid anticompetitive agreements by high-technology monopolists to require preinstalled default status, to shut off distribution channels to rivals, and to make software undeletable. The Complaint alleges that Google is using similar agreements itself to maintain and extend its own dominance. 

The Complaint alleges that Google’s anticompetitive practices have had harmful effects on competition and consumers. Google has foreclosed any meaningful search competitor from gaining vital distribution and scale, eliminating competition for a majority of search queries in the United States. By restricting competition in search, Google’s conduct has harmed consumers by reducing the quality of search (including on dimensions such as privacy, data protection, and use of consumer data), lessening choice in search, and impeding innovation. By suppressing competition in advertising, Google has the power to charge advertisers more than it could in a competitive market and to reduce the quality of the services it provides them. Through filing the lawsuit, the Department seeks to stop Google’s anticompetitive conduct and restore competition for American consumers, advertisers, and all companies now reliant on the internet economy.

Google is a limited liability company organized and existing under the laws of the State of Delaware, and is headquartered in Mountain View, California. Google is owned by Alphabet Inc., a publicly traded company incorporated and existing under the laws of the State of Delaware and headquartered in Mountain View, California.

Source/Seal: Department of Justice • Photo/Graphic: Google

A D V E R T I S E M E N T

%d bloggers like this: