The U.S. Spent $2.2 Million on a Cybersecurity System That Wasn’t Implemented — and Might Have Stopped a Major Hack

ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for The Big Story newsletter to receive stories like this one in your inbox.

NEW YORK—As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.

The massive breach, which U.S. intelligence agencies say was “likely Russian in origin,” penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.

The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors”) that granted them access to an untold bounty of government and company secrets.

The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.

This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole”), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life’s work. In 2013, Popular Science named him as one of its “Brilliant Ten” scientists under 40.

Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn’t happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.

“In security, you almost never go from making something possible to impossible,” Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack.” Although the SolarWinds breach was a “really sneaky” approach, Cappos said, “in-toto definitely can protect against this. It’s very possible to catch it.”

In-toto’s system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto’s federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent.” Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure” and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it’s impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately,” said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies.”

Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams, head of Microsoft’s initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world.” (After Williams’ initial response, Microsoft declined to comment further.)

One senator blasted the government’s failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains.”

The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what’s done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn’t tampered with in transit.

Cappos and a team of colleagues have worked to develop the in-toto approach for years. It’s been up and running since 2018. The project received a three-year grant from the National Science Foundation that year, aimed at promoting “widespread practical use” of in-toto. (Later in 2018, President Donald Trump signed the Federal Acquisition Supply Chain Security Act, aimed at protecting government secrets from software supply-chain threats.)

In-toto could block and reveal countless cyberattacks that currently go undetected, according to Cappos, whose team includes Santiago Torres-Arias, an assistant electrical and computer engineering professor at Purdue University, and Reza Curtmola, co-director of the New Jersey Institute of Technology’s Cybersecurity Research Center. In an August 2019 paper and presentation to the USENIX computer conference, titled “in-toto: Providing farm-to-table guarantees for bits and bytes,” Cappos’ team reported studying 30 major supply-chain breaches dating back to 2010. In-toto, they concluded, would have prevented between 83% and 100% of those attacks.

“It’s available to everyone for free, paid for by the government, and should be used by everyone,” said Cappos. “People may still be able to break in and try to hack around it. But this is a necessary first step and will catch a ton of these things.” The slow pace of adoption is “really disappointing,” Cappos added. “In the long game, we’ll win. I just don’t know that we want to go through the pain that it’ll take for everyone to wise up.”

One of in-toto’s earliest adopters, starting in 2018, was Datadog, a SolarWinds competitor that provides monitoring software for internet cloud applications. Now a publicly traded company with 2020 revenues of nearly $600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog uses in-toto to protect the security of its software updates. In an NYU press release, Datadog staff security engineer Trishank Kuppusamy, who worked on the program’s design and implementation, said that what distinguishes in-toto is that it “has been designed against a very strong threat model that includes nation-state attackers.” (Datadog did not reply to ProPublica’s requests for comment.)

The General Services Administration, which provides access to software for federal government agencies, still lists SolarWinds products available for purchase. But it said in a statement that “compromised versions” of SolarWinds programs identified by DHS are no longer available.

SolarWinds itself declined to weigh in on whether its hack could have been prevented. “We are not going to speculate on in-toto and its capabilities,” a spokesman said in an emailed statement. “We are focused on protecting our customers, hardening our security and collaborating with the industry to understand the attack and prevent similar attacks in the future.”

Previously little known to the general public, SolarWinds is a public company based in Austin, Texas, with projected 2020 revenues of just over $1 billion. It boasts of providing software to 320,000 customers in 199 countries, including 499 of the Fortune 500 companies. In a recent SEC filing, the company said its flagship Orion products, the vehicle for the cyberattack, provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy.”

After the hack was discovered, SolarWinds’ stock plunged, and it is now facing shareholder lawsuits. The company has shifted aggressively into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite Washington lobbyists; a crisis-communications advisor; and the newly formed consulting team of Christopher Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (who was famously fired for contradicting Trump’s claims of mass voting fraud) and Alex Stamos, former security chief at Facebook.

News of what’s now known as the SolarWinds attack first came on Dec. 8. That’s when FireEye, perhaps the nation’s preeminent hack-hunter, announced that it had itself fallen victim to a “highly sophisticated state-sponsored adversary” that had broken into its servers and stolen its “Red Team tools,” which FireEye uses to try to hack into the computer networks of its clients as a test of their cyber-defenses. FireEye soon discovered the attackers had gained access through corrupted updates to the SolarWinds Orion network-monitoring software that it used.

On the evening of Dec. 13, CISA issued an emergency directive, identifying SolarWinds as ground zero for the hack and alerting federal agencies using Orion products to disconnect them immediately. Over the following weeks, investigators discovered that SolarWinds had been targeted back in early September 2019, when hackers started testing their ability to inject code into its software updates. After remaining undetected for months, they inserted malware in new updates between February and June 2020. SolarWinds estimated these infected updates affected “fewer than 18,000 of its customers.”

Precisely what the hackers saw, and stole, has yet to be determined and is under investigation. But the full impact of the breach is becoming clearer, as we now know it touches several tech companies, including Microsoft. The software giant has also labored to limit the damage by helping seize an internet domain in the U.S. that the hackers used to siphon data from some SolarWinds customers.

Stamos told the Financial Times, in an interview after being hired to help SolarWinds, that he believed the attackers had embedded hidden code that would continue to give them access to companies and government agencies for years. He compared the situation to Belgian and French farmers going out into their fields where two world wars were fought and discovering an “iron harvest” of unexploded ordnance each spring.

Dmitri Alperovitch, who co-founded CrowdStrike (the cybersecurity firm SolarWinds has hired to investigate the hack) before leaving last year to start a nonprofit policy group, said he thinks that, in theory, the in-toto system could work. But he warned that software is so complex, with many products and companies in the supply chain, that no one defense is a panacea. Still, he agrees that in-toto could provide protection, and said “it’s always a good thing to have more protection for supply chains.”

Russian intelligence services have clearly identified supply-chain attacks “as a much better way to get in,” offering “a much bigger set of targets,” Alperovitch said. “This is an indictment of the entire cybersecurity industry, as well as the intelligence community, that they were able to orchestrate such a broad, sweeping attack right under our noses.”

Top 25 Tech Stories from 2020

Select here for dynamic Air Force careers in Cyber and Tech

WASHINGTON ―The wait is finally over for the very end of 2020. The latest lists from Industry of top stories in tech, cyber, and STEM likely will loom much larger in the fog of 2021. Topping the charts for 2020 during National BDPA’s 45th Anniversary year were stories directly and indirectly related to COVID-19Big TechCybersecurity, and Social Unrest amidst a wider and much deeper ‘Digital Chasm‘ connecting underserved communities with their respective populations during a pandemic.

The Top 25

Most of the top 25 stories highlighted below for 2020 previously were shared with Industry, communities of color, traditionally marginalized communities, and underserved communities inside or on the covers of weekly and monthly publications. 

25. Earth Day Tech Summit: BDPA and UDC Earth Day Summits go virtual due to COVID-19. [April 2020 edition of bdpatoday]

24. Digital Divide: 51,000 laptops with Internet services were provided to students in Detroit, MI. [bdpatoday.com]

23U.S. Naval Academy: Midshipman First Class Sydney Barber becomes the first Black female to lead Brigade of Midshipmen. Upon graduation in May of 2021, MIDN Barber will receive her commission as an officer in the United States Marine Corps. [bdpatoday 11.14.20 ICYMI edition]

22. Zoom: In order to meet exponential growth and unprecedented demand, Oracle is selected as a cloud infrastructure provider for Zoom meetings. [bdpatoday 05.02.20 ICYMI edition]

21. U.S. Navy: LTJG Madeline G. Swegle becomes the U.S. Navy’s first Black female Tactical Air (TACAIR) jet pilot. [July 2020 edition of bdpatoday]

20. NIST: The National Institute of Standards and Technology’s (NIST) Privacy Framework Version 1.0 was released to help organizations identify and manage privacy risk for building innovative products and services while protecting individuals’ privacy. [ bdpatoday 01.18.20 ICYMI edition]

19. Digital Divide: T-Mobile launches Project 10-Million. [bdpatoday 10.03.20 ICYMI edition]

18. COVID-19: Apple and Google partner on contact-tracing technology. [bdpatoday.com]

17. Tesla: Headquarters and Gigafactory moves are heading to Austin, TX. [bdpatoday.com]

16. HPC: Lewis Hamilton wins 7th FIA Formula One championship powered by edge-to-core analytics with high-performance computing (HPC) from HPE. [bdpatoday 11.28.20 ICYMI edition]

15. COBOL: Federal, State, and Local governments call for more COBOL programmers to assist staff with stimulus, relief, and unemployment checks. [bdpatoday 04.04.20 ICYMI edition]

14. Exascale Day!: October 18th is Exascale Day. HPE, JEF, and BDPA welcomed “10 to the 18th power” or “10^18” during Exascale Day Weekend launching a series of supercomputer, HPC, and artificial intelligence (Ai) webinars. [October 2020 edition of bdpatoday]

13. Digital Divide: Best Buy Foundation awards a community grant to BDPA’s Tablets For Teensprogram and regional initiatives. [bdpatoday.com]

12. BDPA2020: National BDPA’s 45th Anniversary, Annual Technology Conference, Diversity Career Fair, I.T. Showcase, Mobile App Showcase, and the annual National High School Computer Competition (HSCC) collectively go 100-percent virtual for the first time in the Association’s history. #BDPA2020 was successfully delivered across all mobile platforms. [August 2020 Special Edition of bdpatoday

11. USASMDC: The University of Alabama in Huntsville (UAH) and BDPA Huntsville launch a new Cyber Workforce initiative with the U.S. Army’s Space and Missile Defense Command (USASMDC.) [bdpatoday 10.31.20 ICYMI edition]

10. Digital Divide: Microsoft awards $15 million in Community Skills Grants, an investment over three years to fifty (50) Black- and African American-led nonprofits that are working to increase skill development and economic opportunities. The H.O.P.E. Project DMV in partnership with National BDPA’s Greater Washington, D.C. Chapter (bdpadc.org) are one of Microsoft’s grant recipients for 2020. [bdpatoday.com]

9. AFRL: In fiscal year 2021 (FY21), the U.S. Air Force Research Laboratory’s (AFRL) Small Business Technology Transfer (STTR) program seeks to ink new and innovative deals with emerging small businesses and HBCU mission partners to meet or exceed the Department of the Air Force’s (DAF) priorities. [bdpatoday May 2020 edition] 

8. Cryptocurrency: Bitcoin’s latest rise in 2020. For the first time in its history, Bitcoin reached $20,000. According to CNBC, the world’s most-valuable virtual currency traded 5.6% higher on Wednesday, December 16, 2020, to a new price of around $20,600, taking its year-to-date gains north of 180%! [bdpatoday.com]

7. SpaceX: The launch of two NASA astronauts aboard the Crew Dragon spacecraft to the International Space Station (ISS) marked the first private spaceflight company to send a crewed spacecraft into space. [bdpatoday June 2020 edition] 

6. SolarWinds: Government agencies ‘hacked’ again ― stories from Federal, State, and Local agencies are still unfolding as we venture deeper into 2021. [bdpatoday.com]

5. Wall Street: Nasdaq advances “diversity” as stocks in 2020 across most major indices reached record highs. Nasdaq soon may adopt new listing rules related to board diversity and disclosures. [bdpatoday.com]

4. White House: As the daughter of two immigrants from Jamaica and India, Vice President-Elect Kamala Harris’ historic election breaks several barriers. “All eyes are on Georgia” as the next President of the Senate awaits Georgia’s runoff election results to determine control of the U.S. Senate. [bdpatoday November 2020 edition] 

3. Big Tech vs. Uncle Sam: On October 6, 2020, bdpatoday.com featured a story about the House Judiciary Committee’s Antitrust Subcommittee’s release of findings of its more than 16-month long investigation into the state of competition in the digital economy, especially the challenges presented by the dominance of Apple, Amazon, Google, and Facebook and their business practices. On October 20, 2020, bdpatoday.com shared a story from the Department of Justice (DOJ.) DOJ — along with eleven state Attorneys General — filed a civil antitrust lawsuit in the U.S. District Court for the District of Columbia to stop Google from unlawfully maintaining monopolies through anticompetitive and exclusionary practices in the search and search advertising markets and to remedy the competitive harms. In 2021 and the foreseeable future, “Big Tech” will have its day in Congress and the Courts from at least two branches of government. In the pipeline will be renewed battles over Section 230 of the Communications Act of 1934 (at 47 U.S.C. § 230). “Section 230” provides immunity for content providers and website publishers from third-party content. [bdpatoday.com]

2. COVID-19: “All Hands On Deck” for scientists, engineers, physicians, logisticians, STEM technicians, and I.T. professionals. “Digital divides” and “heath desert” challenges across the globe in underserved communities are hampering vaccine distributions as 2020 fades away. [bdpatoday December 2020 edition]

Number 1. Digital Divide and Social Unrest: Black Data Matters, Black Tech Matters, Black Consumers Matter, all lives matter, every student ― #BDPAfuture ― really matters. Founded by Earl A. Pace, Jr. in Philadelphia, PA as Black Data Processing Associates, BDPA was established in 1975 to promote and share awareness across traditionally underserved or marginalized communities of new “Data Processing” careers with related technical job openings in “Computer Science” fields. Today, BDPA’s mission has not waivered.

The pandemic of 2020 coupled with civil unrest across several U.S. cities revealed widening ‘digital chasms’ with news deserts, health deserts, and food deserts in every corner of America leading up to our top stories for 2020. Systemic racism, cultural biases, social discord, oppression from deep within our society’s soul, and gerrymandered redlining transgressions still are shrewdly perpetuated today through outdated laws, discriminatory policies, algorithmic bias, obsolete technology, and dilapidated infrastructures. To this end, our lead success stories feature BDPA, its ICT Industry partners, major corporations, and local BDPA Chapter mission-partners making impact investments to help eliminate “deserts” within digitally divided communities as millions of students, teachers, and parents where forced home; many with little or no access to high speed Internet services. [bdpatoday.com]

In Memoriam

Last October, National BDPA celebrated the life of Vivian C. Wilson, the first women elected to the Association’s chief executive role of National BDPA President.

BDPA Nation also said goodbyes in 2020 to iconic Black Panther star Chadwick Boseman; Dr. George Robert Carruthers, an inventor, physicist, engineer and space scientist; Roderick “Rod” Wesley Flakes, former President, BDPA (Boston) Mass MetroWest Chapter and engineer at Digital Equipment Corporation (DEC); U.S. Supreme Court Justice Ruth Bader Ginsburg; one of NASA’s “Hidden Figures” Katherine Johnson; U.S. Representative John LewisNorman Shakespeare, former Vice President of Strategic Planning, BDPA (Boston) Mass MetroWest Chapter; model and restaurateur Barbara “B” Smith; and Charles “Chuck” Yeager, World War II ace fighter pilot and U.S. Air Force quintessential test pilot.

Up Next

2021 has arrived! In May 2021, bdpatoday (ISSN 1946-1429) launches its 15th year to proudly serve more I.T. technicians and cybersecurity professionals, new HBCU Chapters, new student members, and new consumers in every industry. To add your team’s success stories in tech, cyber, and STEM along with new campaigns or press releases, contact our team directly at: info@bdpatoday.org – or – press@bdpadc.org.

₵ode on¢e. $ell Many!  bdpatoday.com

Select here for bdpatoday Archives

______________________

Key words: #5G #Ai #AirForce #AFRL #Army #BDPA #bdpatoday #BigTech #Bitcoin #COBOL #cryptocurrency #cyber #DigitalDivide #diversity #DOD #DOJ #Gigafactory #hackers #HBCU #HPC #HSCC #Huntsville #ICT #innovation #MarineCorps #Navy #Nasdaq #SBIR #SEC #Section230 #SMDC #SpaceForce #STEM #STTR #supercomputer #tech #technology #USASMDC #WallStreet

Justice Department Sues Google For Violating Antitrust Laws

WASHINGTON—Today, the Department of Justice — along with eleven state Attorneys General — filed a civil antitrust lawsuit in the U.S. District Court for the District of Columbia to stop Google from unlawfully maintaining monopolies through anticompetitive and exclusionary practices in the search and search advertising markets and to remedy the competitive harms. The participating state Attorneys General offices represent Arkansas, Florida, Georgia, Indiana, Kentucky, Louisiana, Mississippi, Missouri, Montana, South Carolina, and Texas.

“Today, millions of Americans rely on the Internet and online platforms for their daily lives.  Competition in this industry is vitally important, which is why today’s challenge against Google — the gatekeeper of the Internet — for violating antitrust laws is a monumental case both for the Department of Justice and for the American people,” said Attorney General William Barr. “Since my confirmation, I have prioritized the Department’s review of online market-leading platforms to ensure that our technology industries remain competitive.  This lawsuit strikes at the heart of Google’s grip over the internet for millions of American consumers, advertisers, small businesses and entrepreneurs beholden to an unlawful monopolist.”

 “As with its historic antitrust actions against AT&T in 1974 and Microsoft in 1998, the Department is again enforcing the Sherman Act to restore the role of competition and open the door to the next wave of innovation—this time in vital digital markets,” said Deputy Attorney General Jeffrey A. Rosen.

As one of the wealthiest companies on the planet with a market value of $1 trillion, Google is the monopoly gatekeeper to the internet for billions of users and countless advertisers worldwide. For years, Google has accounted for almost 90 percent of all search queries in the United States and has used anticompetitive tactics to maintain and extend its monopolies in search and search advertising.  

As alleged in the Complaint, Google has entered into a series of exclusionary agreements that collectively lock up the primary avenues through which users access search engines, and thus the internet, by requiring that Google be set as the preset default general search engine on billions of mobile devices and computers worldwide and, in many cases, prohibiting preinstallation of a competitor. In particular, the Complaint alleges that Google has unlawfully maintained monopolies in search and search advertising by:

  • Entering into exclusivity agreements that forbid preinstallation of any competing search service.
  • Entering into tying and other arrangements that force preinstallation of its search applications in prime locations on mobile devices and make them undeletable, regardless of consumer preference.
  • Entering into long-term agreements with Apple that require Google to be the default – and de facto exclusive – general search engine on Apple’s popular Safari browser and other Apple search tools.
  • Generally using monopoly profits to buy preferential treatment for its search engine on devices, web browsers, and other search access points, creating a continuous and self-reinforcing cycle of monopolization.

These and other anticompetitive practices harm competition and consumers, reducing the ability of innovative new companies to develop, compete, and discipline Google’s behavior. 

The antitrust laws protect our free market economy and forbid monopolists from engaging in anticompetitive practices. They also empower the Department of Justice to bring cases like this one to remedy violations and restore competition, as it has done for over a century in notable cases involving monopolists over other critical industries undergirding the American economy like Standard Oil and the AT&T telephone monopoly. Decades ago the Department’s case against Microsoft recognized that the antitrust laws forbid anticompetitive agreements by high-technology monopolists to require preinstalled default status, to shut off distribution channels to rivals, and to make software undeletable. The Complaint alleges that Google is using similar agreements itself to maintain and extend its own dominance. 

The Complaint alleges that Google’s anticompetitive practices have had harmful effects on competition and consumers. Google has foreclosed any meaningful search competitor from gaining vital distribution and scale, eliminating competition for a majority of search queries in the United States. By restricting competition in search, Google’s conduct has harmed consumers by reducing the quality of search (including on dimensions such as privacy, data protection, and use of consumer data), lessening choice in search, and impeding innovation. By suppressing competition in advertising, Google has the power to charge advertisers more than it could in a competitive market and to reduce the quality of the services it provides them. Through filing the lawsuit, the Department seeks to stop Google’s anticompetitive conduct and restore competition for American consumers, advertisers, and all companies now reliant on the internet economy.

Google is a limited liability company organized and existing under the laws of the State of Delaware, and is headquartered in Mountain View, California. Google is owned by Alphabet Inc., a publicly traded company incorporated and existing under the laws of the State of Delaware and headquartered in Mountain View, California.

Source/Seal: Department of Justice • Photo/Graphic: Google

A D V E R T I S E M E N T

%d bloggers like this: