Worldwide Crowdstrike Outage


New York Times

July 19, 2024 5:27 p.m. ET

Airlines, industries and services across the globe were impacted by tech outages attributed to a software update from CrowdStrike, a widely used cybersecurity firm.Caroline Chia/Reuters

Across the world, critical businesses and services including airlines, hospitals, train networks and TV stations, were disrupted on Friday by a global tech outage affecting Microsoft users.

In many countries, flights were grounded, workers could not get access to their systems and, in some cases, customers could not make card payments in stores. While some of the problems were resolved within hours, many businesses, websites and airlines continued to struggle to recover.

What happened?

A series of outages rippled across the globe as information displays, login systems and broadcasting networks went dark.

The problem affecting the majority of services was caused by a flawed update by CrowdStrike, an American cybersecurity firm, whose systems are intended to protect users from hackers. Microsoft said on Friday that it was aware of an issue affecting machines running “CrowdStrike Falcon.”

But Microsoft had also said there was an earlier outage affecting U.S. users of Azure, its cloud service system. Some users may have been affected by both. Even as CrowdStrike sent out a fix, some systems were still affected by midday in the United States as businesses needed to make manual updates to their systems to resolve the issue.

George Kurtz, the president and chief executive of CrowdStrike, said on Friday morning that it could take some time for some systems to recover.


New York Times

https://www.nytimes.com/live/2024/07/19/business/global-tech-ofutage

Outage for Microsoft Users Knocks Out Systems for Airlines and Hospitals in Chaotic Day

Companies across the world reported disruptions, citing technical issues from a cybersecurity software update.

July 19, 2024, 6:02 p.m. ET

A global technology outage, attributed to a glitch in a software update issued by the cybersecurity firm CrowdStrike, wreaked havoc on airlines, health care systems, banks and scores of other businesses and services around the world on Friday. The disruption, which reached what some experts called “historic” proportions, was a stunning example of the global economy’s fragile dependence on certain software, and the cascading effect it can have when things go wrong.

The software update resulted in crashes of machines running the Microsoft Windows operating system. George Kurtz, CrowdStrike’s chief executive, said it was not a security incident or a cyberattack. He said a fix had been sent out but warned that it could take some time to be put in place.

Lazaro Gamio

July 19, 2024, 4:27 p.m. ET2 hours ago

This outage was unlike how other widespread disruptions tend to occur. Here’s how.

Joe Rennison

July 19, 2024, 4:05 p.m. ET2 hours ago

Stocks ended the day lower, with the S&P 500 down 0.7 percent and the tech-heavy Nasdaq Composite down 0.8 percent – a modest end to a globally disruptive event that left markets across Asia and Europe nursing losses. CrowdStrike itself bore the brunt of the investors’ response, down by around 11 percent for the day.

Karl Russell

July 19, 2024, 3:00 p.m. ET3 hours ago

CrowdStrike’s stock price has taken a hit from the outage. As of the afternoon, its shares had fallen by roughly a quarter from their high just 10 days ago.

CrowdStrike’s stock price so far this year

Danielle Kaye

July 19, 2024, 2:55 p.m. ET3 hours ago

While major U.S. grocers like Kroger reported largely normal operations on Friday, Starbucks grappled with widespread disruptions. Mobile orders were still down in the early afternoon, a Starbucks spokeswoman said, but the coffee chain was still serving customers at the vast majority of its stores.

Danielle Kaye

July 19, 2024, 3:04 p.m. ET3 hours ago

Daniel Huttinger of Bozeman, Mont., said the drive-thru at his local Starbucks was closed on Friday morning. Baristas had to deal with confused customers, he said, and they turned to the old-school method of tracking orders: writing them by hand on cups.

Jim Wilson

July 19, 2024, 2:49 p.m. ET4 hours ago

Terminal 3 in San Francisco International Airport had long lines of weary travelers, but they seemed to be taking it all in stride for the most part. Some had been waiting since yesterday for their flights. Plenty of snacks and water were available to the passengers, some of whom resorted to finding a spot on the carpet to get a few minutes of shut eye.

The Transportation Department said on Friday that U.S.-based airlines must provide affected passengers compensation for flight disruptions set off by the global software outage that had roiled air travel, spurring thousands of delays and cancellations worldwide and in some instances, stranding travelers abroad.

The move is a significant reversal from how airlines were treating the situation earlier in the day, as something inherently outside their control for which their only obligation to passengers was free rebooking of their flights and nothing further. Delta Air Lines and United Airlines had both previously told travelers that they would not foot the bill for lodging.

Sheera Frenkel

July 19, 2024, 2:40 p.m. ET4 hours ago

At CrowdStrike, engineers described an atmosphere of confusion as the company struggled to contain the damage on Friday.

Executives urged employees not to speculate on why the mistake had happened and directed them to instead focus on a fix for the millions of computers that were affected, said two engineers who spoke on the condition of anonymity because they were not authorized to speak publicly. Computers not connected to the cloud required a physical fix to the error introduced by CrowdStrike, they said, which could take weeks.

Hospitals and health care providers across the United States told patients on Friday morning that a global technology outage had downed some information technology systems, resulting in canceled surgeries and other procedures, though the hospitals emphasized that emergency departments remained open.

Some major hospital systems were affected, including the Kaiser Permanente medical system, which runs dozens of hospitals and hundreds of medical offices in the western United States and elsewhere in the country. Kaiser Permanente activated its national command center around 7:30 a.m. Eastern to address “widespread” effects of the outage on its system, said Steve Shivinsky, a spokesman for the health provider.

Jonathan Wolfe

July 19, 2024, 2:36 p.m. ET4 hours ago

Adding to the headaches for travelers, U.S. Customs and Border Protection said it was also experiencing processing delays because of the outage, warning of "longer than normal wait times" at ports of entry.

An outage that crippled businesses around the world turned many Microsoft computers into bricks overnight.

That means a lot of people are seeing the blue screen of death as computers continually try to reboot. The issues stemmed from a faulty software update delivered to Microsoft devices and servers from CrowdStrike, a cybersecurity company that mainly works to help protect bigger companies from attacks.

Court systems across the country were severely affected by the global outage, with some open only for emergencies and others closing their doors altogether.

In New York City, one particularly notable case was affected: A hearing in the sex crimes case of Harvey Weinstein, the disgraced Hollywood mogul, was delayed nearly two hours.

Ella Koeze

July 19, 2024, 2:11 p.m. ET4 hours ago

Airlines have started to resume flights after thousands were canceled earlier on Friday. At points, the cancelations amounted to nearly 8 percent of all scheduled departures around the world.

How the airline cancelations rippled around the world (and across time zones)

Canceled share of departing flights at 25 airports

Bengaluru Kempegowda

Minneapolis-Saint Paul

Berlin Brandenburg

Amsterdam Schiphol

Aaron Krolik

July 19, 2024, 2:00 p.m. ET4 hours ago

This was not the first CrowdStrike outage in recent months. On April 19, CrowdStrike released an update to customers running their Linux product that similarly crashed their systems. The April bug, which appears to be unrelated to today’s, took the CrowdStrike team nearly five days to fix.

Aaron Krolik

July 19, 2024, 2:01 p.m. ET4 hours ago

In a report sent to affected customers, the company attributed the issue to “human error.” CrowdStrike promised to improve its testing, and to “remove the potential for human error.”

Kate Conger

July 19, 2024, 1:56 p.m. ET4 hours ago

The widespread outage raised concerns about how many corporations and governments rely on just a handful of cybersecurity companies to protect them. “When it comes to cybersecurity, we talk about defense in depth — having a moat and then archers and a gate around the castle,” said Matt Mitchell, a hacker and founder of CryptoHarlem, a cybersecurity education and advocacy organization. But, he added, "We are creating a situation where there is a single point of failure.”

Eli Tan

July 19, 2024, 1:48 p.m. ET5 hours ago

The Port of Houston and the Port of Long Beach said they had fully resumed shipping operations after experiencing outages overnight and this morning. Some of the country’s other large ports – Los Angeles, Savannah, New York and New Jersey – have not seen any disruptions.

Jill Cowan

July 19, 2024, 1:35 p.m. ET5 hours ago

Portland’s mayor, Ted Wheeler, declared a city emergency early on Friday morning after some of the city’s computer systems, including ones used in 911 dispatch operations, crashed as a result of the CrowdStrike outages. City officials emphasized that emergency dispatch services went back to normal by about 6 a.m. on Friday, and city services continued throughout the morning, but residents should expect delays in getting non-emergency calls answered.

Jonathan Wolfe

July 19, 2024, 1:09 p.m. ET5 hours ago

The outage affected “all of our hospitals” said Steve Shivinsky, a spokesman for Kaiser Permanente medical system, who called the situation “unprecedented.” “In some situations, we have activated backup systems to support both continuous patient care and to secure access to medical records,” he said. Kaiser's system runs 40 hospitals, more than 600 medical offices and provides care to 12.6 million members across the country.

Kate Conger

July 19, 2024, 12:56 p.m. ET5 hours ago

While little-known outside the security industry, CrowdStrike has built a sterling reputation as the first line of defense against complex cyberattacks. The company, which is worth about $76 billion, investigated the 2014 hack of Sony Pictures and the hack-and-leak operation of the D.N.C. in 2016 that led to the exposure of Hillary Clinton’s emails.

The global technology outage has put a spotlight on how Microsoft, the second-most valuable publicly traded company, is a major provider of software and cloud services for businesses and governments globally.

The tech behemoth, based in Redmond, Wa., has a $3.25 trillion market capitalization, more than the economic output of the United Kingdom, France or Mexico measured in gross domestic product.

Kate Conger

July 19, 2024, 12:42 p.m. ET6 hours ago

Some tech titans indicated that the outage was enough to make them stop using CrowdStrike. “We just deleted CrowdStrike from all our systems,” Elon Musk, the chief executive of Tesla and SpaceX, wrote on X.

David McCabe

July 19, 2024, 12:36 p.m. ET6 hours ago

“Yesterday, CrowdStrike released an update that began impacting IT systems globally,” Satya Nadella, Microsoft’s chief executive, said on X. “We are aware of this issue and are working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”

Some delegates at the Republican National Convention found themselves stuck at Milwaukee Mitchell International Airport, running into delays and cancellations resulting from the global technology outage as they tried to head home.

Michael Holderness, a delegate from Sarasota, Fla., was considering driving 20 hours rather than being stuck at the airport. He recalled spending two days in December 2022 in Nashville after Southwest Airlines experienced computer problems. This time, he said, he did not want to wait for another flight after spending four days on the convention floor.

Isabella Kwai

July 19, 2024, 12:16 p.m. ETJuly 19, 2024

Banner Health, a large healthcare system based in Phoenix, that operates hospitals and healthcare facilities across six states, said that it had been impacted by the I.T. outage. It closed clinics, urgent care and other outpatient facilities on Friday but said that hospitals would remain open for inpatient care and medical emergencies.

Heather Knight

July 19, 2024, 12:13 p.m. ETJuly 19, 2024

U.S. public transit systems seem to be weathering the outage OK, with some delays and signal problems but nowhere near the chaos at airports. Maybe their old-school nature helped. Erica Kato, a spokeswoman for the San Francisco Municipal Transportation System, which runs the city's buses, trains and iconic cable cars, said the agency's main systems are not even connected to the internet. "Long live floppy disks, I guess!" she said.

Jill Cowan

July 19, 2024, 12:10 p.m. ETJuly 19, 2024

The Los Angeles County Superior Court system, one of the largest in the nation, was experiencing “system-wide connectivity issues,” according to a statement. Any business where people were set to appear remotely on Friday would be rescheduled, it said.

Emergency call systems in at least three states were hit by the widespread CrowdStrike technology outage, although most major U.S. cities avoided problems overnight.

In Oregon, some 911 centers, hospitals, airports, and public safety and emergency management agencies had major effects, said Erin Zysett, a spokeswoman for the Oregon Department of Emergency Management.

Mark Walker

July 19, 2024, 11:50 a.m. ETJuly 19, 2024

Airlines began the day by saying they didn’t owe stranded passengers anything more than rebooked flights with waived fees. But the Transportation Department said that the software outage is considered within the airlines’ control. This means they must follow through on their commitments to travelers, which fluctuate but could include paying for hotel rooms, ground transportation and meal vouchers.

Heather Knight

July 19, 2024, 11:42 a.m. ETJuly 19, 2024

The outage led to a very busy 16 minutes for the San Francisco Fire Department. Between 2:34 and 2:50 a.m., firefighters responded to 20 fire alarms in buildings around the city. The alarms were all false and all apparently caused by the outage, a fire department spokesperson said.

Rob Copeland

July 19, 2024, 11:41 a.m. ETJuly 19, 2024

TD Bank, the 10th largest in the U.S., confirmed it had been impacted by the outage. The bank’s customers have filled social media with complaints that they cannot access their online accounts. “Teams are working hard to restore all online banking and other impacted systems,” a spokeswoman said in a statement.

Sign up for the Travel Dispatch newsletter. Essential news on the changing travel landscape, expert tips and inspiration for your future trips.

While commercial air service was slowly recovering on Friday after a technology outage caused thousands of flights worldwide to be canceled or delayed, the ripple effect from the disruption left airports crowded with passengers and airlines working to get planes and crews back in position.

Financial transactions around the world were affected by a short-lived tech outage on Friday, hampering operations as workers struggled to log into their corporate systems.

Employees at companies including JPMorgan Chase and Instinet, a brokerage firm owned by the Japanese bank Nomura, had trouble gaining access to their work stations, according to people with knowledge of the matter who spoke on condition of anonymity. That led to delays in some trades as the companies rushed to find workarounds, the people said. By midday in New York, trading seemed to be functioning normally.


Tech Crunch

https://techcrunch.com/2024/07/19/what-we-know-about-crowdstrikes-update-fail-thats-causing-global-outages-and-travel-chaos/

What we know about CrowdStrike’s update fail that’s causing global outages and travel chaos

A faulty software update issued by security giant CrowdStrike has resulted in a massive overnight outage that’s affected Windows computers around the world, disrupting businesses, airports, train stations, banks, broadcasters and the healthcare sector.

CrowdStrike said the outage was not caused by a cyberattack, but was the result of a “defect” in a software update for its flagship security product, Falcon Sensor. The defect caused any Windows computers that Falcon is installed on to crash without fully loading.

“The issue has been identified, isolated and a fix has been deployed,” said CrowdStrike in a statement on Friday. Some businesses and organizations are beginning to recover, but many expect the outages to drag on into the weekend or next week given the complexity of the fix. CrowdStrike CEO George Kurtz told NBC News that it may take “some time for some systems that just automatically won’t recover.” In a later tweet, Kurtz apologized for the disruption.

Here’s everything you need to know about the outages.

What happened?

Late Thursday into Friday, reports began to emerge of IT problems wherein Windows computers were getting stuck with the infamous “blue screen of death” — a bright blue error screen with a message that displays when Windows encounters a critical failure, crashes or cannot load.

The outages were first noticed in Australia early on Friday, and reports quickly came in from the rest of Asia and Europe as the regions began their day, as well as the United States.

Within a short time, CrowdStrike confirmed that a software update for Falcon had malfunctioned and was causing Windows computers that had the software installed to crash. Falcon lets CrowdStrike remotely analyze and check for malicious threats and malware on installed computers.

At around the same time, Microsoft reported a significant outage at one of its most used Azure cloud regions covering much of the central United States. A spokesperson for Microsoft told TechCrunch that its outage was unrelated to CrowdStrike’s incident.

Around Friday noon (Eastern time), Microsoft CEO Satya Nadella posted on X saying the company is aware of the CrowdStrike botched update and is “working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”

What is CrowdStrike and what does Falcon Sensor do?

CrowdStrike, founded in 2011, has quickly grown into a cybersecurity giant. Today the company provides software and services to 29,000 corporate customers, including around half of Fortune 500 companies, 43 out of 50 U.S. states and eight out of the top 10 tech firms, according to its website.

The company’s cybersecurity software, Falcon, is used by enterprises to manage security on millions of computers around the world. These businesses include large corporations, hospitals, transportation hubs and government departments. Most consumer devices do not run Falcon and are unaffected by this outage.

One of the company’s biggest recent claims to fame was when it caught a group of Russian government hackers breaking into the Democratic National Committee ahead of the 2016 U.S. presidential election. CrowdStrike is also known for using memorable animal-themed names for the hacking groups it tracks based on their nationality, such as: Fancy Bear, believed to be part of Russia’s General Staff Main Intelligence Directorate, or GRU; Cozy Bear, believed to be part of Russia’s Foreign Intelligence Service, or SVR; Gothic Panda, believed to be a Chinese government group; and Charming Kitten, believed to be an Iranian state-backed group. The company even makes action figures to represent these groups, which it sells as swag.

CrowdStrike is so big it’s one of the sponsors of the Mercedes F1 team, and this year even aired a Super Bowl ad — a first for a cybersecurity company.

Who are the outages affecting?

Practically anyone who during their everyday life interacts with a computer system running software from CrowdStrike is affected, even if the computer isn’t theirs.

These devices include the cash registers at grocery stories, departure boards at airports and train stations, school computers, your work-issued laptops and desktops, airport check-in systems, airlines’ own ticketing and scheduling platforms, healthcare networks and many more. Because CrowdStrike’s software is so ubiquitous, the outages are causing chaos around the world in a variety of ways. A single affected Windows computer in a fleet of systems could be enough to disrupt the network.

TechCrunch reporters around the world are seeing and experiencing outages, including at points of travel, doctors’ offices and online. Early on Friday, the Federal Aviation Administration put in effect a ground stop, effectively grounding flights across the United States, citing the disruption. It looks like so far the national Amtrak rail network is functioning as normal.

What is the U.S. government doing so far?

Given that the problem stems from a company, there isn’t much that the U.S. federal government can do. According to a pool report, President Biden was briefed on the CrowdStrike outage, and “his team is in touch with CrowdStrike and impacted entities.” That’s in large part because the federal government is a customer of CrowdStrike and also affected.

Several federal agencies are affected by the incident, including the Department of Education, and Social Security Administration, which said Friday that it closed its offices as a result of the outage.

The pool report said Biden’s team is “engaged across the interagency to get sector by sector updates throughout the day and is standing by to provide assistance as needed.”

In a separate tweet, Homeland Security said it was working with its U.S. cybersecurity agency CISA, CrowdStirke and Microsoft — as well as its federal, state, local and critical infrastructure partners — to “fully assess and address system outages.”

There will no doubt be questions for CrowdStrike (and to some extent Microsoft, whose unrelated outage also caused disruption overnight for its customers) from government and congressional investigators.

For now, the immediate focus will be on the recovery of affected systems.

How do affected customers fix their Windows computers?

The major problem here is that CrowdStrike’s Falcon Sensor software malfunctioned, causing Windows machines to crash, and there’s no easy way to fix that.

So far, CrowdStrike has issued a patch, and it has also detailed a workaround that could help affected systems function normally until it has a permanent solution. One option is for users to “reboot the [affected computer] to give it an opportunity to download the reverted channel file,” referring to the fixed file.

In a message to users, CrowdStrike detailed a few steps customers can take, one of which requires physical access to an affected system to remove the defective file. CrowdStrike says users should boot the computer into Safe Mode or Windows Recovery Environment, navigate to the CrowdStrike directory, and delete the faulty file “C-00000291*.sys.”

The wider problem with having to fix the file manually could be a major headache for companies and organizations with large numbers of computers, or Windows-powered servers in datacenters or locations that might be in another region, or an entirely different country.

CISA warns that malicious actors are ‘taking advantage’ of the outage

In a statement on Friday, CISA attributed the outages to the faulty CrowdStrike update and that the issue was not due to a cyberattack. CISA said that it was “working closely with CrowdStrike and federal, state, local, tribal and territorial partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts.”

CISA did note, however, that it has “observed threat actors taking advantage of this incident for phishing and other malicious activity.” The cybersecurity agency did not provide more specifics, but warned organizations to stay vigilant.

Malicious actors can and will exploit confusion and chaos to carry out cyberattacks on their own. Rachel Tobac, a social engineering expert and founder of cybersecurity firm SocialProof Security, said in a series of posts on X to “verify people are who they say they are before taking sensitive actions.”

“Criminals will attempt to use this IT outage to pretend to be IT to you or you to IT to steal access, passwords, codes, etc.,” said Tobac.

What do we know about misinformation so far?

It’s easy to understand why some might have thought that this outage was a cyberattack. Sudden outages, blue screens at airports, office computers filled with error messages, and chaos and confusion. As you might expect, a fair amount of misinformation is already flying around, even as social media sites incorrectly flag trending topics like “cyberattack.”

Remember to check official sources of news and information, and if something seems too good to be true, it might just well be.

TechCrunch’s Ram Iyer contributed reporting.


CNN

https://edition.cnn.com/2024/07/19/tech/crowdstrike-update-global-outage-explainer/index.html

What is CrowdStrike, the company linked to the global outage?

Sean Lyngaas

The global computer outage affecting airports, banks and other businesses on Friday appears to stem at least partly from a software update issued by major US cybersecurity firm CrowdStrike, experts told CNN.

CrowdStrike told customers early Friday that the outages were caused by “a defect found in a single content update of its software on Microsoft Windows operating systems, according to a post on X from CEO George Kurtz.

Kurtz later apologized to customers Friday, and said the company is “deeply sorry for the inconvenience and disruption,” he posted on X.

He reiterated that the outage was not caused by a security breach or a cyberattack and maintained that CrowdStrike’s customers were “fully protected.”

“We are working with all impacted customers to ensure that systems are back up and they can deliver the services their customers are counting on,” Kurtz said, reiterating that the cause of the outage was not malicious.

The company’s engineers took action to address the problem, according to an advisory viewed by CNN, which told customers to reboot their computers and perform other actions if they were still having technical issues. The issue is specific to Falcon, one of CrowdStrike’s main software products, and is not impacting Mac or Linux operating systems, according to the advisory. Crowdstrike says Falcon is designed to protect files saved in the cloud.

CrowdStrike’s cybersecurity software — used by numerous Fortune 500 companies, including major global banks, healthcare and energy companies — detects and blocks hacking threats. Like other cybersecurity products, the software requires deep-level access to a computer’s operating system to scan for those threats. In this case, computers running Microsoft Windows appear to be crashing because of the faulty way a software code update issued by CrowdStrike is interacting with the Windows system.

The company said the outage was not caused by a security incident or a cyberattack. Kurtz, in his post, said the issue was identified and isolated, and engineers deployed an update to fix the problem.

CrowdStrike’s (CRWD) stock fell 9% in midafternoon trading.

What is CrowdStrike known for?

The massive cybersecurity company does business around the world through software sales and investigations of major hacks.

The company also helps run cybersecurity investigations for the US government. For example, CrowdStrike has tracked North Korean hackers for more than a decade, the company says. It also was tasked with tracking the hacking groups that carried out the 2014 hack on Sony Pictures.

But CrowdStrike is perhaps best known for investigating the Russian hack of Democratic National Committee computers during the 2016 US election. It has been at the center of false conspiracy theories since 2016, most notably after a White House transcript revealed former President Donald Trump mentioned Crowdstrike in his July 2016 call with Ukrainian President Volodymyr Zelensky that led to his first impeachment.

CrowdStrike was the first to publicly sound the alarm about Russia’s interference in the 2016 election and CrowdStrike’s assessment was later confirmed by US intelligence agencies.

This story has been updated with additional context

CNN’s Donie O’Sullivan contributed to this report


Ars Technica

https://arstechnica.com/information-technology/2024/07/major-outages-at-crowdstrike-microsoft-leave-the-world-with-bsods-and-confusion/

Major outages at CrowdStrike, Microsoft leave the world with BSODs and confusion

Nobody's sure who's at fault for each outage: Microsoft, CrowdStrike, or both.

Kevin Purdy - 7/19/2024, 9:22 AM

A passenger sits on the floor as long queues form at the check-in counters at Ninoy Aquino International Airport, on July 19, 2024 in Manila, Philippines.

Enlarge / A passenger sits on the floor as long queues form at the check-in counters at Ninoy Aquino International Airport, on July 19, 2024 in Manila, Philippines.

Ezra Acayan/Getty Images

Millions of people outside the IT industry are learning what CrowdStrike is today, and that's a real bad thing. Meanwhile, Microsoft is also catching blame for global network outages, and between the two, it's unclear as of Friday morning just who caused what.

After cybersecurity firm CrowdStrike shipped an update to its Falcon Sensor software that protects mission-critical systems, blue screens of death (BSODs) started taking down Windows-based systems. The problems started in Australia and followed the dateline from there.

TV networks, 911 call centers, and even the Paris Olympics were affected. Banks and financial systems in India, South Africa, Thailand, and other countries fell as computers suddenly crashed. Some individual workers discovered that their work-issued laptops were booting to blue screens on Friday morning. The outages took down not only Starbucks mobile ordering, but also a single motel in Laramie, Wyoming.

Airlines, never the most agile of networks, were particularly hard-hit, with American Airlines, United, Delta, and Frontier among the US airlines overwhelmed Friday morning.

CrowdStrike CEO "deeply sorry"

Fixes suggested by both CrowdStrike and Microsoft for endlessly crashing Windows systems range from "reboot it up to 15 times" to individual driver deletions within detached virtual OS disks. The presence of BitLocker drive encryption on affected devices further complicates matters.

CrowdStrike CEO George Kurtz posted on X (formerly Twitter) at 5:45 am Eastern time that the firm was working on "a defect found in a single content update for Windows hosts," with Mac and Linux hosts unaffected. "This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed," Kurtz wrote. Kurtz told NBC's Today Show Friday morning that CrowdStrike is "deeply sorry for the impact that we’ve caused to customers."

As noted on Mastodon by LittleAlex, Kurtz was the Chief Technology Officer of security firm McAfee when, in April 2010, that firm sent an update that deleted a crucial Windows XP file that caused widespread outages and required system-by-system file repair.

The costs of such an outage will take some time to be known, and will be hard to measure. Cloud cost analyst CloudZero estimated mid-morning Friday that the CrowdStrike incident had already cost $24 billion, based on a previous estimate.

Multiple outages, unclear blame

Microsoft services were, in a seemingly terrible coincidence, also down overnight Thursday into Friday. Multiple Azure services went down Thursday evening, with the cause cited as "a backend cluster management workflow [that] deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region."

A spokesperson for Microsoft told Ars in a statement Friday that the CrowdStrike update was not related to its July 18 Azure outage. "That issue has fully recovered," the statement read.

News reporting on these outages has so far blamed either Microsoft, CrowdStrike, or an unclear mixture of the two as the responsible party for various outages. It may be unavoidable, given that the outages are all happening on one platform, Windows. Microsoft itself issued an "Awareness" regarding the CrowdStrike BSOD issue on virtual machines running Windows. The firm was frequently updating it Friday, with a fix that may or may not surprise IT veterans.

"We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage," Microsoft wrote in the bulletin. Alternately, Microsoft recommend customers that have a backup from "before 19:00 UTC on the 18th of July" restore it, or attach the OS disk to a repair VM to then delete the file (Windows/System32/Drivers/CrowdStrike/C00000291*.sys) at the heart of the boot loop.

Security consultant Troy Hunt was quoted as describing the dual failures as "the largest IT outage in history," saying, "basically what we were all worried about with Y2K, except it's actually happened this time."

United Airlines told Ars that it was "resuming some flights, but expect schedule disruptions to continue throughout Friday," and had issued waivers for customers to change travel plans. American Airlines posted early Friday that it had re-established its operations by 5 am Eastern, but expected delays and cancellations throughout Friday.

Ars has reached out to CrowdStrike for comment and will update this post with response.

This is a developing story and this post will be updated as new information is available.


NBC

https://www.nbcnews.com/tech/tech-news/microsoft-outage-crowdstrike-global-airlines-windows-fix-rcna162685

What we know about the global Microsoft outage

A massive outage was caused by what was supposed to be a routine update from the cybersecurity company CrowdStrike.

July 19, 2024, 9:40 AM EDT / Updated July 19, 2024, 3:09 PM EDT

A routine software update caused cascading chaos Friday that has engulfed global businesses from airports and banks to retail and law enforcement.

It wasn’t a cyberattack, but was caused by one of the world’s largest cybersecurity companies deploying a flawed update while trying to keep their customers safe from hackers.

CrowdStrike, a cybersecurity company based in Austin, Texas, that’s widely used by businesses and government agencies that run on Microsoft computers, said that a defect in one of its updates for computers running the Windows operating system had caused the issue. The result was arguably the largest global information technology outage in history.

Here’s what we know:

What has been affected?

The outage appeared to affect almost every major business sector in one way or another.

Airports and flights across the world were severely delayed or canceled altogether, as the computers on which these services rely were disrupted by the issue. Three of the largest airlines in the United States — American Airlines, Delta Air Lines and United Airlines — all issued ground stops, citing communications problems.

Long lines snaked through airports from Berlin to Hong Kong, as electronic systems failed and staff members resorted to checking in passengers manually. One flier in the Indian city of Hyderabad posted a picture of a curiosity usually unseen in the modern digital age: a handwritten boarding pass.

Follow live updates on the global tech outage

Mass IT Outage Affects Travel, Businesses And Individual Users Across The Globe

Travelers wait at a check-in counter at Berlin Airport this morning.Sean Gallup / Getty Images

The issue also affected banks, leaving some customers unable to access their money. People across Australia, New Zealand and elsewhere reported problems logging into their accounts at major retail banks. Meanwhile the London Stock Exchange, the largest of its kind in Europe, said some of its services had been disrupted although trading had not been impeded.

In retail, McDonald’s closed some of its stores in Japan because of what it said in an online statement was a “cash register malfunction.” And the British grocery chain Waitrose was forced to put up handwritten notes informing customers that it was only accepting cash. Some Starbucks locations temporarily closed after the company's mobile ordering system went down, while other locations stayed open and left their baristas scrambling to come up with new workflows.

Baltic Hub, a global shipping terminal in Poland, said in a statement it had been battling issues related to the outage. The Alaska State Troopers agency was among the law enforcement agencies to report issues, warning people that 911 was temporarily not working. And the British broadcaster Sky News, which is owned by NBCUniversal's parent company, Comcast, was briefly knocked off the air.

Global Microsoft Outage

Blue screens on a flight board at Chicago's O'Hare International Airport on Friday morning.Maura Barrett / NBC News

Some computers that were not on during the software update appear to have avoided the issue, according to initial reports.

What caused it?

Cybersecurity programs like CrowdStrike’s frequently and automatically update themselves to account for new tactics that malicious hackers have discovered. And there’s always a slight risk that any software update will be incompatible with other programs.

CrowdStrike’s update should have been routine, but an error in its code conflicted with Windows and proved catastrophic.

The company’s CEO, George Kurtz, said Friday morning on the "TODAY" show that while some customers would see their computers fixed automatically, others would require manual work to fix.

“The system was sent an update, and that update had a software bug in it and caused an issue with the Microsoft operating system,” Kurtz said. “Our systems are always looking for the latest attacks from these adversaries that are out there.”

How long until things go back to normal?

CrowdStrike identified the problem and issued a fix early Friday after the problem began to emerge.

The fix required affected computers to download another software update, which some have been able to do automatically.

Many companies and services affected by the chaos, such as the New Hampshire Department of Safety, which said its 911 system was briefly down, reported that they were back online by Friday morning.

Others haven’t been able to load the update, which would potentially require IT workers at some companies to reboot and tinker with every single affected computer.

“Many of the customers are rebooting the system, and it’s coming up, and it’ll be operational because we fixed it on our end,” Kurtz said. “Some of the systems that aren’t recovering, we’re working with them. So it could be some time for some systems that just automatically won’t recover.”

Alexander Smith is a senior reporter for NBC News Digital based in London.

Kevin Collier is a reporter covering cybersecurity, privacy and technology policy for NBC News.


Reuters

July 19, 2024, 9:22 AM EDT

Don Mueang International Airport Terminal 1 amid system outages disrupting the airline's operations in Bangkok

Air Asia passengers queue at counters inside Don Mueang International Airport Terminal 1 amid system outages disrupting the airline's operations, in Bangkok, Thailand, July 19, 2024. REUTERS/Chalinee Thirasupa Purchase Licensing Rights

Cybersecurity firm CrowdStrike (CRWD.O) has deployed a fix for an issue that triggered a major tech outage that affected industries ranging from airlines to banking to healthcare worldwide, the company's CEO said on Friday.

Microsoft (MSFT.O) said separately it had fixed the underlying cause for the outage of its 365 apps and services including Teams and OneDrive, but residual impact was affecting some services.

"This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed," CrowdStrike CEO George Kurtz said in a post on social media platform X.

The issue stemmed from a defect found in a single content update for Microsoft Windows hosts, Kurtz said, adding Mac and Linux hosts were not impacted by the issue.

Shares of CrowdStrike tumbled nearly 12% in premarket trading, while Microsoft was down 1.4%.

A massive IT outage was disrupting operations at companies across multiple industries on Friday, with major airlines halting flights, some broadcasters off-air and sectors ranging from banking to healthcare hit by system problems.

"We're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our company," Kurtz told NBC News' "Today" program.

"Many of the customers are rebooting the system and it's coming up and it'll be operational," Kurtz said. "It could be some time for some systems that won't automatically recover."

CrowdStrike's "Falcon Sensor" software was causing Microsoft Windows to crash and display a blue screen, known informally as the "Blue Screen of Death," according to an alert sent by CrowdStrike earlier to its clients and reviewed by Reuters.

The travel industry was among the hardest hit with airports around the world reporting delays and issues with their system network, while banks and financial institutions from Australia and India to South Africa warned clients about disruptions to their services.

The Technology Roundup newsletter brings the latest news and trends straight to your inbox. Sign up here.

Reporting by Deborah Sophia in Bengaluru; Editing by Krishna Chandra Eluri and Sriraj Kalluvila


Wired

https://www.wired.com/story/microsoft-windows-outage-crowdstrike-global-it-probems/

Huge Microsoft Outage Caused by CrowdStrike Takes Down Computers Around the World

A software update from cybersecurity company CrowdStrike inadvertently disrupted IT systems globally—taking Windows machines offline.

Matt BurgessJul 19, 2024 4:40 AM

Image may contain Logo and Terminal

PHOTO-ILLUSTRATION: WIRED STAFF; GETTY IMAGES

Banks, airports, TV stations, health care organizations, hotels, and countless other businesses are all facing widespread IT outages, leaving flights grounded and causing widespread disruption, after Windows machines have displayed errors worldwide.

In the early hours of Friday, companies in Australia running Microsoft’s Windows operating system started reporting devices showing Blue Screens of Death (BSODs). Shortly after, reports of disruptions started flooding in from around the world, including from the UK, India, Germany, the Netherlands, and the US: TV station Sky News went offline, and US airlines United, Delta, and American Airlines issued a “global ground stop” on all flights.

The widespread Windows outages have been linked to a software update from cybersecurity giant ​​CrowdStrike. It is believed the issues are not linked to a malicious cyberattack, cybersecurity officials say, but rather stem from a misconfigured/corrupted update that CrowdStrike pushed out to its customers.

“Earlier today, a CrowdStrike update was responsible for bringing down a number of IT systems globally," said a Microsoft spokesperson in a statement. "We are actively supporting customers to assist in their recovery.”

Engineers from CrowdStrike posted to the company’s Reddit forum that it has seen “widespread reports of BSODs on Windows hosts” occurring across its software, is working on the problem, and has advised a workaround for impacted systems. It also issued instructions to its customers in an advisory.

The incident has only impacted devices running Windows and not other operating systems. It's unclear exactly how widespread the issues are and how long they will take to resolve.

Hours after the issues started to emerge, CrowdStrike CEO George Kurtz issued a statement about the outages, saying the company has found a “defect” in an update for Windows that it issued. “This is not a security incident or cyberattack,” Kurtz said. “The issue has been identified, isolated, and a fix has been deployed.” In the statement, Kurtz confirmed that Mac and Linux hosts are not impacted by the update and said that its customers should refer to its support portal. He later apologized for the incident during a television interview.

At the same time as the CrowdStrike issues emerged, Microsoft was also dealing with its own, apparently unrelated, outage of its Azure cloud services. The company says the two incidents are not linked.

The widespread Windows outages have been linked to a software update from cybersecurity giant ​​CrowdStrike. It is believed the issues are not linked to a malicious cyberattack, cybersecurity officials say, but rather stem from a misconfigured/corrupted update that CrowdStrike pushed out to its customers.

The outages could result in “millions” being lost by organizations impacted who have had to halt their operations or stop business, says Lukasz Olejnik, an independent cybersecurity consultant, who says the CrowdStrike update appears to be linked to its Falcon Sensor product. The Falcon system is part of CrowdStrike’s security tools and can block attacks on systems, according to the company.

“It reminds us about our dependence on IT and software,” Olejnik says. “When a system has several software systems maintained by various vendors, this is equivalent to placing trust on them. They may be a single point of failure—like here, when various firms feel the impact.”

The outage stemming from the CrowdStrike update has had a huge knock-on impact on public services and businesses around the world. Scores of airports are facing delays and long queues, with one passenger in India sharing a hand-written boarding pass that they have been issued. In the hours after the outages first emerged, more than 4,000 flights around the world have been canceled, although not all of them may have been directly linked to the disruption.

Within health care and emergency services, various medical providers around the world have reported issues with their Windows-linked systems, sharing news on social media or their own websites. The US Emergency Alert System, which issues hurricane warnings, said that there had been various 911 outages in a number of states. In Portland, mayor Ted Wheeler declared a city emergency as a result of some of the outages, although also said many systems were being restored. White House officials say president Joe Biden has been "briefed" on the CrowdStrike outages and his team is monitoring the situation.

Germany’s University Hospital Schleswig-Holstein said it was canceling some nonurgent surgeries at two locations. In Israel, more than a dozen hospitals have been impacted, as well as pharmacies, with reports saying ambulances have been rerouted to unimpacted medical organizations.

In the UK, NHS England has confirmed that GP appointment and patient record systems have been affected by the outages. One hospital has declared a “critical” incident after a third-party IT system it used was impacted. Also in the country, train operators have said there are delays across the network, with multiple companies being impacted.

Indicating the far-reaching nature of the disruption, the organizers of the Paris Olympics, which is due to start next week, said that its systems have been impacted in a “limited way.” According to a statement from the organizers, the affected systems are linked to the delivery of uniforms and its ticketing system hasn’t been impacted.

Among other services, CrowdStrike provides endpoint detection and response (EDR) to companies around the world. This EDR technology runs on thousands of “endpoints”—such as computers, ATMs, and internet-of-things devices—and scans them to identify real-time threats, such as malicious activity from cybercriminals. The company has more than 24,000 customers around the world.

Cybersecurity researcher Kevin Beaumont posted on X that he has seen a copy of the CrowdStrike update that was issued and says the file isn’t properly formatted and “causes Windows to crash every time.” Beaumont says, in further posts, that it appears there isn’t an automated way to fix the issues, at least currently. This may mean that impacted machines need to be manually rebooted before they can come back online, a process that could take hours or days depending on the impacted entity.

Brody Nisbet, the director of overwatch at CrowdStrike, also posted on X indicating that the workaround fix the company had issued involves booting up Windows machines into safe mode, finding a file called “C-00000291*.sys,” deleting it, and then rebooting the machine normally. “There is a fix of sorts so some devices in between BSODs should pick up the new channel file and remain stable,” Nisbet posted.

Update 7/19/24 1:35pm ET: This story has been updated with further comment from Microsoft, and additional details about the outage's impacts.


CNBC

https://www.cnbc.com/2024/07/19/what-is-crowdstrike-crwd-and-how-did-it-cause-global-it-outages.html

How a software update from cyber firm CrowdStrike caused one of the world’s biggest IT blackouts

Ryan Browne

George Kurtz, co-founder and CEO of CrowdStrike Inc., speaks during the Montgomery Summit in Santa Monica, California.

George Kurtz, co-founder and CEO of CrowdStrike Inc., speaks during the Montgomery Summit in Santa Monica, California.

Patrick T. Fallon | Bloomberg | Getty Images

A fault with an update issued by cybersecurity company CrowdStrike led to a cascade effect among global IT systems Friday, with industries ranging from banking to airlines facing outages.

Banks and health-care providers saw their services disrupted and TV broadcasters went offline as businesses worldwide grappled with the ongoing outage. Air travel has been hit hard, too, with planes grounded and services delayed.

At the heart of the issue is Texas-based cybersecurity vendor CrowdStrike. On Friday, the cybersecurity firm experienced a major disruption following an issue with a software update.

So what happened, exactly? CNBC takes a look.

What is CrowdStrike and what does it do?

CrowdStrike is a cybersecurity vendor that develops software to help companies detect and block hacks. It is used by many of the world’s Fortune 500 companies, including major global banks, health-care and energy companies.

Major technical outages worldwide: Here's what to know

CrowdStrike is what’s known as an “endpoint security” firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet.

This differs from alternative approaches used by other cyber firms, which involve applying protection directly to back-end server systems.

“Many companies use [CrowdStrike software] and install it on all of their machines across their organization,” Nick France, chief technology officer at IT security firm Sectigo, told CNBC’s “Squawk Box Europe” on Friday.

“So when an update happens that maybe has problems with it, it causes this problem where the machines reboot, and people can’t get back into their computers.”

What happened on Friday?

On Friday, people around the world began encountering an error screen known as the “blue screen of death.”

This issue — a common problem among PCs, for example if a machine overheats — was the result of an update from CrowdStrike concerning its Falcon product.

Falcon is a platform developed by the company that’s designed to stop cyber breaches using cloud technology — it is at the heart of the firm’s focus on endpoints. CrowdStrike said Friday it is in the process of rolling back the update globally.

CrowdStrike’s software requires deep access to a computer’s operating system to scan for threats. In the case of Friday’s outage, machines running Microsoft’s Windows operating system crashed due to a fault in the way a software update issued by CrowdStrike interacted with Windows.

“We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD [blue screen of death]) and get stuck in a restarting state. We approximate impact started around 19:00 UTC on the 18th of July,” Microsoft said in an update at 5:40 a.m. ET.

“We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance,” the company added.

Satnam Narang, senior staff researcher at Tenable, told CNBC on Friday that the outage was “very unprecedented.”

“The challenge here is that security software — because it’s doing its job to protect organizations — it has to have more privileged access to these machines,” he said.

So, while people may be seeing their IT issues as a problem with Windows, “it’s not actually a Windows issue, it’s related to a faulty or bad update from those security software,” Narang added.

A fix has been issued

Earlier, Microsoft said its cloud services had been restored after an outage that affected its Azure services and Microsoft 365 suite of apps in the central U.S. region. A company spokesperson said these are two different and nonrelated issues — one issue relates to Azure, the other is linked to CrowdStrike.

Major global cyber outage hits airlines, banks and media outlets, impacting millions

They added that they “anticipate a resolution is forthcoming,” in respect to the CrowdStrike problem.

CrowdStrike is “actively working with customers impacted by a defect found in a single content update for Windows hosts,” CEO George Kurtz said Friday in a update on social media platform X. He added that Mac and Linux hosts are not affected.

“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed,” Kurtz said.

That fix could be hard to implement, though. Andy Grayland, chief information and security officer at threat intelligence firm Silobreaker, said that in order to implement a fix, engineers would have to go into each individual data center running windows.

They’d then have to log in, navigate to a certain CrowdStrike file, delete it and then reboot the entire system, he said.

“Where machines are encrypted, complex encryption keys also need to be entered manually. Unless Microsoft and CrowdStrike (if they are involved) pull something miraculous out of the bag, this could be painful to recover from.”

Don’t miss these insights from CNBC PRO

Buy these stocks if Trump and GOP win big in November, says Wolfe Research

Goldman says investors should start shifting out of cash. Here’s where it sees opportunity

Bitcoin could be heading for a ‘summer of 2021 style’ correction between record highs, data shows

Investors bet a Fed rate cut to spur rotation from crowded stocks to neglected as bull market continues