Same in Adelaide. Reports coming in from Gov agencies, utilities etc.

Yep happened to us too. Its global. And it just started happening.

Someone posted this in the thread, but I also can&#x27;t log in to verify&gt; Summary&gt; CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.&gt; Details&gt; Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.&gt; Current Action&gt; Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.&gt; Status updates will be posted below as we have more information to share, including when the issue is resolved.&gt; Latest Updates&gt; 2024-07-19 05:30 AM UTC | Tech Alert Published.&gt; Support&gt; Find answers and contact Support with our Support Portal

They&#x27;ve bumped this support info to a blog post that&#x27;s linked from their home page: <a href="https:&#x2F;&#x2F;www.crowdstrike.com&#x2F;blog&#x2F;statement-on-falcon-content-update-for-windows-hosts&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.crowdstrike.com&#x2F;blog&#x2F;statement-on-falcon-content...</a>It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That&#x27;s probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.

Can you post a summary?  We&#x27;re affected but I don&#x27;t have access to that portal.

I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.

This was particularly interesting (from the reddit thread posted above):&gt; A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have &quot;encryption at rest&quot;).&gt; His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can&#x27;t access any of the documentation to do so, because everything is down.

BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.

MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server):1. Boot the affected machine from the Windows installer disk2. Use &quot;Repair options&quot;3. Click through to the option to spawn a shell4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT.5. In the shell, type: &quot;manage-bde -unlock C: -Password&quot;, enter the password6. The drive is unlocked, now go and execute whatever recovery you have to do.Good luck.

help for if you have bitlocker turned on (because of course you do).<a href="https:&#x2F;&#x2F;x.com&#x2F;attilabubby&#x2F;status&#x2F;1814216589559861673" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;attilabubby&#x2F;status&#x2F;1814216589559861673</a><a href="https:&#x2F;&#x2F;x.com&#x2F;nathanmcnulty&#x2F;status&#x2F;1785094215749476722?s=46" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;nathanmcnulty&#x2F;status&#x2F;1785094215749476722?s=46</a>GPO to fix:<a href="https:&#x2F;&#x2F;gist.github.com&#x2F;whichbuffer&#x2F;7830c73711589dcf9e7a5217797ca617" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;whichbuffer&#x2F;7830c73711589dcf9e7a5217...</a>

Saved to my desktop? How does that help? &#x2F;s

Right after you enter the bit locker recovery key.You do have your bit locker recovery key, right? .....right?

Happy weekend to everyone who works there.

They had me at &quot;crowdstrike engineering&quot;So engineer-like.

It&#x27;s crowdstrike: <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;crowdstrike&#x2F;comments&#x2F;1e6vmkf&#x2F;bsod_error_in_latest_crowdstrike_update&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;crowdstrike&#x2F;comments&#x2F;1e6vmkf&#x2F;bsod_e...</a>&gt; 7&#x2F;18&#x2F;24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.&gt; SCOPE: EU-1, US-1, US-2 and US-GOV-1&gt; Edit 10:36PT - TA posted: <a href="https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19" rel="nofollow">https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-W...</a>&gt; Edit 11:27 PM PT:&gt; Workaround Steps:&gt; Boot Windows into Safe Mode or the Windows Recovery Environment&gt; Navigate to the C:\Windows\System32\drivers\CrowdStrike directory&gt; Locate the file matching “C-00000291*.sys”, and delete it.&gt; Boot the host normally.

I think there are companies with 10000+ or probably more

Our company is in panic mode. 15 machines blue screened for no apparent reason and stuck in boot loop. I’m a gloating Linux user :)Also in Australia

We have outages across whole APAC and most EMEA. Despite being a very big client of CS, we do not have an official resolution yet, an hour into the incident.

According to Reddit It&#x27;s hitting Croatia, Philippines, US, Germany, Mexico, India, Japan. SAP servers dropping like flies, that&#x27;s Defence,Banks, Payroll all affected. Major Retail Chains like Big W down.

Australian news reporting this has hit hospitals, fire and rescue, banking, media, airlines and many other companies worldwide.

yep spreading quickly now.<a href="https:&#x2F;&#x2F;www.geekersdigest.com&#x2F;windows-blue-screen-crash-caused-by-bad-crowdstrike-update&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.geekersdigest.com&#x2F;windows-blue-screen-crash-caus...</a>
<a href="https:&#x2F;&#x2F;www.timesnownews.com&#x2F;technology-science&#x2F;latest-crowdstrike-update-causes-blue-screen-of-death-on-microsoft-windows-multiple-users-affected-article-111854018" rel="nofollow">https:&#x2F;&#x2F;www.timesnownews.com&#x2F;technology-science&#x2F;latest-crowd...</a>This is gonna be a tough Friday for IT departments...

I was listening to Triple J (one of ABC&#x27;s radio stations), they said: &quot;welcome to our first and possibly last ever Triple J&#x27;s USB Fridays, we can&#x27;t play any of our usual music because the computers are all down, all we can play is the songs that happen to be on the USB stick that one of us had in our pocket&quot;. LOL!

Newsreaders on the ABC are reading off paper notepads lmao

The impact is Australia is immense. <a href="https:&#x2F;&#x2F;downdetector.com.au&#x2F;" rel="nofollow">https:&#x2F;&#x2F;downdetector.com.au&#x2F;</a>, it&#x27;s almost every major org in the country. NSW Government is completely offline.

Starting to get media coverage: <a href="https:&#x2F;&#x2F;www.abc.net.au&#x2F;news&#x2F;2024-07-19&#x2F;technology-shutdown-abc-media-banks-institutions&#x2F;104119960" rel="nofollow">https:&#x2F;&#x2F;www.abc.net.au&#x2F;news&#x2F;2024-07-19&#x2F;technology-shutdown-a...</a>

Its paid antivirus software, they cater to businesses

I assume you have to install &quot;CrowdStrike&quot; yourself (i.e. not bundled with Windows by default)? I have no idea what it is before.

we will found a troubleshooting , we start in revovery mode and rename the path
C:\windows\system32\drivers\Crowdstrike

Do this at your own risk!!
Apparently there are two NON-OFFICIAL solutions:1. Rename csagent.sys ( the file causing the BSOD )
2. Rename c:\windows\System32\Drivers\CrowdstrikeAgain, do this at your own risk.
Both workaround have been reported as &quot;working&quot;.
I am Linux user so I cannot tell.

Anyone found any fixes, while Crowdstrike comes up with a fix?

This seems like a pretty severe point of failure.

Husband is a deputy in California. His department and many others here are down as well (including PDs, jails, ambulance companies, etc.)

We have ~50 thousand laptops in reboot loop and ~1.5k servers as well. No resolution yet.

We had a few machines come out of the boot loop - only to re-enter it 20 mins later. I am sure CS pulled the patch from their CDNs but ...maybe some cached versions still linger?

Faced the same issue few minutes back after few loops of reboot my system is up

A system restore helps. But obviously not when you’ve got an environment of ~500 or more clients

Not likely, not unless the disk is also not BitLockered.

Could you potentially do the same by just attaching the HDD to another computer as a secondary drive and renaming the folder if safe mode falls through?

Quick fix that worked for us,
In safe mode:1.enter in drive C:
2.system 32 folder
3. Drivers
4. Rename crowdstrike folder to something else doesent matter what.

Was just using the energy vic website and thought I&#x27;d been rate limited when their API stopped working. Seems like it could be this.

Wading out my depth here, so forgive any stupidity following.And there&#x27;s a certain amount of sense to that, it has to get &quot;under&quot; the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).

This is not the case. There are many possible AV architectures, with or without kernel drivers and&#x2F;or administrator level permissions.

&gt; an application shouldn&#x27;t be able to do this kind of damage to an operating systemAntivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.

&gt; &quot;they install a kernel driver&quot;And therein lies the problem!

&gt; Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn&#x27;t be able to do this kind of damage to an operating system.It doesn&#x27;t operate in user space, they install a kernel driver.

&gt; This unchecked global rollout appears to just be a violation of every good software engineering practice we know.Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.

Administrators are to blame because management (and a lot of &#x27;cybersecurity policies&#x27;) demand there&#x27;s a virus scanner on the machines?While virus scanners might pick up some threats not addressed by OS updates yet every one of them I&#x27;ve seen is a rootkit in disguise wanting full system privileges. There are numerous incidents with security holes and crashes caused by these security products. They also aren&#x27;t that clever: repeatedly scanning the same files &#x27;on access&#x27; over and over again wasting CPU and IO is not going to give you any extra security.

As someone who worked for a company, who&#x27;s a Crowdstrike partner, I assure you that Crowdstrike does not sell to administrators. It is very much a product sold to management and company auditors.Where you&#x27;re correct is that it&#x27;s on the administrators to rollout the updates, but I&#x27;m not sure that&#x27;s how Crowdstrike works. It&#x27;s a managed solution and updates are done for you, maybe that can be disabled, but I honestly don&#x27;t know.

This should clue you in.CS is not sold to SA or technical types. It&#x27;s sold to management as a risk reduction.The whole point is that if you are technical, you are so untrusted that management is willing to require circumvention of known good practices and force installation of this software against technical advice.

At the end of the day, if you give an application a deep set of permissions, that&#x27;s on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.

I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest?Also heard today crowd stroke.

There&#x27;s a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...

I feel the need to make a mustang car crowdsrike meme

maybe they installed crowdstrike because they wanted updated without testing. and crowdstrike failed at testing them in their environment.sounds like they didnt test all cases and stumbled on a windows bug

&gt; constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else.It&#x27;d be worth giving him a call, just to check in how he&#x27;s going, and take him up on the offer to fire you as a client.

Ouch, considering the devil works under Oracles lawyers, thats bad!

&gt;I hope this sort of thing humbles them a little.Hopefully not. It would be better that this company is sued into oblivion by all the customers that were affected by this huge outage.

sounds like a very inexperienced person.If their mission is to protect businesses they should understand your concerns.Speed is useless without control.

&gt; I hope this sort of thing humbles them a little.What I hope, is that they stop to exist as a product and as a company. They have caused inconvenience, economic damage in global scale and probably also loss of life, given that many hospitals, ER units had outages. It has been proven that their whole way of working is wrong, from the very foundation to the top.

My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.

All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn&#x27;t have that despites its clear technical advantages.It is an incredibly frustrated battle akin to Sisyphus.

oh, you think security won&#x27;t mandate to run CS on linux.Granted it didn&#x27;t down linux this time but nothing is stopping it.

&gt; Linux being the most advanced operating system in the world without question.Only if you don&#x27;t need a GUI&#x2F;Desktop.

I don&#x27;t understand this sort of blindness? Linux fails all the time, with rather terrible nobody to root vulns because some idiot failed to use the right bounds check. Ye gods, XZ utils was barely a few months ago!

Crowdstrike has a linux version. It is mandatory in our linux servers in my company so that is not the solution.I would say issue 1 is management&#x2F;compliance forcing admins to install malwares like crowdstrike. But issue 1 is because of issue 2 which is about admins &#x2F; app devs &#x2F; users aren&#x27;t smart enough to not have their machines compromised on a regular basis in the first place. And issue 2 is because issue 3 of the software industry not focusing on quality and making bug free software.All in all this should be mitigated by more diversity in OS, software and &quot;said security solution&quot;. Standardization and monopolies works well until they don&#x27;t and you get this kind of shit.

&gt; Maybe it&#x27;s time that critical systems switch to Linux.I switched critical systems to illumos and BSD years ago and it&#x27;s been smooth sailing ever since. Nowadays there really is no need to contribute to linux monoculturization whatsoever.

I too want to see Linux more widely adopted, but it won&#x27;t prevent this from happening. People will install corrupted kernel modules on Linux too for anti-virus purposes.

Do Linux systems not crash if a third party kernel module crashes? Or was your comment sarcastic?

The discussed issue is not related to any meaningful difference between Windows and Linux – Crowdstrike used a kernel driver, apparently containing a serious bug, which took down the system, which is something any kernel driver can do, no matter which kernel you use. At least Windows have a well-developed framework for writing userspace drivers, unlike Linux.&gt; Linux being the most advanced operating system in the world without question.Very strong and mostly unfounded claim; there are specific aspects where Linux is &quot;more advanced&quot;, and others where Windows come out ahead (e.g. almost anything related to hardware-based security and virtualization).&gt; your system will be supported for many yearsWindows Server 2008 was supported until earlier this year, longer than any RHEL release.&gt; you won&#x27;t suffer BSoDs in 2024Until you install a shitty driver for a dubious (anti)malware service.

I suggest switching to macOS. They don&#x27;t allow third-party kernel drivers which is already a big advantage over Windows or Linux.

<a href="https:&#x2F;&#x2F;access.redhat.com&#x2F;solutions&#x2F;7068083" rel="nofollow">https:&#x2F;&#x2F;access.redhat.com&#x2F;solutions&#x2F;7068083</a>

Maybe it&#x27;s time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question.Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days.Just use Linux. You will save money and time, and your system will be supported for many years, you won&#x27;t be charged per E-Core, you won&#x27;t suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance.Really there&#x27;s no downside. If you were building your own company you would base your tech stack on Linux and not Windows.Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.

Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.

Don&#x27;t root kit yourself then cry about it when it falls over. Problem solved.

Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.

In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here?A Crowdstrike update being able to blue-screen Windows Desktops and Servers.Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn&#x27;t be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it&#x27;s a Windows bug, maybe more-so than it is a Crowdstrike bug).There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it&#x27;s put into production?Un-audited dependence on an increasingly long chain of third-parties.All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there&#x27;s safety in the crowd. Just point at &quot;them too&quot;, and stay the path. This isn&#x27;t a profitable differentiation. But it should be! (raised fists towards the sky).

Doesn&#x27;t actually look like. Noone really figured out the solution yet

If that’s the case I wonder why we’re only now seeing everyone’s systems BSODing. Should be an interesting write up (if they release one)

I&#x27;d guess that it&#x27;s related. Azure, OneDrive, and Office 365 were having issues this morning Australian time (about 6 hours ago).

Crazy- wasn’t Azure having an outage earlier today? Is this related?

This is effecting our company. A colleague visited her local supermarket (Woolworths) and all the self-service checkouts were effected.

use windows recovery to boot into safe mode and update latest crowdstrike. That is the only option at this time.

Anyone has a good news? or still it&#x27;s on the BSOD loop?

c:\system32\drivers\csagent.sys renaming this file on server with safe mode boot fixes the issue but disables agent

But at least the best instant messaging app in the world Microsoft Teams and the best web browser in the world Microsoft Edge are working fine, right?

&quot;Always look on the bright side of life!&quot; - M Python

We&#x27;re really prepared for epic to go down and have an isolated cluster that we access in emergencies. I transitioned from software engineering so I&#x27;ve only been in the ED for a year, but from what I could see there didn&#x27;t seem to be a plan for what to do if every computer in the department bluescreened at once.

Do you have offline backup processes at least? Nasty situation.

My entire emergency department got knocked offline by this. Really scary when you have ambulances coming in and are trying to stabilize a heart attack.Update: 911 is down in Oregon too, no more ambulances at least.

It depends on how global the issue is.But yeah I&#x27;ve seen US companies for example only doing their initial releases in the USA only which has zero value for issues that might appear with different localization&#x2F;language settings for example.

Canary releases aren&#x27;t a magical bug free fix. They might be doing it, but the conditions that trigger a problem can be sneaky and can happen outside your canary period, rendering it useless. It&#x27;s a best effort method.

Buying 10 computers with different configurations is not hard either. They are just lazy

Aren&#x27;t they doing canary release? Seems weird this would not have been detected on a smaller scale before with a good release process.

Not gonna be awesome for an org that has 20,000 laptops out there at home with users who don&#x27;t have admin...

what we do is safemode the PC and then open the run CMD as admin. then issue this command. sc delete csagent. reinstall crowdstrike using previous version.

If the workstations are stuck in a boot loop, how will they be able to push a hotfix out?

Uninstall the Current Version:Open the Command Prompt as an administrator.
Run the following command to uninstall the current version:
shellsc delete csagent

temporary fix is released<a href="https:&#x2F;&#x2F;x.com&#x2F;s0lst1c3&#x2F;status&#x2F;1814182835269337548" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;s0lst1c3&#x2F;status&#x2F;1814182835269337548</a>

CIO here. They are known to be incredibly pushy. In my company we RFP&#x27;d for our endpoint &amp; cyber security. Found the CS salesperson went over me to approach our CEO who is completely non-technical to try and seal a contract because I was on leave for 1 week out of service (and this was known to them). When I found out by our CEO informing me of the approach we were happy to sign with SentinelOne

Crowdstrike must have some slick salespeople in New Zealand. Seems like nearly everyone uses it. Single point of failure problem.

Banks seem fine now. At least ING and NAB.

000 was never down, and most supermarkets and servos were still up. It was bad, but ABC appear to not have the internal capacity to validate all reports.

Same in Australia 
<a href="https:&#x2F;&#x2F;www.abc.net.au&#x2F;news&#x2F;2024-07-19&#x2F;global-it-outage-crowdstrike-microsoft-banks-airlines-australia&#x2F;104119960" rel="nofollow">https:&#x2F;&#x2F;www.abc.net.au&#x2F;news&#x2F;2024-07-19&#x2F;global-it-outage-crow...</a>Banks are down so petrol stations and supermarkets are basically closed.
People can&#x27;t check in to airline flights, various government services including emergency telephone and police are down. Shows how vulnerable these systems are if there&#x27;s just one failure point taking all those down.

You mean you think the update was timed and waves will hit every hour for the next nearly 24h, or it’s just 3AM in NYC right now and under the radar?

Similar being reported in Australia. Can see the effects being reported as other timezones become more active.

Probably more to do with luck than being &quot;good&quot;.

Netherlands is somewhat affected: Two main airports, Rotterdam harbour, a few hospitals and news reporting.Surprisingly: banks, government, police, fire department, railways, buses are mostly unaffected.Maybe they have a good IT department&#x2F;provider.

I was at the supermarket here last night about the time it kicked off. It seemed payWave was down, there were a few people walking out empty handed as they only had Apple Pay, etc on them. But the vast majority of people seemed fine, my chipped credit card worked without issue.

Looks like this also took down half of New Zealand&#x27;s economy.<a href="https:&#x2F;&#x2F;www.nzherald.co.nz&#x2F;nz&#x2F;bank-problems-reports-bnz-asb-kiwibank-anz-visa-paywave-services-down&#x2F;R2EY42QKQBALXNF33G5PA6U3TQ&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.nzherald.co.nz&#x2F;nz&#x2F;bank-problems-reports-bnz-asb-...</a>

&quot;F8 + last known good configuration&quot; worked for us

You&#x27;re saying I should diversify my 100% Linux operation to also use Windows?

Disrespect to every CIO to make their business depend on a single operating system, running automatic updates of system software without any canaries and phased deployments.

Agreed. What happened to Patch Tuesdays?!

Presumably they do test their updates, they&#x27;re just maybe not good enough tests.The ideal would be to do canary rollouts (1%, then 5%, 10% etc.) to minimise blast radius, but I guess that&#x27;s incompatible with antiviruses protecting you from 0-day exploits.

The biggest mistake is not ringfencing this update in a test environment before sign-off for general deployment.

I don&#x27;t think the day matter anymore really.The issue is update rollout process, lack of diversity of these kind of tools in the industry, and the absolute failure of the software industry to make decent software without bug and security holes.

Yeah, airlines prefer mid-week chaos &amp; grounding.

The biggest mistake here is running a global update on a Friday. Disrespect to every sysadmin worldwide.

Official CrowdStrike workarround:
1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it. 
4. Boot the host normally.

Workaround + update (within a authenticated portal)<a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;crowdstrike&#x2F;comments&#x2F;1e6vmkf&#x2F;bsod_error_in_latest_crowdstrike_update&#x2F;ldvwkbn&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;crowdstrike&#x2F;comments&#x2F;1e6vmkf&#x2F;bsod_e...</a>

It only applies to a few US carriers<a href="https:&#x2F;&#x2F;abcnews.go.com&#x2F;US&#x2F;american-airlines-issues-global-ground-stop-flights&#x2F;story?id=112092372&amp;cid=social_twitter_abcn" rel="nofollow">https:&#x2F;&#x2F;abcnews.go.com&#x2F;US&#x2F;american-airlines-issues-global-gr...</a>

Just boarded a flight in Iceland, no mention of being grounded.

All flights grounded. World wide airline systems outage.

HP laptops are booting and its loopingDELL laptops are observed, after blue dump . Server is up and running fineTemporary workaround leads compliance issue.From India

Netflix uses FreeBSD for certain parts, but IIRC it&#x27;s Linux outside of the streaming delivery.(Some of the team members lurk around here, maybe we&#x27;ll get lucky and one comments)

It actually runs on a lot of FreeBSD iirc

Flippant commentary:Thank fuck Netflix runs on Linux. I just hope the full chain from my TV to Netflix is immune...

It’s most likely the flame war filtering algorithm of HN. Posts that create a lot of discussion quickly are down ranked until an admin fix the rank manually, or not.

Agreed, I assume people are flagging this post for some reason?

The impact is more than huge.
Our whole workforce is impacted as the world&#x27;s largest tech service provider.

I&#x27;d guess also a significant proportion of us are involved in our company&#x27;s response to the incident.

Most HNers are on the clock while browsing (for work purposes of course) and forcibly afk at the moment.

That is not an understatement. This is literally the largest failing of internet infrastructure to date.Alas, using the internet has given us a lot of efficiency. The trade off is resilience. The entire global system is more brittle than ever but it what gave it such speed.

Did you mean the Azure thing or CS thing? Just happened that the Azure thing did not impact me as I was already off...

This is literally a black swan event that&#x27;ll be memorialized in textbooks, and that&#x27;s not even considering the actual fallout that will follow.

HN has some flame war protection I remember having read, downrating some discussions where it assumes nothing useful is being discussed.Maybe a false positive? (100% speculation. That&#x27;s the problem using closed source software)

I don&#x27;t understand why this outage and the Azure outage earlier don&#x27;t make it to the front page.I&#x27;m getting more up to date technical details from the regular media.This outage looks to be huge.

<a href="https:&#x2F;&#x2F;archive.is&#x2F;AWVuv" rel="nofollow">https:&#x2F;&#x2F;archive.is&#x2F;AWVuv</a>

Dont see how anyone is getting out of this without applying the workaround or reimaging their whole fleet.

Fix: Boot Windows into Safe Mode or the Windows Recovery Environment Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normallyDid it work for you?

BBC live coverage: <a href="https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;live&#x2F;cnk4jdwp49et" rel="nofollow">https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;live&#x2F;cnk4jdwp49et</a>Looks like this is a big deal.

<a href="https:&#x2F;&#x2F;www.bbc.co.uk&#x2F;news&#x2F;live&#x2F;cnk4jdwp49et" rel="nofollow">https:&#x2F;&#x2F;www.bbc.co.uk&#x2F;news&#x2F;live&#x2F;cnk4jdwp49et</a>, seems to be quite a wide impact from this, e.g. Sky News in the UK is off air!

US Based and got a NANOG alert email just in time. At least half our windows servers down.I went into our crowdstrike policies and disabled auto update of the sensor. Hopefully this means it doesnt hit everything. Double check your policies!!!Edit:Crowdstrike has an article out on the manual fix:<a href="https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19" rel="nofollow">https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-W...</a>

POS down? Payment processor down?Lotta different grocery stores and processors in USA. WHO?

I’m Amazed at that. I had no idea the US had caught up and had electronic payments.

Electronic payment at grocery stores in the US is affected.

&gt; It&#x27;s morning here in Europe, departure peak time. We&#x27;re still flying, but...Looks like some airlines aren&#x27;t flying, KLM being one of them.

It&#x27;s not all of Europe. The airports in Norway are operating as normal. One of the airlines reported booking issues on their website, but I haven&#x27;t read about any other issues.Some international flights have been delayed or canceled of course, depending on the route.

&gt; It needs to be done manual by each crew.I can&#x27;t imagine believing that this computation, ordinarily automated by these out of service systems, can be performed correctly by crews that probably haven&#x27;t had to do this in ... years?

It&#x27;s morning here in Europe, departure peak time. We&#x27;re still flying, but...The problems mean the takeoff and weight &amp; balance data is missing. It needs to be done manual by each crew. Baggage handling is also manual, so that means counting bags and the cabin crew counting people. Then manually calculating performance data before you can take off.Big delays everywhere.

Not all. Only some. And many are already lifted.

All airlines in Russia are operating as usual. Sanctions work in unpredictable ways

All major US airlines have put in a total ground stop. No flights can take off anymore.

CrowdStrike achieved PMF - Product-Market Fiasco!

Good news for crowdstrike! It shows how critical their services are. Stock to go up! (And down a bit when they get sued, and up a bit when they don&#x27;t get sued too much, etc.)

I was just at Coles, one of Australia&#x27;s big 2 supermarkets.50% of the self-serve terminals were down. Presumably the others aren&#x27;t far behind.Total chaos.

Oh, look at that, TV has suddenly become watchable without all of the scrolling garbage all over the screen.

Sky News (UK) is back on air, but they seem to have no astons &#x2F; chyrons &#x2F; on screen graphics at all, and I don&#x27;t think they&#x27;re able to show prerecorded material either (it&#x27;s just people in the studio from what I&#x27;ve seen), so presumably they&#x27;re still having fun issues with their general production systems.

If the vending machine handles credit cards, wouldn&#x27;t Visa &#x2F; Mastercard &#x2F; etc. basically require it as part of their security requirements? Or it&#x27;s just general CYA from someone that&#x27;s backfired badly.

presumably they handle credit card transactions

&quot;It&#x27;s a vending machine that sells back physical stolen credit cards.&quot; Input the dollars, receive stolen credit card. Put it next to an airport terminal for maximum impact.

I have seen some pictures shared of vending machines with BSOD.. why would you have an EDR on a vending machine? To protect the valuable assets?!

Schipol (Amsterdam, europe&#x27;s largest) and Melbourne Airport are down

&gt; - Services at London Stock Exchange disruptedjust their news service.&gt; The London Stock Exchange says it&#x27;s working as normal - but says there are problems with its RNS (regulatory news service).&gt; &quot;RNS news service is currently experiencing a third party global technical issue, preventing news from being published on www.londonstockexchange.com,&quot; the statement says.&gt; &quot;Technical teams are working to restore the service. Other services across the group, including London Stock Exchange, continue to operate as normal.&quot;

GLOBAL OUTAGES- Major banks, media and airlines affected by major IT outage- Significant disruption to some Microsoft services- 911 services disrupted in several US states- Services at London Stock Exchange disrupted- Sky News is off air- Flights in Berlin grounded- Reports the issue relates to problem at global cybersecurity firm Crowdstrike

Renaming it .old or whatever would be what a sane person does.They recommended deleting the thing: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41002199">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41002199</a>

No, the userspace program will replace it with a good version.

Yeah, the simple renaming .sys files in safe mode does seem like it would inhibit protection.

Windows IT admins of the world, now is your time. This is what you&#x27;ve trained for. Everything else has led to this moment. Now, go and save the world!!

I think my company has more than 300k+ machines down right now :)SLAs will be breached anyway

It&#x27;s incredibly bad practice, but it seems to be industry normal as we learned today.

This is insane. The company I currently work for provides dinky forms for local cities and such, where the worst thing that could happen is that somebody will have to wait a day to get their license plates, and even we aren&#x27;t this stupid.I feel like people should have to go to jail for this level of negligence.

Their announcement (see Reddit for example) says it was a “content deployment” issue which could suggest it’s the AV definitions&#x2F;whatever rather than the driver itself… so even if you had gradual rollout for drivers, it might not help!

True for all systems, but AV updates are exempt from such policies. When there is a 0day you want those updates landing everywhere asap.Things like zscaler, cs, s1 are updating all the time, nearly everywhere they run.

Surely i&#x27;ts not normal practice to allow patches to be rolled out without a staging&#x2F;testing area on an estate of that size?

Every company I’ve ever worked at has wound up having to install antivirus software to pass audits. The software only ever caused problems and never caught anything. But hey, we passed the audit so we’re good right?

EDR vendors are generally lying unless they tell you anything but &quot;install us if you want to pass certification&quot;.

Yes, from experience, I can say that CS does offer real protection.

AFAIK, most of the people I know that deploy CrowdStrike (including us) just do it to check a box for audits and certifications. They don&#x27;t care much about protections and will happily add exceptions on places where it gives problems (and that&#x27;s a lot of places)

Yes, it offers very real protection. Crowdstrike in particular is the best in the market, speaking from experience and having worked with their competitor&#x27;s products as well and responded to real world compromises.

As a redteamer I guarantee you that a Windows endpoint without EDR is caviar for us...

Just adding my two cents: I work as a pentester and arguably all of my colleagues agree that engagements where Crowdstrike is deployed are the worst because it&#x27;s impossible to bypass.

Anyone who claims CS is nothing but a compliance checkbox has never worked as an actual analyst, of course it&#x27;s effective...no, dur, its worth 50bn for no reason...god some people are stupid AND loud

&gt; Does CS actually offer any real protection? I always thought it was just feel-good software, that Windows had caught up to separating permissions since after XP or so. Either one is lying&#x2F;scamming, but which one?Our ZScaler rep (basically, they technically work for us) come out with massive impressive looking numbers of the thousands of threats they detect and eliminate every monthOddly before we had zscaler we didn&#x27;t seem to have any actual problems. Now we have it and while we have lots of zscaler caused problems around performance and location, we still don&#x27;t have any actual problems.Feels very much like a tiger repelling rock. But I&#x27;m sure the corporate hospitality is fun.

Does CS actually offer any real protection? I always thought it was just feel-good software, that Windows had caught up to separating permissions since after XP or so. Either one is lying&#x2F;scamming, but which one?

The workaround involves booting into Safe mode or Recovery environment, so I&#x27;d guess that&#x27;s a personal visit to most machines unless you&#x27;ve got remote access to the console (e.g. KVM)The info is apparently behind here: <a href="https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19" rel="nofollow">https:&#x2F;&#x2F;supportportal.crowdstrike.com&#x2F;s&#x2F;article&#x2F;Tech-Alert-W...</a>

Our experience so far has been:• Servers, you have to apply the workaround by hand.• Desktops, if you reboot and get online, CrowdStrike often picks up the fix before it crashes. You might need a few reboots, but that has worked for a substantial portion of systems. Otherwise, it’ll need a workaround applied by hand.

The Dildo of Consequences rarely comes lubed, it seems.

Does it require to physically go to each machine to fix it? Given the huge number of machines affected, it seems to me that if this is the case, this outage could last for days.

A bunch of unprotected endpoints all at once on critical systems… what could possibly go wrong?Hope they role out a proper fix soon…

Wow that&#x27;s terrible.  I&#x27;m curious as to whether your contract with them allows for meaningful compensation in an event like this or is it just limited to the price of the software?

This feels like an auto-update functionality. For something that&#x27;s running in kernel space (presumably, if it can BSOD you?) Which is fucking terrifying.

If this comment tree: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41003390">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41003390</a> is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: &quot;Fuck it, let&#x27;s test it in production&quot;, and just pushed it to everyone.

Crowdstrike auto-updates. Please do not spread misinfo.

&gt; &quot;50k windows-based endpoints or so. All down.&quot;I&#x27;m a dev rather than infra guy, but I&#x27;m pretty sure everywhere I&#x27;ve worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.

Are you rolling out CS updates as is everywhere? Are you not testing any published updates immediately at least with some N-1 staging involved?

I&#x27;m pretty sure crowdstrike autoupdates, with 0 option to disable or manually rollout updates. Even worse people running N-1 and N-2 channels also seem to have been impacted by this.

Apparently, CS and ZScaler can apply updates on their own and thats by design, with 0day patches expected to be deployed the minute they are announced.

CS, S1, Zscaler etc auto updates and they have to. Thats the point of the product. If they dont get definitions they cannot protect.

I don&#x27;t mean this to be rude or as an attack, but do you just auto update without validation?This appears to be a clear fault from the companies where the buck stops - those who _use_ CS and should be validating patches from them and other vendors.

Do you need to manually fix all your windows boxes? Or is there a way to update it remotely?

We are a major CS client, with 50k windows-based endpoints or so. All down.There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).

My workplace has a number of people reporting Windows blue-screening and going into a boot loop. The IT Department have a number of servers recently gone offline and have said there&#x27;s a chance that the two issues are related, and potentially due to a Crowd Strike application update.My laptop blue-screened and rebooted, but is working fine after the reboot.A local radio station has also said they&#x27;ve got the same issues with their laptops and their phone system is down as a result.Not seeing anything on news sites yet. Anyone else seeing similar?Above is all based in Australia.

CrowdStrike Update: Windows Bluescreen and Boot Loops