If you don’t have Crowdstrike security software on your Windows 10+ computers, you’re one of the lucky ones, along with folks running Linux or macOS.
If you do run Crowdstrike on Windows, this is a bad day, because manual, in-person intervention is the only way to get past the infinitely-looping Blue Screen Of Death affecting millions of computers this morning.
Of course, even if you’ve managed to avoid Crowdstrike on Windows, you’re likely to be affected by this bug, because there’s a good chance that services you use are going to be dealing with it today, and in the meantime will be unusable. That includes 911 call systems, airlines, healthcare providers, and banks.
What happened?
In the early hours of July 19, Crowdstrike pushed out an update for its security software. Crowdstrike client software on millions of computers dutifully applied the update, and the nightmare began.
Microsoft’s advice is apparently to try rebooting affected computers… up to 15 times. Apparently, eventually Windows figures out something is wrong and reverts the problematic update. Or possibly Windows runs long enough to download and install the fixed update from Crowdstrike. I don’t know if this is serious advice or not.
IT folks who manage hundreds or even thousands of affected computers are going to have a very bad day. It will be even worse if those computers are using full disk encryption. Some people are opting to recover from backups, but that gets tricky when encryption is used.
The part that really bugs me about this mess is that Crowdstrike staff clearly did not test the problematic update at all before pushing it out. If they had tested it even once, the problem would have been revealed.
Crowdstrike’s stock is apparently tumbling today, and I’m okay with that, because it will provide ample motivation for the company to improve its testing process.
In the meantime, it might be a good idea to take the next few days off, cancel travel plans, and pick up a good book. Unless you’re an IT person, in which case you’re going to be very busy today.
More about this from The Verge.
Brian Krebs reports on the problem.
Followup analysis from The Verge.
A new recovery tool from Microsoft can help with remediation efforts.