Category Archives: Things that are bad

MORE Windows 10 update problems

Today’s nightmare is brought to you by Microsoft

An open letter to Microsoft:

Dear Microsoft –

Please either allow us to disable Windows 10 updates, or stop pushing out updates that break millions of computers worldwide every few weeks.

Sincerely,
Almost a billion Windows 10 users

The problems with Windows 10 updates are getting worse, not better. The last major feature update (1903) had major issues at release, and more seem to be turning up with each new set of “quality” updates. Those quotes around the word ‘quality’ are very intentional, by the way.

I’ve just spent most of a day troubleshooting and fixing a heinous set of problems related to printing, affecting most of the computers at a retail client. Printing is a critical function for this client, as it is for most businesses.

What follows is the sequence of events leading up to the printing problem, and what finally fixed it.

All of the computers are running 64-bit Windows Professional release 1903 (build 18362.356).

SUMMARY: Update 4522016, which apparently caused these printing problems on some computers, was never installed on any of the affected PCs at this business. Update 4524147 caused the printing problems it was supposed to fix. Uninstalling update 4524147 fixed the printing problems on three otherwise up-to-date Windows 10 PCs.

  1. 2019Oct03: Update 4524147 was installed automatically on all affected PCs. This happened overnight, which is normal for these PCs.
  2. 2019Oct04: The client reported printing problems on several PCs.
  3. 2019Oct04: The usual troubleshooting for printing issues was ineffective. Research eventually showed that a recent Windows update (4522016) was causing printing problems for many users. But that update was never installed on any of the affected PCs.
  4. 2019Oct04: Since printing was working fine before 4524147 was installed, I uninstalled that update, and printing started working again. Repeating this on all affected computers resolved all the printing problems.
  5. 2019Oct05: On trying to log into one of the recently-fixed PCs, Windows 10 told me that the Start menu was broken. Research showed that update 4524147 was causing this problem (the second time an update broke the Start menu in recent weeks). I checked, and sure enough, 4524147 had been reinstalled automatically overnight. Uninstalling it fixed the Start menu.
  6. 2019Oct05: To delay recurrence of the printing problem, I used the Advanced settings on the Windows Update screen to delay updates as long as possible. On most of the PCs, I was able to delay updates for between 30 and 365 days. On one PC, these settings were inexplicably missing. I eventually had to use the Local Group Policy Editor to make the necessary changes.
  7. 2019Oct04: I reported this bizarre situation to Microsoft via its Windows 10 Feedback hub. It’s difficult to know whether anyone at Microsoft will actually see this, or take it seriously. I have doubts, which means that this problem seems likely to reappear at some point.

As predicted

This is in fact the nightmare scenario envisioned by myself and others when it became clear that Windows 10 updates would not be optional. While Microsoft has — grudgingly — made it possible to delay updates, it’s still not possible to avoid them completely, and if you’re one of the unlucky Windows 10 Home users, even that’s not an option.

Questions for Microsoft

Why did an update intended to fix printing problems actually cause those exact problems?

Why are some of the advanced Windows Update settings missing from one of several identically-configured Windows 10 PCs running the same build?

Why are you inflicting this garbage on us? Do you hate us?

WHY DON’T YOU LET US TURN OFF UPDATES? This is the simplest solution, and while I understand that you want Windows 10 installs to be secure (and that means installing fixes for security vulnerabilities), until you can produce updates that don’t cause massive problems, we don’t want them.

Related links

Update 2019Oct10: Apparently update 4517389, released on October 8 along with the rest of October’s updates, addresses this problem.

Firefox 66.0.4 fixes major add-on problem

On May 3, Firefox users all over the world noticed that the browser’s add-ons suddenly stopped working and disappeared from the toolbar. This caused major consternation, as you might imagine. Mozilla has previously made changes to Firefox which disabled some add-ons, so there was initially some concern that this was intentional. However, it turns out that someone at Mozilla failed to renew a critical security certificate, which then expired on May 3rd.

Mozilla added certificate checking to Firefox’s add-ons (extensions, themes, search engines, language packs) some time ago to weed out malicious add-ons and prevent them from being used. When the main certificate expired, Firefox suddenly identified all add-ons as invalid, and disabled them.

Many people use Firefox without add-ons, and those people were unaffected by this problem. Some people, including myself, use add-ons to provide functionality without which Firefox is almost unusable. For example, I use uBlock Origin to prevent Javascript from running on all web pages by default, and Dark Reader to make dark-themed web pages readable.

Once people started noticing the problem, they naturally tried to find workarounds, some of which did more harm than good. Mozilla scrambled to solve the problem, and on May 4 pushed out an official, temporary workaround using a little-known Firefox feature called Studies. Once installed, this fix did re-enable add-ons for many users, but didn’t help if the Studies feature was disabled, and was only effective for desktop versions of the browser.

On May 5 a new version of Firefox was released by Mozilla. Firefox 66.0.4 includes a single change that fixes the certificate expiry problem. There are a few caveats: some add-ons may need to be re-enabled manually. Certain add-ons will remain disabled. Other add-ons may need to be reconfigured.

This was a major (and embarassing) blunder, but Mozilla handled it reasonably well, although the information they published was occasionally somewhat misleading. There’s a useful record of what happened on this Mozilla blog post.

Update 2019May10: Yesterday, Mozilla published a followup/apology post.

Latest Google rug-pull: Google+

Google will terminate Google+ for individuals in the near future. The service will continue to exist for organizations, which presumably includes what Google calls ‘brand accounts’. But for anyone who bought into Google’s hype about the social media service, this is a major disappointment.

Just ask Mike Elgan, one of the more prolific Google+ contributors. In two recent posts, Mike expresses his profound disappointment with Google’s tendency to create new services, coerce people into using them, and then kill those services. I know all about this, having been a victim of Google’s rug-pulling shenanigans myself.

The rationale for Google’s decision to kill Google+ is the discovery of a huge hole in one of its programming interfaces (APIs). Apparently any developer using this API had access to Google+ user data beyond what was supposedly allowed. Lucky for Google+ users, hardly anyone was using this API, just as hardly anyone was using Google+. Anyway, Google fixed the hole back in March but didn’t tell anyone about it.

Okay, Google. This one doesn’t hurt me very much, as my use of Google+ is limited to parroting posts from my blogs to associated brand accounts. I’ll keep the brand accounts around, but I won’t be expanding my use of them. Fool me once… actually, I’ve lost track of how many times this has happened.

Windows 10 October Update is deleting user files

As you may be aware, there’s no longer any practical way to avoid installing Windows 10 updates. Once Microsoft pushes them out, they’re going to end up on your computer whether you want them or not. But maybe you trust Microsoft to make changes to your computer while you sleep (for the record, I’m definitely not). On the other hand, when an update ends up causing problems, it makes these forced updates look downright irresponsible.

According to numerous reports, the recently-announced October Update for Windows 10 is causing user files to be silently deleted. Now, before you go into panic mode, keep in mind that the October Update is not yet being pushed out to all Windows 10 computers: the only way to install it is to manually check for available Windows Updates. For now, the only people affected are those eager types who like to install shiny new things before looking closely at them.

Microsoft is aware of the problem, and they are looking into it, although it’s not at all clear when it might be resolved. Hopefully Microsoft will either pull the update, or at least delay pushing it out to all Windows 10 computers.

If you’re worried about losing files, I strongly suggest backing up all your documents, images, music, video, and other data files. Which you really should be doing anyway. I back up all my data nightly to an external hard drive, using the freeware Cobian Backup.

Update 2018Oct07: Microsoft put a halt to the planned rollout of the October update. The update is still available via Windows Update, so don’t think seeing it listed there means the problem has been fixed. All it means is that the update won’t be pushed out until the issue has been resolved.

Update 2018Oct08: When you shift testing away from professionals and to your user base, quality will suffer. Things are going to slip through. That’s why formal software testing is so important, especially for operating systems and other critical software. Microsoft seems to have made an erroneous assumption: that if you have a (nearly) infinite number of monkeys people using your software, they will find (and reliably reproduce) every bug. In fact, the people doing this unpaid “testing” are mostly power users who are just hoping that their own specific needs will be better served by the latest version. They aren’t testing every scenario, just the same one they tested for the last version. Power users are also much less likely to make the kinds of obvious mistakes that regular folks make, which can lead to surprises even after an update is pushed out to the general public. This situation seems likely to get worse, sadly. The Verge weighs in.

Update 2018Oct16: On October 9, Microsoft made a new (fixed) version of the October update available to users subscribed to the Windows Insider program. Microsoft also seems to understand that the current user-focused testing process is less than ideal: the Windows Insider Feedback Hub now allows users to provide an indication of impact and severity when filing User Initiated Feedback.

More CPU flaws discovered

Microsoft and Google just announced a new CPU speculative execution flaw that’s similar to Spectre and Meltdown: Speculative Store Bypass.

As with Spectre and Meltdown, almost all CPU chips made in the last ten years are affected by this issue.

The Verge: Google and Microsoft disclose new CPU flaw, and the fix can slow machines down.

Bruce Schneier thinks there are more speculative execution flaws coming. And he’s probably right.

Spectre update

Intel has decided not to produce Spectre microcode updates for some of the oldest of their affected CPUs, leaving most Core 2 chips without any hope of a Spectre fix. As for first generation CPUs, some will get updates, and some will not. Microcode updates for all CPUs from generation 2 through generation 8 have already been released.

Not sure whether your computer is affected by Spectre? If you’re running Windows, Gibson Research’s free InSpectre tool will tell you what you need to know. Looking for a Spectre BIOS update for your computer? PCWorld’s guide is a good starting point.

Intel has produced new microcode for most Spectre-affected CPUs, but some manufacturers have yet to provide corresponding BIOS updates for all affected motherboards. They may have decided not to bother developing updates for older motherboards. That’s a potential problem for millions of computers running older CPUs that are new enough to be vulnerable to Spectre. If the manufacturer hasn’t released a BIOS update with Spectre fixes for your motherboard, consider contacting them to find out when that’s going to happen.

Update 2018May24: I contacted Asus about a particular desktop PC I happen to own, and was told that “details on whether or not there will be a Spectre BIOS update for the <model> is [sic] currently not available.” That doesn’t sound very encouraging. It feels like they’re waiting to see how many complaints they get before committing resources to developing patches.

Latest Google rug-pulling is a victory for censorship

Normally when Google cancels a service, it’s annoying and baffling, but we grumble and find an alternative. Google’s latest rug-pull is much worse: it effectively hands a massive win to those who wish to prevent access to things they don’t like.

Until the feature was disabled recently by Google, it was possible to use Google’s App Engine to make web sites and other online resources available to users who would normally be blocked due to state- and corporate-sponsored censorship. The method used was referred to as domain fronting.

Google says they never meant for domain fronting to be possible with App Engine, but they also allowed it to happen for years, without any indication that it was a problem or would be stopped. So people started to rely on the service to get around censorship.

There’s a lot of hate directed towards Google these days, and a lot of it is misguided. From my perspective, enticing users with new services, only to kill those services once they are widely used, is their most infuriating habit.

Spectre/Meltdown nightmare continues

Microsoft has just released ‘out of band’ (outside the usual Patch Tuesday) updates that disable or reverse earlier updates that mitigate Spectre V2. These updates for updates are happening because Intel’s firmware fixes are causing a lot of problems for some folks.

If you were diligent and installed firmware updates on your Windows computers, you should install the new Microsoft updates as soon as possible. Of course doing that will leave your computer exposed to Spectre V2. There’s no solution, other than to be vigilant and extremely careful about visiting shady web sites, installing downloaded software, and clicking links in email.

I guess I’m lucky that no firmware updates are even available for my computers. If they were available and I had installed them, I might be suffering random reboots and even data loss.

Black-hat hackers who are working on malware that exploits the Spectre and Meltdown vulnerabilities are no doubt enjoying this mess, and I have no doubt that we’ll start seeing real-world examples of their handiwork before long.

Spectre/Meltdown CPU flaws: latest news

It’s been about two weeks since the Spectre and Meltdown CPU flaws were revealed to the world, and we now have a better picture of the scope and impact of those flaws.

Intel CPU chips are vulnerable to both Spectre and Meltdown: almost every Intel CPU made since 1995 is affected. AMD CPUs are vulnerable to Spectre, and ARM CPUs, found in millions of mobile and IoT devices, are vulnerable to Meltdown.

Spectre variant 1 and Meltdown have been patched in Windows, macOS, iOS, Android, and Linux. So far, these updates don’t seem to have affected performance on those platforms.

Spectre variant 2 can only be fixed with a firmware update, which will be optional on most platforms, but also seems likely to result in reduced performance. Firmware updates are more difficult to install than software updates. The task should not be undertaken by casual users, since mistakes can result in ‘bricked’ (unusable) devices. One possible exception is Linux, which in some cases allows for updates to be read from a file during startup, eliminating the need for updating firmware.

Intel is making available firmware updates that will hopefully eliminate the threat on affected computers, but — as Microsoft has demonstrated — many of those computers will be slowed significantly by the updates. Intel is downplaying the performance impact, saying that many users won’t even notice the difference.

Microsoft estimates the performance impact of firmware updates on Windows computers with Intel processors will vary depending on:

  • CPU: Haswell and older will be affected more
  • O/S version: Windows 7 and 8 will be affected more than Windows 10
  • I/O bound servers could be affected greatly (Microsoft may recommend avoiding the firmware updates in this case)

Unfortunately, many PC and device makers first learned of the CPU flaws when the rest of us did: on January 3. While Intel, Microsoft, and the other major players knew about the problem months earlier, less high-profile companies are now scrambling to develop firmware updates for their devices. Most are concentrating on their most recent models, and may never release updates for older devices. For example, as of January 21, the Asus web site does not show any recent firmware updates for my Asus M70AD PC. Millions of other devices seem likely to remain permanently vulnerable to Spectre 2.

The Spectre and Meltdown flaws are very deep inside the internal hardware of almost all computers. This makes them very unusual: more difficult to fix, and potentially very dangerous. Even worse, many Internet of Things devices use affected chips; these devices are usually difficult (if not impossible) to update, and may never be fixed.

The vulnerabilities were discovered in early June 2017, and disclosed privately to CPU chip makers first, then to O/S makers, browser makers, cloud and server providers. Some arguably important groups were left out, including CERT, but despite disclosure being handled responsibly, the news leaked out ahead of schedule on January 4. A lot of work had already been done, but hardly anyone was truly ready.

Intel’s response to the flaws in their CPUs has been criticized by some, and it does seem that the chip giant is not being completely transparent. Intel continues to downplay the seriousness of the flaws, and the performance impact of firmware updates. It’s also fair to ask whether in the rush to increase processor speed, security is being neglected by Intel and the other chip makers. The Spectre and Meltdown flaws should arguably have been caught in development.

What are the actual risks involved?

A malicious process on your computer could read data from another process (such as your banking app) and send it to anyone. This kind of exploit has been demonstrated as effective, and it can even be accomplished using specially-crafted Javascript code on a web site.

A malicious process on a web-based service, server, or virtual machine could read data from another process on that machine or a virtual machine that’s controlled by someone else.

Risks going forward: this has all been rushed (despite some advance warning), and the changes are at the core of CPUs and O/S kernels. Emergency fixes have a way of causing new, hidden problems. We will probably be dealing with the fallout from these flaws for months.

Update 2018Jan23: Intel is now telling us to avoid earlier firmware updates while they work on new updates that (hopefully) avoid rebooting issues on computers running Haswell and Broadwell CPUs. Meanwhile, there’s some strong language coming from Linus Torvalds (Linux’s creator) about the quality of the firmware fixes coming from Intel.

Major slowdowns headed for almost all computers

Major patches are coming, for most operating systems and devices running modern (made in the last 10 years or so) processors. Changes to Windows, Linux, macOS, and most other systems will modify the way memory is used, ameliorating critical CPU security flaws, and slowing them down significantly in the process.

There’s been a lot of secrecy around this issue, with details of the flaws — discovered several months ago — only now coming to light as O/S vendors scramble to prepare patches. The flaws (commonly referred to as Spectre and Meltdown) involve potential leaking of information, as described in a recent post on The Register:

At best, the vulnerability could be leveraged by malware and hackers to more easily exploit other security bugs.

At worst, the hole could be abused by programs and logged-in users to read the contents of the kernel’s memory. Suffice to say, this is not great. The kernel’s memory space is hidden from user processes and programs because it may contain all sorts of secrets, such as passwords, login keys, files cached from disk, and so on.

Much of this is still speculation, but the reality may be even worse, so hang onto your socks, since this is going to get ugly. It’s easy to imagine class action lawsuits arising out of the mess.

Those of you running light operating systems on older hardware may have the last laugh: while many of the world’s computers will soon be noticeably — and unavoidably — slower, yours will keep chugging along unaffected… at least until they’re used to access any of the millions of computers that power web sites and services. Major providers may have no choice but to install the updates, significantly reducing the processing power of their systems.

For computers running Windows 10, system updates are literally unavoidable, and the slowdown inevitable. The rest of us will need to decide whether to risk leaving the vulnerabilities exposed, or patch them and deal with the resulting performance hit. Exploiting the vulnerabilities is not straightforward, and it should be possible to stay safe by avoiding risky behaviour, such as indiscriminately running unknown software, visiting dubious web sites, and opening links in email. However, the full extent of the risks involved is not yet known.

Related articles

The Verge: Intel’s processors have a security bug and the fix could slow down PCs
The Verge: Microsoft issues emergency Windows update for processor security bugs
The Verge: Intel says processor bug isn’t unique to its chips and performance issues are ‘workload-dependent’
The Verge: Processor flaw exposes 20 years of devices to new attack
The Verge: How to protect your PC against the major ‘Meltdown’ CPU security flaw
Google Security Blog: Today’s CPU vulnerability: what you need to know
Bruce Schneier: Spectre and Meltdown Attacks
SANS InfoSec: Spectre and Meltdown: What You Need to Know Right Now
Techdirt: A Major Security Vulnerability Has Plagued ‘Nearly All’ Intel CPUs For Years

Update 2018Jan04: Corrected title and content to show that the problem affects all modern processors, not just those made by Intel, and that there are multiple vulnerabilities. Also added more related articles.

KRACK Wi-Fi vulnerability: what you need to know

Last week, security researchers identified a series of vulnerabilities affecting almost all Wi-Fi devices, from computers to refrigerators. The vulnerability could allow attackers to intercept wireless communications and potentially steal credentials and other sensitive information. The vulnerabilities are collectively referred to as KRACK.

The good news is that computers running Windows and Linux already have patches available. Microsoft included fixes in the October 2017 Patch Tuesday updates.

Apple says that fixes are ready for MacOS, but there’s no word on exactly when they will actually be made available.

The bad news is that mobile devices, particularly those that run Google’s Android operating system, are vulnerable, and in some cases, might stay that way indefinitely. That’s because even though Google has prepared fixes for Android, those fixes won’t get to devices made by other vendors until those vendors make them available. Some vendors are better than others at pushing updates to their devices. Worse, some devices running older O/S versions may never get updates at all, rendering them permanently insecure.

There are mitigating factors. First, because of the responsible way in which these vulnerabilities were reported, Microsoft and other major players have had time to develop fixes, while details of the vulnerabilities were kept relatively secret until recently. That means we have a head start on the bad guys this time.

Second, exploiting these vulnerabilities requires close proximity. Attacks based on these vulnerabilities can’t be executed over the Internet.

Use caution with unpatched devices

If you use a public Wi-Fi access point with an unpatched device, you’re exposed. So until patches for your device become available, you might want to disable its Wi-Fi when you’re not at home. Most devices have settings that prevent automatically connecting to Wi-Fi networks it finds in the vicinity.

IoT devices may remain vulnerable forever

‘Internet of Things’ (IoT) devices, including thermostats, cars, appliances, and basically anything that can have a computer stuffed into it, often connect to the Internet using Wi-Fi. There are no security standards for IoT devices yet, and many are extremely unlikely to ever be patched.

Recommendation: identify all of your IoT devices that have the ability to connect to the Internet. For each, make sure that you’re using a wired connection, or disable networking completely, if possible. As for devices that connect to the Internet via Wi-Fi and cannot or won’t be patched or disabled, consider taking them to the nearest landfill.

References