Friday, April 12, 2013

Do Not Install Every Windows Update


As a Managed Service Provider (MSP), we manage our client's computers. One of the most important things we do is to make sure that patches, fixes, and updates are installed. But that's a bit more complicated than it sounds.

For example, Microsoft released an update three days ago that is now causing many machines to fail.Today we see this headline:


Microsoft Tells Windows 7 Users to Uninstall Update - PCs Can Fail to Restart



Please see "The Problem" at the end of this blog post. The bottom line is: An automatic update applied to all machines on Tuesday is making some machines fail and others to display false messages about licensing.

Now Microsoft is recommending that you un-install this patch and wait for a patched patch that doesn't have the problem.


Lesson: Do Not Install Every Update!

Okay. So you should not install every update. Or at least you need a system to determine which updates should and which should not be installed.

A good computer consultant (managed service provider) will have a process for patching machines. It looks something like this:

- Microsoft releases patch

- Patch is tested by a third party to verify that there are no problems

- If blacklisted, patch is not deployed to client machines

- If whitelisted, patch is deployed to client machines

A good computer consultant will use a Remote Monitoring and Management tool to deploy patches. That way, everyone gets them at once. But they only get the ones that are safe! That's one of the reasons we call it managed services. We manage your machines.

You might think this service costs extra money. But most MSPs simply include it as part of their regular support. After all, the computer consultant has to spend less time fixing machines if the "patches" are all safe and whitelisted. And you, the client, can keep working without interruption.

In a perfect world, you should never have to know that a patch was released and failed. You should just keep working. Your consultant should help you avoid these issues altogether. Your consultant should NOT be charging you to uninstall patches like this simply because he doesn't have a system to avoid them in the first place.

If you don't have a managed service agreement, of course you'll need to pay someone to uninstall this patch. But if you do have a managed service agreement for your computers, then this is just another beautiful Spring day where you can worry about what YOU do for a living and not about your technology.


Lesson: Hire The Right Consultant


Before you hire a technician or consultant, ask about the patch management system they use.

If they stare at you, then blink, and say that they rely on Microsoft's "Automatic Updates," you need to keep interviewing. Automatic Updates put your machine at risk. Yes, they're safe 99.9% of the time. But if you just spent three days trying to figure out why your machines won't start, then that .1% becomes very expensive.

In the 21st Century, every computer consultant should be using an automated patch management (remote monitoring and management) system. If your I.T. person doesn't even know what that is, you should step up to more professional support.

It's guaranteed to cost you less money because you'll have fewer problems and more UP-Time for your computers.


The Problem

Here's what's going on.

1) Microsoft tried to fix a potential security problem. See Microsoft Security Bulletin MS13-036. The "fix" was release in Microsoft Update 2823324. See the Knowledge Base article on Microsoft Update 2823324.

2) Some machines (Windows 7 as far as I can tell from the reports) do not restart after the patch is applied.

3) Some machines give false reports that software licenses are not valid.

4) Now Microsoft is recommending that you uninstall that patch while they work on patching the patch.

Microsoft maintains a blog for talking about these things. See the Microsoft Security Center Response blog. Here's what they say about the issue:

"We are aware that some of our customers may be experiencing difficulties after applying security update 2823324, which we provided in security bulletin MS13-036 on Tuesday, April 9. We’ve determined that the update, when paired with certain third-party software, can cause system errors. As a precaution, we stopped pushing 2823324 as an update when we began investigating the error reports, and have since removed it from the download center."
:-)

9 comments:

  1. You mention "- Patch is tested by a third party to verify that there are no problems"

    I have always done my own testing. Can you suggest some 3rd party patch verification vendors?

    ReplyDelete
  2. We use Continuum (http://www.continuum.net/), formerly Zenith. Pretty much all of the patch management systems now verify and whitelist/blacklist updates. Or they make recommendations and you implement your own lists.

    Some line of business applications are stuck in the past, so you have to stop Internet Explorer updates and sometimes even service packs.

    The simplest test is often just to wait. Give an update two weeks in the wild before you apply it. Let the world be your beta tester.

    ReplyDelete
  3. No third party is as well qualified as you to vet patches. That's because they don't have your (or your customers') environment to test against. Sure, that new patch may work great on a virgin Windows 7 system - but what happens when that same patch is applied to a Windows 7 system with Java, Adobe Reader, etc.. already installed? Never mind those environments running "That App" (you know, that business critical application that was written 17 years ago which *has* to work.) Bottom line: test environments, virtual environments and sample testing are the most reliable ways to safely deploy updates into any environment. If you farm that work our, you deny the environment its best resource - your experience.

    ReplyDelete
  4. The best method is not what you think...

    Although the size of the business/budget typically plays a factor on how much testing is done, the real measure of how much testing should be done is a business risk analysis based on cost of downtime (wasted expenses, lost revenue, lost customer confidence, length of downtime, etc.)

    Without knowing what the risk is, how do you know how much time (i.e. $$) to invest in preventing downtime?
    For a 3 person company where their only apps are Quickbooks and MS Office, and they all live in the cloud, having a PC freak out due to a bad patch is not all that significant since they can likely work from any other device or location. (Mind you, I am not saying they shouldn't monitor their systems.)

    For a 500 seat company with payroll full of highly paid engineers, and a critical LoB application that relies on various components to work, they should likely do testing similar to LabTechRob's recommendations above.

    There are many variables here, and many possible solutions to mitigate losses due to bad updates. Some solutions work proactively (relatively expensive), e.g. test environments, and some work reactively, e.g. backup/restore/rollback solutions (relatively inexpensive), and most others fall somewhere in-between. Sometimes more than one method is needed/justified.

    As a professional IT consultant, it is our responsibility to advise our customers/employers what the business risks are related to IT based on business criteria, and recommend the appropriate solution to meet business goals.
    Any other solution is incorrect.

    ReplyDelete
  5. Ok I see what can happen sometime when you do patch, but what would happen if you don't. Virus and malware attracts and other thing.
    I know some problem may show up but over all patches our a good things to do.

    ReplyDelete
  6. To patch or not to patch, to wait or to not wait, bad patch or virus? Darned if you do, darned if you don't. Based on practical, historical, experience, which do you suppose is more likely to happen - a bad patch or a virus infection? I'm pretty confident in Microsoft patches (not 100%, but close), and I'm also very confident in the protection mechanisms employed and regular patch schedule that we already follow for our customers. I feel it is sufficiently acceptable as Karl suggested, to hold off on patching systems for 1 - 2 weeks (unless Microsoft came out and said this one is HUGELY critical to install now) without increasing our customers' exposure to some sort of compromise resulting from a released patch that has not been installed. We manage about 100 systems and not one was directly impacted by this patch nor any malware...

    ReplyDelete
  7. It's interesting ... the days of fast-spreading horrible viruses like Melissa are gone. We still find horrible viruses, but so many systems are patched - and people are trained - that there's a limit to how fast things can spread.

    At the same time, Microsoft's vetting process for patches has gotten better and better. But glitches still show up. Balancing the danger of viruses vs. the danger of creating a glitch that didn't exist before is part art as well as science.

    And that's where a professional comes in!

    ReplyDelete
  8. I've run across some MSP firms that do NO patching on servers at all. They rely on a weekly scan with MalwareBytes free edition to find issues. I found a SQL injections exploit running, malware, crapware, adware and rootkits running with nothing done by these "IT Professionals". I hate that companies like this exist in my profession.

    ReplyDelete