Optimizing Patch Deployment

When running the "Install Wizard" for patch deployment, you must realize that patches will be 1) downloaded to the Master-Agent immediately if needed, and then 2) copied to the previously selected target machines as soon as possible. Scheduling a future date/time in the wizard controls when the patch installation takes place, but for the installation to take place, the files must be on the target machines already, which is why they get pushed out immediately. As an UpdateEXPERT administrator, you must consider:

• How much load am I putting on my network to deploy the patches?

• Will this deployment be disruptive to the users?

• Will this deployment complete in a timely manner?

As an UpdateEXPERT administrator, you need to know:

• Logical and Physical network topology and usage (this is always customer specific)

• How UpdateEXPERT performs file transfers

• UpdateEXPERT usage tips that may help

• UpdateEXPERT deployment options that may help

How UpdateEXPERT performs file transfers

If you want to install 10 patches on 1000 machines there are 10,000 file transfers that need to be accomplished. The Master-Agent uses an efficient thread model to support concurrent tasks. For the file-transfer task, the implemented thread count of 40 means that 40 concurrent file-transfers can be started and managed as resumable-tasks, ie., restartable in case the Master-Agent is rebooted (which does not happen often). As one file-transfer completes, another begins. This means you have a "moving window" of 40 file-transfers.

The speed with which this moving window progresses through the 10,000 file transfers is directly affected by your basic link speed, and any intermediate devices between the Master-Agent and the target machines. For example, file-transfers on a 10-Mbit link will not complete as fast as on a 100-Mbit link. Likewise, 16 targets plugged into a 16-port HUB share bandwidth. 16 targets plugged into a 16-port switch have dedicated bandwidth. This will greatly impact the speed with which files get copied to the targets.

Patch count and size will also greatly affect the aggregate amount of data being pushed across a network. Small patches tend to range between 250K to 1MB. Big patches or service-packs are much bigger, up to 120+MB for service-packs.

Below we have 6 actual patches adding up to 3.3 MB per machine, x 10000 machines = 33000 MB / 1000 = 33 Gigabytes of data! This isn't data being generated from a server application and written to a back-end storage array, this is packetized data flying around your network, being managed by TCP/IP! Eliminating the 898KB patch till a later deployment, reduces the network data load significantly, down to 24 GigaBytes, just to show what a difference one patch can make.

When dealing with large deployments, It may be worthwhile to calculate the overall load. In time, you'll get a good feel for what is an acceptable load for your network, or different parts of your network. Acceptable means "how many machines can I deploy patches too, and have it get done in a timely manner, so that my patch installation occurs on time, with minimal user disruption."

UpdateEXPERT usage tips that may help

TIP-1: Some administrators prefer to manually download the patches they know they'll need for deployment later. For instance, you might manually download security patches in the morning that you intend to deploy with the "install wizard" in the afternoon.

You don't have to do this since patches are downloaded automatically as a result of running the "install wizard", but you can. Remember that downloading files from Microsoft is a one-time operation, meaning that you copy one or more (possibly big) files from Microsoft to your Master-Agent machine for later deployment, so this overhead is not huge.

TIP-2: Always test the patch deployment to one or possibly several machines in order to detect if there are deployment issues to be resolved. Never do a massive deployment without testing at least one target first. This way, you can see several key things:

• Did the target get hung or have a problem of any kind?

• How quiet was the install? (some are quieter than others)

• How many times did it reboot?

• How long did it take to complete?

TIP-3: Start small and scale up, timing the file transfer operation. If you want to deploy patches to say 1000 machines, run the Install Wizard and deploy to 100 machines, timing how long the operation takes.

If it takes 50 minutes for the 40 thread "moving window" to complete the file transfers to 100 machines, it should take 500 minutes (in theory) to perform the file transfers to 1000 machines... this is 8 hours and 20 minutes. Obviously, there are variables here, but you get the idea. Patch count and size will matter, so will the link-speeds and intermediate devices between the Master-Agent and the targets. Is 8 hours and 20 minutes acceptable? Based on "TIP-2" you should know how long the patch will take. Lets say 30 minutes. You now need a 9 hour window minimum for your 1000 machine deployment and installation.

TIP-4: Sometimes administrators want to deploy during business hours, scheduling the actual installation for night. Transferring many files to many machines at once can saturate the network and aggravate users. Based on TIP-3, you should have a sense of how long file transfers take. You know you're network is under some load already. What you can do is select smaller numbers of machines, run the install wizard, monitor when the file transfers are completing as explained below, and then do it again for another set of machines.

Pick a machine, look in the deployment status window, watch for the status to switch from "preparing files …;" to "pending", this refresh and status change occurs automatically, and indicates that patches have been transferred to the target and that the "installer service" is waiting for your scheduled date/time to proceed with the patch installation. When the machines all become "pending", you can schedule another group. The advantage here is you can start deploying patches early in the work day; just do it in smaller chunks so the job completes in some reasonable amount of time. You can simply avoid deployment during "peak" hours. If people really start complaining, you have a smaller number of machines to "cancel".

Lastly, consider targeting machines that you know are on different subnets. That way, each subnet gets some of the file transfer load, but not all of it.

TIP-5: Operations between the Master-Agent and machines on the other side of a T1 link, or a HUB sharing network bandwidth (rather than a switch) may require staggered deployment to smaller numbers. Consider deploying another Master-Agent on the other side of the slow link.

UpdateEXPERT deployment options that may help

Deploying a Master-Agent is traditionally recommended when you have a slow link somewhere in your network, and you are trying to avoid file transfers over the slow link. The Master-Agent is installed and configured to download from the internet, then do file transfers to targets, effectively side-stepping the slow link.

However, In a healthy 100Mbit/1000Mbit network, with efficient switches, and efficient subnetworks, one may be able to sustain higher simultaneous operations by taking advantage of multiple Master-Agents, not to get around a slow-link, but to perform parallel file transfers, ideally on subnetworks.

If one Master-Agent, with 40 threads, cannot perform file transfers fast enough to deal with a high machine count, across a large network, within an acceptable time-frame (see "TIP-3"), then consider installing additional Master-Agents to service different network segments. You can connect to each Master-Agent, select machines in that Master-Agents subnet, and other subnets that make sense, and run the install wizard one or more times to get the patches transferred.

This is a "divide and conquer" strategy, where multiple Master-Agents are used to download and transfer patches in PARALLEL. To optimize this strategy, you need to have isolated, not intermixed packet traffic. Get with your network topology expert, and figure out the best places to install Master-Agents, and which targets should "belong" to the Master. Ideally, the Master-Agent would deploy to the targets in its own subnet, and other locally/efficiently connected subnets in close or reasonable proximity.