Upgrading 3750X can take longer than you think

No Comments Share:

Many years ago I upgraded a Cisco 3750X stack to a newer version of IOS. Since the production system I was planning to upgrade had some critical systems on it, I tested the process on a stack in the lab first.

At the outset I figured “no problem, this will take a few minutes to reboot and we’ll be back up and running.” Little did I know I was in for a long wait… Unbeknownst to me the version of code I was upgrading to included a Microcode update which adds a considerable amount of time to the process… emphasis on considerable.

The stack was running 12.2-53.SE2 and I was upgrading it to 12.2-58.SE2, which at the time of the upgrade was the recommended 12 version (I’ve had problems with IOS 15 on this platform, so I’ll be sticking with version 12 for now).

After copying the new version to the stack switches and setting the boot variable, I rebooted the switch stack.

Switch(config)#boot system flash:/c3750e-universalk9-mz.122-58.SE2.bin
Switch(config)#exit
Switch#reload

Looked like a normal reload until the console displayed this:

extracting front_end/front_end_ucode_info (309 bytes)
 Software version is different so extracting
 the whole bundle

front_end/ (directory)
extracting front_end/fe_type_4 (78520 bytes)
extracting front_end/fe_type_3 (78520 bytes)
extracting front_end/fe_type_2 (78520 bytes)
extracting front_end/fe_type_5_1 (307958 bytes)
extracting front_end/fe_type_5_2 (25150 bytes)
extracting front_end/fe_type_5_0 (594412 bytes)
extracting front_end/fe_type_5_3 (84734 bytes)
extracting front_end/front_end_ucode_info (309 bytes)
extracting ucode_info (77 bytes)

Front-end Microcode IMG MGR: Installed 4 image(s) in cache:

Front-end Microcode IMG MGR: found microcode images for 4 devices.
Image for front-end 0: flash:/front_end_ucode_cache/ucode.3
Image for front-end 0: flash:/front_end_ucode_cache/ucode.4
Image for front-end 0: flash:/front_end_ucode_cache/ucode.5
Image for front-end 0: flash:/front_end_ucode_cache/ucode.6

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssssssssss
sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssss
ssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss
sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssss
sssssssss
% Front-end Microcode IMG MGR: address 0, PROGRAM failed


% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache

 

The switch automatically reloaded itself after this and that’s when the really long wait began. Here’s the output from the console during the upgrade portion of the reload:

 

extracting front_end/front_end_ucode_info (309 bytes)
 Software version is different so extracting
 the whole bundle

front_end/ (directory)
extracting front_end/fe_type_4 (78520 bytes)
extracting front_end/fe_type_3 (78520 bytes)
extracting front_end/fe_type_2 (78520 bytes)
extracting front_end/fe_type_5_1 (307958 bytes)
extracting front_end/fe_type_5_2 (25150 bytes)
extracting front_end/fe_type_5_0 (594412 bytes)
extracting front_end/fe_type_5_3 (84734 bytes)
extracting front_end/front_end_ucode_info (309 bytes)
extracting ucode_info (77 bytes)

Front-end Microcode IMG MGR: Installed 4 image(s) in cache:

Front-end Microcode IMG MGR: found microcode images for 4 devices.
Image for front-end 0: flash:/front_end_ucode_cache/ucode.3
Image for front-end 0: flash:/front_end_ucode_cache/ucode.4
Image for front-end 0: flash:/front_end_ucode_cache/ucode.5
Image for front-end 0: flash:/front_end_ucode_cache/ucode.6

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrwsssssssssssssssssss
pssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss
sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssss
sssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssss
sssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssss
sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssss
sssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssss
sssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssss
sssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsss
sssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssssps
sssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssss
pssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss
sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssss
sssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssss
sssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssss
sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssss
sssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssss
sssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssss
sssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsss
sssssssssssssssspssssssssssssssssssspssssssssssssss.... (you get the idea)
% Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload

One more reload after this and things were back up and running.

Normal reboot time is approximately 5 minutes. This upgrade included three reboots and the total downtime was approximately 33 minutes. Thank goodness I tested this in a lab environment first to get the appropriate times for a change window. Otherwise it would have been one of those helpless frustrating 2:00 am phone calls with TAC.

So, if you are planning on upgrading your 3750X stack, be sure to plan accordingly so you don’t blow your change window.

Previous Article

How to kill a process in linux

Next Article

SecureCRT Chat Window Feature

You may also like