Many years ago I upgraded a Cisco 3750X stack to a newer version of IOS. Since the production system I was planning to upgrade had some critical systems on it, I tested the process on a stack in the lab first.
At the outset I figured “no problem, this will take a few minutes to reboot and we’ll be back up and running.” Little did I know I was in for a long wait… Unbeknownst to me the version of code I was upgrading to included a Microcode update which adds a considerable amount of time to the process… emphasis on considerable.
The stack was running 12.2-53.SE2 and I was upgrading it to 12.2-58.SE2, which at the time of the upgrade was the recommended 12 version (I’ve had problems with IOS 15 on this platform, so I’ll be sticking with version 12 for now).
After copying the new version to the stack switches and setting the boot variable, I rebooted the switch stack.
Switch(config)#boot system flash:/c3750e-universalk9-mz.122-58.SE2.bin Switch(config)#exit Switch#reload
Looked like a normal reload until the console displayed this:
extracting front_end/front_end_ucode_info (309 bytes) Software version is different so extracting the whole bundle front_end/ (directory) extracting front_end/fe_type_4 (78520 bytes) extracting front_end/fe_type_3 (78520 bytes) extracting front_end/fe_type_2 (78520 bytes) extracting front_end/fe_type_5_1 (307958 bytes) extracting front_end/fe_type_5_2 (25150 bytes) extracting front_end/fe_type_5_0 (594412 bytes) extracting front_end/fe_type_5_3 (84734 bytes) extracting front_end/front_end_ucode_info (309 bytes) extracting ucode_info (77 bytes) Front-end Microcode IMG MGR: Installed 4 image(s) in cache: Front-end Microcode IMG MGR: found microcode images for 4 devices. Image for front-end 0: flash:/front_end_ucode_cache/ucode.3 Image for front-end 0: flash:/front_end_ucode_cache/ucode.4 Image for front-end 0: flash:/front_end_ucode_cache/ucode.5 Image for front-end 0: flash:/front_end_ucode_cache/ucode.6 Front-end Microcode IMG MGR: Preparing to program device microcode... Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes. Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssssssssss sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssss ssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssss sssssssss % Front-end Microcode IMG MGR: address 0, PROGRAM failed % Front-end Microcode IMG MGR: unable to program device: 0 % Front-end Microcode IMG MGR: malformed microcode cache
The switch automatically reloaded itself after this and that’s when the really long wait began. Here’s the output from the console during the upgrade portion of the reload:
extracting front_end/front_end_ucode_info (309 bytes) Software version is different so extracting the whole bundle front_end/ (directory) extracting front_end/fe_type_4 (78520 bytes) extracting front_end/fe_type_3 (78520 bytes) extracting front_end/fe_type_2 (78520 bytes) extracting front_end/fe_type_5_1 (307958 bytes) extracting front_end/fe_type_5_2 (25150 bytes) extracting front_end/fe_type_5_0 (594412 bytes) extracting front_end/fe_type_5_3 (84734 bytes) extracting front_end/front_end_ucode_info (309 bytes) extracting ucode_info (77 bytes) Front-end Microcode IMG MGR: Installed 4 image(s) in cache: Front-end Microcode IMG MGR: found microcode images for 4 devices. Image for front-end 0: flash:/front_end_ucode_cache/ucode.3 Image for front-end 0: flash:/front_end_ucode_cache/ucode.4 Image for front-end 0: flash:/front_end_ucode_cache/ucode.5 Image for front-end 0: flash:/front_end_ucode_cache/ucode.6 Front-end Microcode IMG MGR: Preparing to program device microcode... Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes. Front-end Microcode IMG MGR: Programming device 0...rrrrrrwsssssssssssssssssss pssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssss sssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssss sssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssss sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssss sssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssss sssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssss sssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsss sssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssssps sssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssss pssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssss sssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssss sssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssss sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssss sssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssss sssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssss sssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsss sssssssssssssssspssssssssssssssssssspssssssssssssss.... (you get the idea) % Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload
One more reload after this and things were back up and running.
Normal reboot time is approximately 5 minutes. This upgrade included three reboots and the total downtime was approximately 33 minutes. Thank goodness I tested this in a lab environment first to get the appropriate times for a change window. Otherwise it would have been one of those helpless frustrating 2:00 am phone calls with TAC.
So, if you are planning on upgrading your 3750X stack, be sure to plan accordingly so you don’t blow your change window.
- Tags: Cisco, code upgrade