How to recover a Cisco 3850 switch from zombie mode

I have done it again, I haven't been really successful in deploying any of my stack switches. I always had problems with firmware and adding new switch you name it. Today I had a switch which was auto upgrading as I added it into an existing stack and I had to pull the plug; the reason was because it drop my last switch on the stack. I went into panic mode and just pull the plug.

I did some research and noticed that I could just copy the firmware onto a USB stick and boot the .bin file from USB but that failed. Good thing there was a management port behind the switch to initial emergency-install.


Booting...Initializing and Testing RAM ++++@@@@####...################################++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done.
Memory Test Pass!

Base ethernet MAC Address: 0c:f5:a4:1d:4e:80

Interface GE 0 link down***ERROR: PHY link is down
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 2 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 82665136
Image failed verification, cannot boot this image
flash:packages.conf: bad/non-bootable Nova bundle file

Error loading "flash:packages.conf"

Interrupt within 5 seconds to abort boot process.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 82665136
Image failed verification, cannot boot this image
flash:/packages.conf: bad/non-bootable Nova bundle file

Error loading "flash:/packages.conf"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.


switch: flash_init
Initializing Flash...
...The flash is already initialized.

switch: dir flash:
Directory of flash:/

 7745  drwx  4096       .
    2  drwx  4096       ..
 7746  -rwx  2097152    nvram_config
 7747  -rwx  74369716   cat3k_caa-base.SPA.03.02.03.SE.pkg
 7748  -rwx  5808828    cat3k_caa-drivers.SPA.03.02.03.SE.pkg
 7749  -rwx  32496484   cat3k_caa-infra.SPA.03.02.03.SE.pkg
 7750  -rwx  30418104   cat3k_caa-iosd-universalk9.SPA.150-1.EX3.pkg
 7751  -rwx  16059104   cat3k_caa-platform.SPA.03.02.03.SE.pkg
 7752  -rwx  64586444   cat3k_caa-wcm.SPA.10.0.120.0.pkg
 7753  -rw-  1244       packages.conf
 7754  -rwx  796        vlan.dat
 7755  -rw-  82665136   cat3k_caa-base.SPA.03.06.04.E.pkg
 7756  -rw-  33784816   cat3k_caa-infra.SPA.03.06.04.E.pkg
 7757  -rw-  27417488   cat3k_caa-platform.SPA.03.06.04.E.pkg
 7758  -rw-  43021636   cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
 7759  -rw-  111180608  cat3k_caa-wcm.SPA.10.2.140.0.pkg
 7760  -rw-  4913852    cat3k_caa-drivers.SPA.03.06.04.E.pkg

1393414144 bytes available (254480384 bytes used)

switch: boot flash:cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
flash:cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg: bad/non-bootable Nova bundle file

Error loading "flash:cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin
usbflash0:cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin: no such device
usbflash0:cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin: no such device

Error loading "usbflash0:cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

switch: dir

List of filesystems currently registered:

              xmodem[0]: (read-only)
                null[1]: (read-write)
                  bs[3]: (read-only)
               flash[18]: (read-only)
                tftp[19]: (read-write)


switch: show boot
Unknown cmd: show

switch: cp flash:packages.conf flash:packages.conf.badop
Unknown cmd: cp

switch: dir usb
unable to stat usb/: no such device

switch: dir ?
unable to stat ?/: no such device

switch: dir

List of filesystems currently registered:

              xmodem[0]: (read-only)
                null[1]: (read-write)
                  bs[3]: (read-only)
               flash[18]: (read-only)
                tftp[19]: (read-write)


switch: dir flashusb
unable to stat flashusb/: no such device
     
switch dir flash
Unknown cmd:

switch: dir flash:
Directory of flash:/

 7745  drwx  4096       .
    2  drwx  4096       ..
 7746  -rwx  2097152    nvram_config
 7747  -rwx  74369716   cat3k_caa-base.SPA.03.02.03.SE.pkg
 7748  -rwx  5808828    cat3k_caa-drivers.SPA.03.02.03.SE.pkg
 7749  -rwx  32496484   cat3k_caa-infra.SPA.03.02.03.SE.pkg
 7750  -rwx  30418104   cat3k_caa-iosd-universalk9.SPA.150-1.EX3.pkg
 7751  -rwx  16059104   cat3k_caa-platform.SPA.03.02.03.SE.pkg
 7752  -rwx  64586444   cat3k_caa-wcm.SPA.10.0.120.0.pkg
 7753  -rw-  1244       packages.conf
 7754  -rwx  796        vlan.dat
 7755  -rw-  82665136   cat3k_caa-base.SPA.03.06.04.E.pkg
 7756  -rw-  33784816   cat3k_caa-infra.SPA.03.06.04.E.pkg
 7757  -rw-  27417488   cat3k_caa-platform.SPA.03.06.04.E.pkg
 7758  -rw-  43021636   cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
 7759  -rw-  111180608  cat3k_caa-wcm.SPA.10.2.140.0.pkg
 7760  -rw-  4913852    cat3k_caa-drivers.SPA.03.06.04.E.pkg

1393414144 bytes available (254480384 bytes used)

switch:

switch: ?
           ? -- Present list of available commands
         arp -- Show arp table or arp-resolve an address
        boot -- Load and boot an executable image
         cat -- Concatenate (type) file(s)
        copy -- Copy a file
      delete -- Delete file(s)
         dir -- List files in directories
emergency-install -- Initiate Disaster Recovery
  flash_init -- Initialize filesystem(s)
      format -- Format a filesystem
        fsck -- Check filesystem consistency
        help -- Present list of available commands
   mgmt_init -- initialize management port
       mkdir -- Create dir(s)
        more -- Concatenate (display) file(s)
        ping -- Send ICMP ECHO_REQUEST packets to a network host
      rename -- Rename a file
       reset -- Reset the system
       rmdir -- Delete empty dir(s)
         set -- Set or display environment variables
      set_bs -- Set attributes on a boot sector filesystem
   set_param -- Set system parameters in flash
        type -- Concatenate (type) file(s)
       unset -- Unset one or more environment variables
     version -- Display boot loader version

switch: format ?
Unknown filesystem "?"

switch: format
usage: format <filesystem>

switch: copy flash:packagesconf flash:packages.conf.bad
flash:package.conf: no such file or directory

switch: copy flash:packages.conf flash:packages.bad
flash:packages.bad: read only file system

switch: version
CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)
Compiled Tue Mar 19 14:29:45 PDT 2013 by rel

switch: ?
           ? -- Present list of available commands
         arp -- Show arp table or arp-resolve an address
        boot -- Load and boot an executable image
         cat -- Concatenate (type) file(s)
        copy -- Copy a file
      delete -- Delete file(s)
         dir -- List files in directories
emergency-install -- Initiate Disaster Recovery
  flash_init -- Initialize filesystem(s)
      format -- Format a filesystem
        fsck -- Check filesystem consistency
        help -- Present list of available commands
   mgmt_init -- initialize management port
       mkdir -- Create dir(s)
        more -- Concatenate (display) file(s)
        ping -- Send ICMP ECHO_REQUEST packets to a network host
      rename -- Rename a file
       reset -- Reset the system
       rmdir -- Delete empty dir(s)
         set -- Set or display environment variables
      set_bs -- Set attributes on a boot sector filesystem
   set_param -- Set system parameters in flash
        type -- Concatenate (type) file(s)
       unset -- Unset one or more environment variables
     version -- Display boot loader version

switch: del flash:packages.conf
Are you sure you want to delete "flash:packages.conf" (y/n)?y
File "flash:packages.conf" not deleted -- read only file system

switch: set IP_ADDR 192.168.1.1/255.255.255.0

switch: set DEFAULT_ROUTER 192.168.1.2

switch: ping 192.168.1.2
ping 192.168.1.2 with 32 bytes of data ...
Up 1000 Mbps Full duplex (port  0) (SGMII)
Request timed out.
Host 192.168.1.2 is alive.

switch: dir sda9:
Directory of sda9:/

    2  drwx  1024       .
    2  drwx  1024       ..
   11  -rwx  18920900   cat3k_caa-recovery.bin

36941824 bytes available (20828160 bytes used)

switch: emergency-install tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin
Unknown cmd: emerency-install

switch: emergency-install tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin
The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery (tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin)...
Reading full image into memory......................done
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x67c0f5d8
Kernel Size       : 0x31794f/3242319
Initramfs Address : 0x67f26f28
Initramfs Size    : 0xdbec9d/14412957
Compression Format: .mzip

Bootable image at @ ram:0x67c0f5d8
Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x811060f0
Loading Linux kernel with entry point 0x811060f0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

hub 1-0:1.0: unable to enumerate USB device on port 1
usb 2-1: device descriptor read/64, error -32
usb 2-1: device descriptor read/64, error -32
usb 2-1: device descriptor read/64, error -32
usb 2-1: device descriptor read/64, error -32
usb 2-1: device not accepting address 4, error -32
usb 2-1: device not accepting address 5, error -32
hub 2-0:1.0: unable to enumerate USB device on port 1


Initiating Emergency Installation of bundle tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin


Downloading bundle tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin...

Validating bundle tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin...
Installing bundle tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin....
Verifying bundle tftp://192.168.1.2/cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin...
Package cat3k_caa-base.SPA.03.06.05E.pkg is Digitally Signed
Package cat3k_caa-drivers.SPA.03.06.05E.pkg is Digitally Signed
Package cat3k_caa-infra.SPA.03.06.05E.pkg is Digitally Signed
Package cat3k_caa-iosd-universalk9.SPA.152-2.E5.pkg is Digitally Signed
Package cat3k_caa-platform.SPA.03.06.05E.pkg is Digitally Signed
Package cat3k_caa-wcm.SPA.10.2.150.0.pkg is Digitally Signed
Preparing flash....
Syncing device....
Emergency Install successful... Rebooting
Restarting system.


Booting...Initializing and Testing RAM ++++@@@@####...################################++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done.
Memory Test Pass!

Base ethernet MAC Address: 0c:f5:a4:1d:4e:80
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 83297376
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6042d34c
Kernel Size       : 0x41884e/4294734
Initramfs Address : 0x60845b9c
Initramfs Size    : 0xe778c3/15169731
Compression Format: .mzip

Bootable image at @ ram:0x6042d34c
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 380
Loading Linux kernel with entry point 0x8166a600 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services


FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
MCU type = 0 and image name is main_mcu_ucode_bundle_6_2_0.tar
Final tar file: main_mcu_ucode_bundle_6_2_0.tar


Front-end Microcode IMG MGR: found 4 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_0
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_1
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_2
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_3

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.

ST27

Burger addict that can't say no to fries. Weighted in at 105kg once upon a time and love Japan a lot.

1 comment:

  1. Couple notes:

    #1. You did it wrong - for bundle mode, you boot the bin file, not the pkg file. Then, you do software expand running switch 1 to flash: to convert back to install mode. It requires another reload, but still quicker than tftp.

    #2. You're lucky you did it wrong - emergency-install usually works. Bundle mode, which is supposed to work, often doesn't.

    #3. Don't buy 3850 switches. As you are finding, they are nothing but trouble.

    ReplyDelete