How to use emergency-install for Cisco 3850 recovery

Over the weekend I upgraded my Cisco 3850 48 P-L from IOS 3.x.x to Denali 16.x.x. During this process the switch upgrade process failed and the switch was stuck


Booting...
Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 22301472
Bundle Image
--------------------------------------
Kernel Address    : 0x53779384
Kernel Size       : 0x34e9e1/3467745
Initramfs Address : 0x53ac7d65
Initramfs Size    : 0x119d5bb/18470331
Compression Format: mzip

Bootable image at @ ram:0x53779384
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 623
Loading Linux kernel with entry point 0x816902d0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

>>> Boot Failed - pulling status and logs:
● late-network.service - Late network setup
   Loaded: loaded (/lib/systemd/system/late-network.service; static)
   Active: failed (Result: exit-code) since Sun 2018-07-29 18:19:54 Universal; 10s ago
  Process: 1033 ExecStart=/etc/init.d/network start (code=exited, status=203/EXEC)
Main PID: 1033 (code=exited, status=203/EXEC)

[  123.064196] localhost systemd[1]: late-network.service: main process exited, code=exited, status=203/EXEC
[  123.065260] localhost systemd[1]: Failed to start Late network setup.
[  123.066309] localhost systemd[1]: Unit late-network.service entered failed state.

● sshd.service - SSH Daemon
   Loaded: loaded (/lib/systemd/system/sshd.service; static)
   Active: failed (Result: exit-code) since Sun 2018-07-29 18:19:55 Universal; 9s ago
  Process: 1113 ExecStart=/bin/mcp_pkg_wrap rp_security /etc/init.d/sshd start (code=exited, status=127)

[  123.931317] localhost mcp_pkg_wrap[1113]: /bin/mcp_pkg_wrap: line 122: /tmp/sw/rp/0/0/rp_security/mount/etc/init.d/sshd: No such file or directory
[  123.933982] localhost systemd[1]: sshd.service: control process exited, code=exited status=127
[  123.935175] localhost systemd[1]: Failed to start SSH Daemon.
[  123.936180] localhost systemd[1]: Unit sshd.service entered failed state.

● tdl.service - TDL Resolve
   Loaded: loaded (/lib/systemd/system/tdl.service; static)
   Active: failed (Result: exit-code) since Sun 2018-07-29 18:20:04 Universal; 69ms ago
  Process: 1736 ExecStart=/usr/binos/conf/tdl_boottime.sh (code=exited, status=1/FAILURE)
Main PID: 1736 (code=exited, status=1/FAILURE)

[  132.996508] RP_0 tdl_boottime.sh[1736]: luajit: /usr/binos/conf/epoch_lib.lua:788: attempt to compare two nil values
[  132.997578] RP_0 tdl_boottime.sh[1736]: stack traceback:
[  132.998608] RP_0 tdl_boottime.sh[1736]: /usr/binos/conf/epoch_lib.lua:788: in function 'add_and_merge_record'
[  132.999626] RP_0 tdl_boottime.sh[1736]: /usr/binos/conf/epoch_lib.lua:806: in function 'generate_merged_metadata'
[  133.000634] RP_0 tdl_boottime.sh[1736]: /usr/binos/conf/epoch_lib.lua:896: in function 'generate_domain'
[  133.001652] RP_0 tdl_boottime.sh[1736]: /usr/binos/conf/epoch_resolve.lua:206: in main chunk
[  133.002701] RP_0 tdl_boottime.sh[1736]: [C]: at 0x100029a0
[  133.007577] RP_0 tdl_boottime.sh[1736]: Time to check parameters: 0.01 seconds.
[  133.008642] RP_0 tdl_boottime.sh[1736]: Time to build package table: 0 seconds.
[  133.009654] RP_0 tdl_boottime.sh[1736]: Time to cache objects: 3.74 seconds.
[  133.015651] RP_0 systemd[1]: tdl.service: main process exited, code=exited, status=1/FAILURE
[  133.017178] RP_0 systemd[1]: Failed to start TDL Resolve.
[  133.024600] RP_0 systemd[1]: Unit tdl.service entered failed state.
>>> Rebooting
%IOSXEBOOT-5c8e9d6656e9d89a8dedeae457871084-new_cksum: (rp/0): 4
%IOSXEBOOT-5c8e9d6656e9d89a8dedeae457871084-saved_cksum: (rp/0): 4
octeon_wdt: WDT device closed unexpectedly.  WDT will not stop!
reboot: Restarting system


I had to download a normal 3.x.x IOS and try to use the emergency-install feature in ROMMON
I ran into a few issues as the switch mentions that there was no file in my usbfile and then I realise I was missing a "/"


I just noticed that I was missing a “/” with emergency install – retrying

switch: emergency-install usbflash0:/cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
Unknown cmd: emergency-ins/cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin

switch: emergency-install usbflash0:/cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery (usbflash0:/cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin)...
Reading full image into memory......................done
Bundle Image


This recovery completed successfully and also deleted all unnecessary files off the flash!

ST27

Burger addict that can't say no to fries. Weighted in at 105kg once upon a time and love Japan a lot.

2 comments: