Adding new Cisco 3850 to an existing stack without outage


I've just recently added a new Cisco 3850 switch into an existing 5 switch stack during production hours. One thing I learn from past experience is that you need to make sure the license needs to match and IOS firmware needs to be of a similar build.

Output of IOS firmware in my stack:
Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
     1 32    WS-C3850-24P       03.02.03.SE       cat3k_caa-universalk9 INSTALL
     2 32    WS-C3850-24P       03.02.02.SE       cat3k_caa-universalk9 INSTALL
     3 32    WS-C3850-24P       03.02.03.SE       cat3k_caa-universalk9 INSTALL
     4 56    WS-C3850-48P       03.02.02.SE       cat3k_caa-universalk9 INSTALL
     5 32    WS-C3850-24P       03.02.02.SE       cat3k_caa-universalk9 INSTALL

The SW version aren't identical but still from the same number family version 03.02.0x.

I made sure all my switch was purchased with IPBASE license
License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

Once you have verified both IOS software and license pre-requisite are met you can start to rack and stack. From my Google search prior to adding the new switch into the stack, I have read that you can pre-provision the new switch before you even add it into the stack. Yes this is possible but not necessary.

Step 1
Physically rack your new switch into place

Step 2


Remove the stack cable from the last switch (in my case, switch 5). Remember to remove only 1 end of the stack cable from switch 5. This should be the longest cable that goes from switch 5 to switch 1.

Step 3
MOST IMPORTANT NOTE TO ADDING NEW SWITCH INTO STACK! DO NOT POWER ON THE NEW SWITCH.

Insert stack cable that you remove from switch 5 and insert it into the new switch (switch 6).

Step 4
Power on switch 6
Wait for the switch to boot and join the stack.

Output session from existing stack:
scg-wa-maddington-stc#
scg-wa-maddington-stc#
May 19 01:19:47.090: %NGWC_PLATFORM_FEP-1-FRU_PS_ACCESS: Switch 1: power supply B is not responding
May 19 01:19:55.380: %STACKMGR-1-STACK_LINK_CHANGE: STANDBY:1 stack-mgr:  Stack port 2 on switch 1 is up  (scg-wa-maddington-stc-1)
scg-wa-maddington-stc#
scg-wa-maddington-stc#
May 19 01:20:04.398: %STACKMGR-1-SWITCH_ADDED: 3 stack-mgr:  Switch 6 has been added to the stack.
May 19 01:20:04.423: %STACKMGR-1-SWITCH_ADDED: MEMBER: 5 stack-mgr:  Switch 6 has been added to the stack.
May 19 01:20:04.416: %STACKMGR-1-SWITCH_ADDED: MEMBER: 2 stack-mgr:  Switch 6 has been added to the stack.
May 19 01:20:04.443: %STACKMGR-1-SWITCH_ADDED: MEMBER: 4 stack-mgr:  Switch 6 has been added to the stack.
May 19 01:20:04.396: %STACKMGR-1-SWITCH_ADDED: STANDBY:1 stack-mgr:  Switch 6 has been added to the stack.  (scg-wa-maddington-stc-1)
scg-wa-maddington-stc#
scg-wa-maddington-stc#
May 19 01:20:14.912: %STACKMGR-1-SWITCH_READY: MEMBER: 2 stack-mgr:  Switch 6 is ready.
May 19 01:20:16.159: %STACKMGR-1-SWITCH_READY: MEMBER: 5 stack-mgr:  Switch 6 is ready.
scg-wa-maddington-stc#
scg-wa-maddington-stc#
scg-wa-maddington-stc#show switch stack-ports

Switch#   Port1     Port2    
----------------------------
1         OK        OK       
2         OK        OK       
3         OK        OK       
4         OK        OK       
5         DOWN      OK       
6         OK        DOWN     

scg-wa-maddington-stc#
May 19 01:20:18.362: %STACKMGR-1-SWITCH_READY: STANDBY:1 stack-mgr:  Switch 6 is ready.  (scg-wa-maddington-stc-1)
May 19 01:20:19.326: %STACKMGR-1-SWITCH_READY: MEMBER: 4 stack-mgr:  Switch 6 is ready.
May 19 01:20:33.570: %STACKMGR-1-SWITCH_READY: 3 stack-mgr:  Switch 6 is ready.
May 19 01:20:33.576: Starting SWITCH-ADD sequence, switch 6
May 18 18:01:10.835: %STACKMGR-1-SWITCH_READY: MEMBER: 6 stack-mgr:  Switch 1 is ready.
May 18 18:01:12.321: time correction too large 26362.728 +/- 0.000+0.029 seconds 6 sntp: MEMBER:
May 19 01:20:50.614: %PLATFORM_PM-5-UNKNOWN_REQ: Ack received for unknown request from driver process (FED). msg_type:FEC_MSG (36), Request id:2147483648
May 19 01:20:50.950: Starting SWITCH-ADD sequence, switch 6 (scg-wa-maddington-stc-1)
May 19 01:20:51.591: SWITCH-ADD sequence complete, switch 6 (scg-wa-maddington-stc-1)
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/1, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/2, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/3, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/4, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/1, changed state to down
May 19 01:20:53.938: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/2, changed state to down
May 19 01:20:53.938: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/3, changed state to down
May 19 01:20:53.949: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/4, changed state to down
May 19 01:21:50.991: SWITCH-ADD sequence complete, switch 6
May 19 01:22:05.215: %STACKMGR-1-STACK_LINK_CHANGE: MEMBER: 5 stack-mgr:  Stack port 1 on switch 5 is up
May 19 01:22:05.182: %STACKMGR-1-STACK_LINK_CHANGE: MEMBER: 6 stack-mgr:  Stack port 2 on switch 6 is up
      
scg-wa-maddington-stc#
scg-wa-maddington-stc#
scg-wa-maddington-stc#show switch stack-ports

Switch#   Port1     Port2    
----------------------------
1         OK        OK       
2         OK        OK       
3         OK        OK       
4         OK        OK       
5         OK        OK       
6         OK        OK       

scg-wa-maddington-stc#
scg-wa-maddington-stc#

Output session from switch 6 console:
Booting...Initializing and Testing RAM ++++@@@@####...################################++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done.
Memory Test Pass!

Base ethernet MAC Address: bc:67:1c:2d:10:80

Interface GE 0 link down***ERROR: PHY link is down
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 74369716
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6042d350
Kernel Size       : 0x31794f/3242319
Initramfs Address : 0x60744ca0
Initramfs Size    : 0xdbe613/14411283
Compression Format: .mzip

Bootable image at @ ram:0x6042d350
Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 380
Loading Linux kernel with entry point 0x811060f0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services
18 18:00:40 %STACKMGR-1-DISC_START: Switch 1 is starting stack discovery.

18 18:00:41 %STACKMGR-1-DISC_DONE: Switch 6 has finished stack discovery.
18 18:00:41 %STACKMGR-1-SWITCH_ADDED: Switch 6 has been added to the stack.

*******************************
User Access Verification

Username:
*******************************
May 19 01:20:50.614: %PLATFORM_PM-5-UNKNOWN_REQ: Ack received for unknown request from driver process (FED). msg_type:FEC_MSG (36), Request id:2147483648
May 19 01:20:50.950: Starting SWITCH-ADD sequence, switch 6 (scg-wa-maddington-stc-1)
May 19 01:20:51.591: SWITCH-ADD sequence complete, switch 6 (scg-wa-maddington-stc-1)
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/1, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/2, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/3, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1/4, changed state to down
May 19 01:20:53.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/1, changed state to down
May 19 01:20:53.938: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/2, changed state to down
May 19 01:20:53.938: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/3, changed state to down
May 19 01:20:53.949: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/1/4, changed state to down
May 19 01:21:50.991: SWITCH-ADD sequence complete, switch 6
May 19 01:22:05.215: %STACKMGR-1-STACK_LINK_CHANGE: MEMBER: 5 stack-mgr:  Stack port 1 on switch 5 is up
May 19 01:22:05.182: %STACKMGR-1-STACK_LINK_CHANGE: MEMBER: 6 stack-mgr:  Stack port 2 on switch 6 is up 
May 19 01:23:34.120: %NGWC_USB_CONSOLE-6-USB_INSERT: Switch 6: USB cable inserted. Console media-type USB is in effect
May 19 01:23:34.121: %NGWC_USB_CONSOLE-6-CONFIG_ENABLE: Switch 6: Console media-type changed to default
May 19 01:23:34.123: %NGWC_PLATFORM_FEP-6-FRU_PS_OIR: Switch 6: FRU power supply A inserted
May 19 01:23:34.169: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 6 stack power cable 2 inserted
May 19 01:23:34.369: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 6 stack power cable 1 inserted
May 19 01:23:34.370: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 5 stack power cable 2 inserted
May 19 01:23:34.372: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 6 stack power cable 2 removed


Step 5
Close the stack ring but inserting the stack cable from switch 5 to switch 6.
May 19 01:23:34.372: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 5 stack power protocol is up on cable 2
May 19 01:23:34.373: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 6 stack power protocol is up on cable 1

Step 6
You can now start to configure your switch ports.

ST27

Burger addict that can't say no to fries. Weighted in at 105kg once upon a time and love Japan a lot.

No comments:

Post a Comment