StackWise Virtual Configuration on Catalyst 9500-48Y4C

In this post, I will show you how to configure StackWise Virtual on a Catalyst 9500-48Y4C pair of switches and how to choose what ports to build the StackWise Virtual Link (SVL).

Cisco StackWise Virtual is a switch virtualization technology that simplifies network management by combining two physical switches into a single logical virtual switch. With one virtual switch, you can create port channels to other leaf/spoke switches without the use of Spanning Tree as a failover protocol.

StackWise Virtual & Port Channel Design

To configure Cisco’s StackWise Virtual, I’m going to give you the CLI configuration I have used to set up StackWise Virtual on a pair of Catalyst 9500-48Y4C switches and what ports to choose as part of your SVL.

Advertisement

StackWise Virtual Sample Configuration

As always, it’s good to start with a diagram to lay out what ports you’re going to use and all the connections to other switches, routers, firewalls, etc.

StackWise Virtual Design with Catalyst 9500

StackWise Virtual configuration on Switch 1:

Switch#switch priority 15
Switch#
Switch#config terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#
Switch(config)#stackwise-virtual
Please reload the switch for Stackwise Virtual configuration to take effect
Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-stackwise-virtual)# domain 101
Switch(config-stackwise-virtual)# exit
Switch(config)#
Switch(config)#interface range hu1/0/49,hu1/0/51
Switch(config-if-range)# speed 100000
Switch(config-if-range)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/49 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/51 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if-range)# exit
Switch(config)#
Switch(config)#int twe1/0/48
Switch(config-if-range)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/48 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if-range)# end
Switch#wr
Building configuration...
[OK]
Switch#
Switch#reload
Proceed with reload? [confirm]
Mar  3 15:51:46.285: %PMAN-5-TACTION: C1/0: pvp: Process mger is exiting: reload fru action r  3 15:51:46.307: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fru action requested
Mar  3 15:51:47.284: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload chassis code

<cropped>

Press RETURN to get started!


Switch>

StackWise Virtual configuration on Switch 2:

Switch#switch renumber 2
WARNING: Changing the switch number may result in a configuration change for that switch.  The interface configuration associated with the old switch number will remain as a provisioned configuration. New Switch Number will be effective after next reboot. Do you want to continue?[y/n]? [yes]: yes
Switch#
Switch#config terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#
Switch(config)#stackwise-virtual
Please reload the switch for Stackwise Virtual configuration to take effect
Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-stackwise-virtual)# domain 101
Switch(config-stackwise-virtual)# exit
Switch(config)#interface range hu1/0/49,hu1/0/51
Switch(config-if-range)# speed 100000
Switch(config-if-range)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/49 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/51 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if-range)# exit
Switch(config)#
Switch(config)#int twe1/0/48
Switch(config-if-range)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/48 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if-range)# end
Switch#
Switch#wr
Building configuration...
[OK]
Switch#
Switch#reload
Proceed with reload? [confirm]
Mar  3 15:53:29.424: %PMAN-5-TACTION: C1/0: pvger is exiting: reload fru action r  3 15:53:29.435: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fru action requested
Mar  3 15:53:30.386: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload chassis code

<cropped>

Press RETURN to get started!


Switch-stby>
Standby console disabled

Switch-stby>

Now, why would you pick ports hu1/0/49 and hu1/0/51 for the VSL?

The Catalyst 9500-48Y4C is powered by a Unified Access Data Plane (UADP) 3.0 ASIC, which offers significantly larger tables bandwidth compared to other UADP ASICs.

This UADP 3.0 ASIC holds two core CPUs: Core 1 and Core 0.

  • Ports 1-24 and 49-50 are mapped to ASIC0/Core1
  • Ports 25-48 and 51-52 are mapped to ASIC0/Core0
Catalyst 9500 48Y4C ASIC Port Mapping

So I took port 49 (mapped to ASIC0/Core1) and port 51 (mapped to ASIC0/Core0) to create the StackWise Virtual Link.

After switches reload, here are some verification commands you can use to verify:

Switch#show stackwise-virtual 
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 101

Switch  Stackwise Virtual Link  Ports
------  ----------------------  ------
1       1                       HundredGigE1/0/49           
                                HundredGigE1/0/51           
2       1                       HundredGigE2/0/49           
                                HundredGigE2/0/51           

Switch#
Switch#
Switch#
Switch#
Switch#show stackwise-virtual link
Stackwise Virtual Link(SVL) Information:
----------------------------------------
Flags:
------
Link Status
-----------
U-Up D-Down
Protocol Status
---------------
S-Suspended P-Pending E-Error T-Timeout R-Ready
-----------------------------------------------
Switch  SVL     Ports                           Link-Status     Protocol-Status
------  ---     -----                           -----------     ---------------
1       1       HundredGigE1/0/49               U               R              
                HundredGigE1/0/51               U               R              
2       1       HundredGigE2/0/49               U               R              
                HundredGigE2/0/51               U               R              

Switch#
Switch#
Switch#
Switch#show stackwise-virtual dual-active-detection
In dual-active recovery mode: No
Recovery Reload: Enabled

Dual-Active-Detection Configuration:
-------------------------------------
Switch  Dad port                        Status
------  ------------                    ---------
1       TwentyFiveGigE1/0/48            up     
2       TwentyFiveGigE2/0/48            up     

Switch#
Switch#
Switch#

So what happens if you lose the StackWise Virtual Link? Let’s see what happens and what the solution would be in the next session.

Advertisement

StackWise Virtual Dual Active Detection Sample Configuration

In the rare event of all the SVL links go down, the control-plane standby switch will become active because it can no longer communicate with its peer. At that point, you’ll have two active switches with exactly the same configuration, which includes the same IP addresses on all routed and Switched Virtual (SVI) interfaces. Duplicate IP addresses everywhere!

To mitigate this scenario, you can configure a Dual Active Detection (DAD) link. When all member links of the SVL link go down, the standby switch will know that the active switch is up.

In the configuration above, ports twe1/0/48 and twe2/0/48 were configured for dual-active detection with fast hellos; however, this is a feature that can be configured after the virtual stack is up.

interface range TwentyFiveGigE1/0/48,TwentyFiveGigE2/0/48
 stackwise-virtual dual-active-detection
 exit

Advertisement

StackWise Virtual Best Practices

I recommend the following Cisco StackWise Virtual best practices:

  • Domain ID: Use a StackWise Virtual domain ID between 1 and 255 to avoid conflict in the network. The default is domain ID 1.
  • Switch ID: Use a Switch ID between 1 and 9. I always use 1 and 2. I configure the switch on the left or on top (one switch atop the other) as Switch 1.
  • Switch Priority: The default Switch priority is 1 and the highest is 15. Cisco recommends increasing the priority to one of the switches in the stack to make it more likely to become the switch with the active role. I always configure Switch 1 with a priority of 15 to pre-determine what switch becomes active. To become active, a higher priority is preferred.
  • Stack MAC Address: Default is infinite. Cisco recommends keeping default settings.
  • StackWise Virtual Link: You should configure the SVL as a multiple-link port channel and spread it across multiple ASICs if possible as well as multi-line cards.

LOOKING FOR Certification Guides & Practice Tests?

Online Learning Platform for Network Engineers

(formerly Safari Books Online)

Final Comments

A great benefit of Cisco’s StackWise Virtual is that you can dual home all of your access layer switches and multiple other devices like firewalls, routers, etc., and provide immediate redundancy at the link and core level without the use of Spanning Tree. Everything could be dual-homed with port channels.

I hope this post was informative for you.

Comments are encouraged. Please, leave any comments below.

Thank you!

Alirio Zavarce Faceshot

ABOUT THE AUTHOR

Alirio Zavarce, CCIE #28672, is a seasoned enterprise route-switch consultant with 30 years of experience with data networks. Alirio started this networking blog to help his peers become better network engineers and share all his everyday experiences and troubleshooting tips. More about me...

If Alirio had to prepare to take it again, here's what he would do to pass the CCIE lab.

Please Share

43 thoughts on “StackWise Virtual Configuration on Catalyst 9500-48Y4C”

  1. Hello
    This is simple but how to do what one node is crased and cisco send you new node wtih clean configuration 🙂 Can I configure new node and add to exist virtual stack without dropout?

    Reply
    • Correct. In a StackWise Virtual configuration, if your crashed switch was Switch 2, you can preconfigure your new replacement switch as Switch 2 to prevent it from rebooting when it realizes that there’s a Switch 1 already and then reboot to convert to Switch 2. The switch that remained active (the switch that didn’t crash) should add the new switch to the stack.

      Reply
  2. Hi Alirio,
    Appreciate your hardwork and time to create this important post. Currently I am working on 9500 (16 units at different MDF locations). I am confused in your one sentence regarding DAD Link. If you please explain it a bit more.

    !
    “When all member links of the SVL link go down, the standby switch will know that the active switch is up.”

    Secondly, How can I check on the switch, which of the ports are associated to Core0 and which ports to Core1.

    Thank You in Advance
    Regards
    Syed

    Reply
    • Hey Syed. What I meant with that statement is that even when the SVL goes down (both ports), since the standby switch has a DAD link with the primary switch, the standby switch won’t become active.

      You can verify all these ASIC-to-physical port mappings with the command show platform software fed switch 1 ifm mappings and see the columns Asic and Core.
      And for switch 2: show platform software fed switch 2 ifm mappings

      Let me know if you have any other questions.

      Reply
  3. Hello, great post.
    A couple of questions about DAD:
    1- If I have already a stackwise virtual working and in production, If I add a DAD link (fast hello), does this impact the working stack? I mean I could add that and there will be no problem in the network…
    2- The DAD link could be 1Gbps in speed right? I am working with models C9500-32C and C9500-48Y4C .

    Thank you in advance!
    Regards

    Reply
    • Hi Lucas. Thanks for your questions.

      1. I’m not 100% sure, but I’m almost positive there won’t be any downtime if you configure DAD on a production switch; nevertheless, you should apply those changes during a maintenance window.
      2. Yes, I normally create 1 Gbps DADs.

      Reply
    • Lucas, you need to reload the whole stack.

      Cisco-9500(config-if-range)#stackwise-virtual dual-active-detection
      WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/3 on reboot.
      INFO: Upon reboot, the config will be part of running config but not part of start up config.

      Reply
  4. Dear Alirio Zavarce,
    Thank you for this great article about this subject, really explicit, I really like it.
    Question: May you please tell me the type of SFP to use to link the switches by the interfaces hu1/0/49,hu1/0/51?
    I’m new in configuring core switches and I don’t really know which one to use since these ports are larger than normal.

    Thanks for your answer.
    PS: I’m using the same model you use to make the tutorial.

    Widmael

    Reply
      • Thank you very much. It works for me!!!
        I bought 2 Cisco QSFP-100G-CU2M Compatible 100G QSFP28 Passive Direct Attach Copper Twinax Cable #65898
        and a fiber connection between TwentyFiveGigE1/0/48
        and TwentyFiveGigE2/0/48

        Reply
    • Hi Ernesto. The StackWise Virtual Link should have 2 or more links. More than 2 gives you more redundancy. The last think you want is your SVL going down. Three 25G links for SVL are good. As far as DAD, 1G interfaces are more adequate since 10G SFPs are more expensive for just a simple Keepalive; however, if 10G SFPs is all you have for DAD, that’s ok too.

      Reply
  5. Hi Alirio,
    I am sitting with a unique scenario where I have to remove an interface from the SVL that consist of 3 Interfaces. I cant find any information if a reboot is required of the Virtual-Stack when removing an interface from the SVL. From what I can gather it does look like a reboot is required – but I am in a situation where i don’t have an environment to test this in

    thanks

    Reply
    • I’m not 100% sure, though, but I don’t think you need a reboot to remove an interface from the StackWise Virtual Link of 3 links. The SVL stays up with 2 links, so the SVL is still in compliance. I found this statement in a StackWise document: “To associate an interface with configured StackWise Virtual link, use the stackwise-virtual link command in the interface configuration mode. To disassociate the interface, use the no form of the command.” No mention of a reboot.

      I think one of the switches reboots when the SVL goes down and comes back. Don’t remember exactly.

      Reply
  6. i have a pair of c9500 running as a stackwise virtual switch. connected to this svs is a bunch of c9300 stacks in our SERs, each using 2 25gb uplinks.

    due to some miscommunication during the ordering i received 40gb multimode gbics to build this svs instead of the single mode ones i needed since our MER 1&2 only have single mode fibers between them.

    waiting for the correct ones was no option due to shortages, so the SVS was setup using 25gb interfaces for the SVL (2x25gb) and DAD (1x25gb) connections.

    after a long wait the correct 40gb optics have arrived, so it is upgrade time.
    but how to do this with minimal impact ? simply adding them wont work due to different interface speeds.

    My first thought was tot simply disable all 25gb downlinks to the SERs on the 2nd c9500, then break the whole SVS setup while our SERs keep operational using their 1 remaining 25gb to the 1st c9500. then switch the fiber routes to the 40gb interfaces (the 100gb optics where sadly out of budget at the time) and rebuild the SVS.

    i am not sure if it is enough to just remove the SVL config using 25gb interface and recreate it using 40gb interfaces without touching the rest of the SVS settings. do i need the SVL setting on both switches as well ?

    (also, will the 1st n9500 continue working for the SERs traffic or will this be interrupted ?)

    later on i realized that the configuration uses a numbered SVL setup :

    stackwise-virtual link 1 <- does the 1 mean i can create and add a second SVL set using the 40gb interfaces and remove the 1st ?

    basically.. how do i upgrade my SVL's to higher speed interface while causing the least amount of degraded speed or shortest traffic interruptions ??

    Reply
    • Hey RO1RTD, thanks for your question. Interesting StackWise Virtual case. I don’t know the right answer to your question, but after reading your questions and thinking about it, I’d try this:

      1. Shut down on 25G SFPs on Core-2 going to your SERs (I’m guessing these are you IDFs or access switches).
      2. Leave the StackWise Virtual Link and the Dual-Active-Detection on.
      3. Create a second SVL with the 40G QSFPs using the interface range command and the stackwise-virtual link 2 interface command. I don’t know whether this is possible! If you see both SVLs when running the show stackwise-virtual link command, we should be in good shape. The interface range command should include Core-1 and Core-2 ports so you form the second SVL in one shot.
      4. Disable the first SVL and then run the show stackwise-virtual link command again.
      5. Reenable one SER and see whether the port channel forms.

      Again, I’ve never done this so I don’t know the result, but you should move cautiously and step by step. You should do this during a maintenance window.

      If this doesn’t work because one SVL can exist at a time, you’ll probably have to rebuild the SVL. I don’t think that’ll be a problem because you have the DAD, which will know that the peer switch is up and running even though the SVL went down. Create a script on Notepad with all your commands beforehand so all you have to do is copy and paste.

      I hope this helps. Let me know how it went. If I spot a pair of Catalyst 9500s, I’ll do the test for you.

      Reply
      • CORESVS1(config-if)#int hu 1/0/50
        CORESVS1(config-if)#stackwise-virtual link ?
        Stackwise Virtual link number

        🙁 A secondary SVL group is not accepted but the switch.

        Reply
  7. CORESVS1(config-if)#int hu 1/0/50
    CORESVS1(config-if)#stackwise-virtual link ?
    Stackwise Virtual link number

    🙁 A secondary SVL group is not accepted by the switch. Group 1 is the only option available.

    Reply
    • Hi Ruud. Sorry I didn’t get back to you sooner. Yesterday, I was migrating an IDF to a 10-slot Catalyst 9410.

      Hmmm…so that’s unfortunate. I guess the only option is to bring down the StackWise Virtual Link by takin the old SFPs and installing the new QSFPs. The Dual Active Detection link will know that both switches are up; however, to prevent a dual active scenario, the previously active switch will enter in recovery mode and the previously standby switch will become the new active switch.

      Check this out from the Cisco Catalyst 9000 Platform StackWise Virtual White Paper.

      “Recovery from a dual-active scenario

      If a StackWise Virtual link flap occurs, the system currently does not recover automatically. Upon a link-up event from any of the StackWise Virtual links, the previous active switch that is now in recovery mode has to be manually reloaded, allowing it to initialize as the hot-standby switch. From 16.11.1 release, the system does not require a manual reload and will recover automatically.

      …After the chassis reloads, it reinitializes and the switch enters into SV standby mode.

      Finally, traffic convergence associated with a StackWise Virtual link failure scenario involves the dual-active detection mechanisms, the recovery mode operations, and the restoration period, which involves a reload of the previously active StackWise Virtual switch. In each of the three previous stages, there is a possibility for some traffic disruption. Typically, the disruption will be similar to the NSF/SSO switchover scenarios described previously, but other factors from the overall network design can have an impact as well. These include the types of connections to the StackWise Virtual link, either Layer 2 or Layer 3 multichassis EtherChannels or equal-cost paths.”

      https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/nb-06-cat-9k-stack-wp-cte-en.html#Highavailability

      Reply
  8. Alirio given the switch pair has an active / standby structure with SV, what is best practice for load sharing? Previously we have implemented costing to force STP to prefer one switch for half the VLANs and vice versa.

    Reply
    • Hi there. The active-standby functionality applies to the control plane (one supervisor on one switch is active while the other switch’s supervisor is in standby). The data plane, however, is active on both physical switches. Remember that both switches appear as one logical switch so there’s no need to use STP for load balancing VLANs.

      You should really implement port channels to avoid using Spanning Tree. So, if you have a stack of switches connecting to the StackWise Virtual switch with 1G or 10G links, those two links should be configured as a port channel to bundle them into one logical port. The load sharing of frames between the two links will be implemented based on the load balancing algorithm on both ends (access switches and StackWise Virtual 9500 cores). Enable port channel load balancing based on source and destination Mac and IP address AND Layer 4 ports if possible (some platforms won’t support all those header fields.)

      I hope I was clear. Let me know if you have any more questions.

      Reply
  9. Estimado Alirio, buenos días, disculpe mi ignorancia, el enlace del DAD, ¿también debe tener conectado un cable?

    Reply
    • Hola Carlos. Si. Al menos un puerto en cada switch configurado como DAD deber tener un cable que conecte a los dos puertos.

      Reply
  10. Hi Alirio,

    I am wondering if you have procedure to create SVL through DNAC?
    aapreciate your help!

    Thanks,

    BR\Syed

    Reply
    • Hi Syed. Unfortunately, I don’t have a step-by-step process to create an SVL with DNA Center. Thanks for asking, though.

      Reply
  11. Hi Alirio, great post so far!

    This helps me a lot to understand the VSS scenario. But I have a question. Is there any cisco documentation where they recommend assigning an interface to each core?

    Reply
    • Thank you, Ernesto. Did you mean assigning an interface from an access layer switch to each physical switch in the Virtual Stack?

      Reply
      • Hello Alirio, thank you for your quick response.

        Sorry for my question. I meant, is there any CISCO documentation where they recommend assigning each SVL interface to each CPU CORE (CORE0/CORE1) as you did (setting port 49 (assigned to ASIC0/Core1) and port 51 (assigned to ASIC0/ Core0).)?

        Greetings.

        Reply
        • For SVLs specifically, I haven’t found in the Cisco documentation a recommendation about distributing SVL links across ASICs. It’s always been a good practice to distribute redundant connections among ASICs because ASICs break.

          This is my own take on SVLs as it is common sense to me. If an ASIC Core breaks, the other ASIC Core remains up as well as the second SVL link connected to it. If an SVL link goes down as a result of one of the ASCI Cores going down, your Virtual Stack will remain up.

          If you put both SVL links on one ASIC Core and that ASIC Core breaks, your SVL will brake. It’s just a matter of minimizing the chance all of your SVL links going down. Distributing the SVL links among all ASICs is a better redundant design.

          Reply
    • I don’t know for sure, but I would guess you can’t. The best way to find out is by going on the management interface and configuring it as the DAD interface with the “stackwise-virtual dual-active-detection” command.

      Reply
  12. According to Cisco doc:
    A Cisco StackWise Virtual active switch that detects a dual-active condition shuts down all of its non-SVL or non-DAD interfaces to remove itself from the network. The switch then waits in recovery mode until the SVLs recover. You should physically repair the SVL failure and the switch automatically reloads and restores itself as the standby switch. To enable the switch to remain in recovery mode after restoring the SVL links, see Disabling Recovery Reload section.

    My question is: Does this switch become the active one with the prioirty 15 config or does it always remain standby?

    Reply
    • I don’t think having priority 15 makes you the active switch preemptively. If the switch with priority 15 shuts down and comes back, that switch doesn’t become the master again since the process is not preemptive. The priority matters when, let’s say, there’s an election going on because both switches come up at the same time.

      Reply
  13. Hi Alirio,
    Do you have any info on Cisco having the ability to allow more than two C9500-48Y4C devices to be used in in StackWise Virtual?

    Reply
  14. would you happen to know how are the cores distributed on a C9500-40X I think ports 1 – 16 are on core 0 and 17 – 32 on core 1 but how about ports 33- 40 ?
    and when you install the 40GB modules to which core are they mapped?

    Reply
  15. Hello Alirio,

    Thank you very much for taking the time and sharing such a helpful document. The questions of fellow network engineers/architects have helped me understand scenarios and also answer a few of my queries, so that’s really helpful.

    I have one additional question regarding the DAD links.

    Between a StackWise Virtual cluster of 2 x Cisco C9500-48Y4C switches, can we provision 2 x 1GigE Fiber links into an EtherChannel and have them function as the DAD link, with two physical 1G fiber links as the member interfaces?
    If yes, is there a limitation to follow with regard to leveraging only a specific physical port # on the C9500-48Y4C switches to achieve this?

    I do not have a test environment to test this at the moment.

    Appreciate your time and attention.

    Thanks,
    Ronak Vyas

    Reply
    • Hi Ronak. No, I don’t think you can configure a port channel as a DAD link. You can, however, configure multiple interfaces for Dual Active Detection.

      interface range twe1/0/47-48,twe2/0/47-48
      description DUAL ACTIVE DETECTION
      stackwise-virtual dual-active-detection
      end

      I don’t think there’s a limitation as to what ports you can use for DAD. I normally use 1 Gbps interfaces, but if 10 Gbps is all you have, you can configure those SFPs as DAD.

      Reply

Leave a Comment

Share to...