StackWise Virtual Configuration on Catalyst 9500-48Y4C

43 thoughts on “StackWise Virtual Configuration on Catalyst 9500-48Y4C”

1. Hello
   This is simple but how to do what one node is crased and cisco send you new node wtih clean configuration 🙂 Can I configure new node and add to exist virtual stack without dropout?

   Reply

   • Correct. In a StackWise Virtual configuration, if your crashed switch was Switch 2, you can preconfigure your new replacement switch as Switch 2 to prevent it from rebooting when it realizes that there’s a Switch 1 already and then reboot to convert to Switch 2. The switch that remained active (the switch that didn’t crash) should add the new switch to the stack.

     Reply

2. Hi Alirio,
   Appreciate your hardwork and time to create this important post. Currently I am working on 9500 (16 units at different MDF locations). I am confused in your one sentence regarding DAD Link. If you please explain it a bit more.

   !
   “When all member links of the SVL link go down, the standby switch will know that the active switch is up.”

   Secondly, How can I check on the switch, which of the ports are associated to Core0 and which ports to Core1.

   Thank You in Advance
   Regards
   Syed

   Reply

   • Hey Syed. What I meant with that statement is that even when the SVL goes down (both ports), since the standby switch has a DAD link with the primary switch, the standby switch won’t become active.

     You can verify all these ASIC-to-physical port mappings with the command show platform software fed switch 1 ifm mappings and see the columns Asic and Core.
     And for switch 2: show platform software fed switch 2 ifm mappings

     Let me know if you have any other questions.

     Reply

3. Hello, great post.
   A couple of questions about DAD:
   1- If I have already a stackwise virtual working and in production, If I add a DAD link (fast hello), does this impact the working stack? I mean I could add that and there will be no problem in the network…
   2- The DAD link could be 1Gbps in speed right? I am working with models C9500-32C and C9500-48Y4C .

   Thank you in advance!
   Regards

   Reply

   • Hi Lucas. Thanks for your questions.

     1. I’m not 100% sure, but I’m almost positive there won’t be any downtime if you configure DAD on a production switch; nevertheless, you should apply those changes during a maintenance window.
     2. Yes, I normally create 1 Gbps DADs.

     Reply

   • Lucas, you need to reload the whole stack.

     Cisco-9500(config-if-range)#stackwise-virtual dual-active-detection
     WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/3 on reboot.
     INFO: Upon reboot, the config will be part of running config but not part of start up config.

     Reply

4. Dear Alirio Zavarce,
   Thank you for this great article about this subject, really explicit, I really like it.
   Question: May you please tell me the type of SFP to use to link the switches by the interfaces hu1/0/49,hu1/0/51?
   I’m new in configuring core switches and I don’t really know which one to use since these ports are larger than normal.

   Thanks for your answer.
   PS: I’m using the same model you use to make the tutorial.

   Widmael

   Reply

   • Hey Widmael. Those are QSFPs (40 or 100 Gbps) interfaces.

     Reply

     • Thank you very much. It works for me!!!
       I bought 2 Cisco QSFP-100G-CU2M Compatible 100G QSFP28 Passive Direct Attach Copper Twinax Cable #65898
       and a fiber connection between TwentyFiveGigE1/0/48
       and TwentyFiveGigE2/0/48

       Reply

       • Awesome. I’m glad you’re enjoying the beauty of StackWise Virtual! Now you can take advantage of one control plane.

         Reply

5. Good day Alirio, is good practice to configure 3 25G links on SVL and 2 10g links in dad?. Thank you.

   Reply

   • Hi Ernesto. The StackWise Virtual Link should have 2 or more links. More than 2 gives you more redundancy. The last think you want is your SVL going down. Three 25G links for SVL are good. As far as DAD, 1G interfaces are more adequate since 10G SFPs are more expensive for just a simple Keepalive; however, if 10G SFPs is all you have for DAD, that’s ok too.

     Reply

6. Thank you very much Alirio. Best regards.

   Reply

7. Hi Alirio,
   I am sitting with a unique scenario where I have to remove an interface from the SVL that consist of 3 Interfaces. I cant find any information if a reboot is required of the Virtual-Stack when removing an interface from the SVL. From what I can gather it does look like a reboot is required – but I am in a situation where i don’t have an environment to test this in

   thanks

   Reply

   • I’m not 100% sure, though, but I don’t think you need a reboot to remove an interface from the StackWise Virtual Link of 3 links. The SVL stays up with 2 links, so the SVL is still in compliance. I found this statement in a StackWise document: “To associate an interface with configured StackWise Virtual link, use the stackwise-virtual link command in the interface configuration mode. To disassociate the interface, use the no form of the command.” No mention of a reboot.

     I think one of the switches reboots when the SVL goes down and comes back. Don’t remember exactly.

     Reply

8. i have a pair of c9500 running as a stackwise virtual switch. connected to this svs is a bunch of c9300 stacks in our SERs, each using 2 25gb uplinks.

   due to some miscommunication during the ordering i received 40gb multimode gbics to build this svs instead of the single mode ones i needed since our MER 1&2 only have single mode fibers between them.

   waiting for the correct ones was no option due to shortages, so the SVS was setup using 25gb interfaces for the SVL (2x25gb) and DAD (1x25gb) connections.

   after a long wait the correct 40gb optics have arrived, so it is upgrade time.
   but how to do this with minimal impact ? simply adding them wont work due to different interface speeds.

   My first thought was tot simply disable all 25gb downlinks to the SERs on the 2nd c9500, then break the whole SVS setup while our SERs keep operational using their 1 remaining 25gb to the 1st c9500. then switch the fiber routes to the 40gb interfaces (the 100gb optics where sadly out of budget at the time) and rebuild the SVS.

   i am not sure if it is enough to just remove the SVL config using 25gb interface and recreate it using 40gb interfaces without touching the rest of the SVS settings. do i need the SVL setting on both switches as well ?

   (also, will the 1st n9500 continue working for the SERs traffic or will this be interrupted ?)

   later on i realized that the configuration uses a numbered SVL setup :

   stackwise-virtual link 1 <- does the 1 mean i can create and add a second SVL set using the 40gb interfaces and remove the 1st ?

   basically.. how do i upgrade my SVL's to higher speed interface while causing the least amount of degraded speed or shortest traffic interruptions ??

   Reply

   • Hey RO1RTD, thanks for your question. Interesting StackWise Virtual case. I don’t know the right answer to your question, but after reading your questions and thinking about it, I’d try this:

     1. Shut down on 25G SFPs on Core-2 going to your SERs (I’m guessing these are you IDFs or access switches).
     2. Leave the StackWise Virtual Link and the Dual-Active-Detection on.
     3. Create a second SVL with the 40G QSFPs using the interface range command and the stackwise-virtual link 2 interface command. I don’t know whether this is possible! If you see both SVLs when running the show stackwise-virtual link command, we should be in good shape. The interface range command should include Core-1 and Core-2 ports so you form the second SVL in one shot.
     4. Disable the first SVL and then run the show stackwise-virtual link command again.
     5. Reenable one SER and see whether the port channel forms.

     Again, I’ve never done this so I don’t know the result, but you should move cautiously and step by step. You should do this during a maintenance window.

     If this doesn’t work because one SVL can exist at a time, you’ll probably have to rebuild the SVL. I don’t think that’ll be a problem because you have the DAD, which will know that the peer switch is up and running even though the SVL went down. Create a script on Notepad with all your commands beforehand so all you have to do is copy and paste.

     I hope this helps. Let me know how it went. If I spot a pair of Catalyst 9500s, I’ll do the test for you.

     Reply

     • CORESVS1(config-if)#int hu 1/0/50
       CORESVS1(config-if)#stackwise-virtual link ?
       Stackwise Virtual link number

       🙁 A secondary SVL group is not accepted but the switch.

       Reply

9. CORESVS1(config-if)#int hu 1/0/50
   CORESVS1(config-if)#stackwise-virtual link ?
   Stackwise Virtual link number

   🙁 A secondary SVL group is not accepted by the switch. Group 1 is the only option available.

   Reply

   • Hi Ruud. Sorry I didn’t get back to you sooner. Yesterday, I was migrating an IDF to a 10-slot Catalyst 9410.

     Hmmm…so that’s unfortunate. I guess the only option is to bring down the StackWise Virtual Link by takin the old SFPs and installing the new QSFPs. The Dual Active Detection link will know that both switches are up; however, to prevent a dual active scenario, the previously active switch will enter in recovery mode and the previously standby switch will become the new active switch.

     Check this out from the Cisco Catalyst 9000 Platform StackWise Virtual White Paper.

     “Recovery from a dual-active scenario

     If a StackWise Virtual link flap occurs, the system currently does not recover automatically. Upon a link-up event from any of the StackWise Virtual links, the previous active switch that is now in recovery mode has to be manually reloaded, allowing it to initialize as the hot-standby switch. From 16.11.1 release, the system does not require a manual reload and will recover automatically.

     …After the chassis reloads, it reinitializes and the switch enters into SV standby mode.

     Finally, traffic convergence associated with a StackWise Virtual link failure scenario involves the dual-active detection mechanisms, the recovery mode operations, and the restoration period, which involves a reload of the previously active StackWise Virtual switch. In each of the three previous stages, there is a possibility for some traffic disruption. Typically, the disruption will be similar to the NSF/SSO switchover scenarios described previously, but other factors from the overall network design can have an impact as well. These include the types of connections to the StackWise Virtual link, either Layer 2 or Layer 3 multichassis EtherChannels or equal-cost paths.”

     https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/nb-06-cat-9k-stack-wp-cte-en.html#Highavailability

     Reply

10. Alirio given the switch pair has an active / standby structure with SV, what is best practice for load sharing? Previously we have implemented costing to force STP to prefer one switch for half the VLANs and vice versa.

    Reply

    • Hi there. The active-standby functionality applies to the control plane (one supervisor on one switch is active while the other switch’s supervisor is in standby). The data plane, however, is active on both physical switches. Remember that both switches appear as one logical switch so there’s no need to use STP for load balancing VLANs.

      You should really implement port channels to avoid using Spanning Tree. So, if you have a stack of switches connecting to the StackWise Virtual switch with 1G or 10G links, those two links should be configured as a port channel to bundle them into one logical port. The load sharing of frames between the two links will be implemented based on the load balancing algorithm on both ends (access switches and StackWise Virtual 9500 cores). Enable port channel load balancing based on source and destination Mac and IP address AND Layer 4 ports if possible (some platforms won’t support all those header fields.)

      I hope I was clear. Let me know if you have any more questions.

      Reply

11. Estimado Alirio, buenos días, disculpe mi ignorancia, el enlace del DAD, ¿también debe tener conectado un cable?

    Reply

    • Hola Carlos. Si. Al menos un puerto en cada switch configurado como DAD deber tener un cable que conecte a los dos puertos.

      Reply

12. Hi Alirio,

    Just want to ask if I can use 25Gig ports to configure SVL on my C9500-48Y4C ?

    Reply

    • Yes, you should be able to. I have an SVL up with 25G links on a pair of C9500 24Y4C switches.

      Reply

13. Hi Alirio,

    I am wondering if you have procedure to create SVL through DNAC?
    aapreciate your help!

    Thanks,

    BR\Syed

    Reply

    • Hi Syed. Unfortunately, I don’t have a step-by-step process to create an SVL with DNA Center. Thanks for asking, though.

      Reply

14. Hi Alirio, great post so far!

    This helps me a lot to understand the VSS scenario. But I have a question. Is there any cisco documentation where they recommend assigning an interface to each core?

    Reply

    • Thank you, Ernesto. Did you mean assigning an interface from an access layer switch to each physical switch in the Virtual Stack?

      Reply

      • Hello Alirio, thank you for your quick response.

        Sorry for my question. I meant, is there any CISCO documentation where they recommend assigning each SVL interface to each CPU CORE (CORE0/CORE1) as you did (setting port 49 (assigned to ASIC0/Core1) and port 51 (assigned to ASIC0/ Core0).)?

        Greetings.

        Reply

        • For SVLs specifically, I haven’t found in the Cisco documentation a recommendation about distributing SVL links across ASICs. It’s always been a good practice to distribute redundant connections among ASICs because ASICs break.

          This is my own take on SVLs as it is common sense to me. If an ASIC Core breaks, the other ASIC Core remains up as well as the second SVL link connected to it. If an SVL link goes down as a result of one of the ASCI Cores going down, your Virtual Stack will remain up.

          If you put both SVL links on one ASIC Core and that ASIC Core breaks, your SVL will brake. It’s just a matter of minimizing the chance all of your SVL links going down. Distributing the SVL links among all ASICs is a better redundant design.

          Reply

15. Can we use mgmt port for DAD connection?

    Reply

    • I don’t know for sure, but I would guess you can’t. The best way to find out is by going on the management interface and configuring it as the DAD interface with the “stackwise-virtual dual-active-detection” command.

      Reply

16. According to Cisco doc:
    A Cisco StackWise Virtual active switch that detects a dual-active condition shuts down all of its non-SVL or non-DAD interfaces to remove itself from the network. The switch then waits in recovery mode until the SVLs recover. You should physically repair the SVL failure and the switch automatically reloads and restores itself as the standby switch. To enable the switch to remain in recovery mode after restoring the SVL links, see Disabling Recovery Reload section.

    My question is: Does this switch become the active one with the prioirty 15 config or does it always remain standby?

    Reply

    • I don’t think having priority 15 makes you the active switch preemptively. If the switch with priority 15 shuts down and comes back, that switch doesn’t become the master again since the process is not preemptive. The priority matters when, let’s say, there’s an election going on because both switches come up at the same time.

      Reply

17. Hi Alirio,
    Do you have any info on Cisco having the ability to allow more than two C9500-48Y4C devices to be used in in StackWise Virtual?

    Reply

    • Hi Faruk. You can only have up to two switches in StackWise Virtual. A Cisco employee confirms this here: https://community.cisco.com/t5/switching/stackwise-virtual-on-3-x-c9500-40x/td-p/4114336

      Reply

18. would you happen to know how are the cores distributed on a C9500-40X I think ports 1 – 16 are on core 0 and 17 – 32 on core 1 but how about ports 33- 40 ?
    and when you install the 40GB modules to which core are they mapped?

    Reply

    • Hey Luis. Sorry for the delay.

      Port mapping
      * Ports 1 to 12 are mapped to ASIC1/Core1 and ports 13 to 24 are mapped to ASIC1/Core0.
      * Ports 25 to 36 are mapped to ASIC0/Core1 and ports 37 to 40 are mapped to ASIC0/Core0.
      * Uplink ports 41/42 or 1 to 8 are mapped to ASIC0/Core0.

      Check this link below and scroll down to the C9500-40X:
      https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9500-series-switches/nb-06-cat9500-architecture-cte-en.html#Frontpanelinterfaces

      Reply

19. Hello Alirio,

    Thank you very much for taking the time and sharing such a helpful document. The questions of fellow network engineers/architects have helped me understand scenarios and also answer a few of my queries, so that’s really helpful.

    I have one additional question regarding the DAD links.

    Between a StackWise Virtual cluster of 2 x Cisco C9500-48Y4C switches, can we provision 2 x 1GigE Fiber links into an EtherChannel and have them function as the DAD link, with two physical 1G fiber links as the member interfaces?
    If yes, is there a limitation to follow with regard to leveraging only a specific physical port # on the C9500-48Y4C switches to achieve this?

    I do not have a test environment to test this at the moment.

    Appreciate your time and attention.

    Thanks,
    Ronak Vyas

    Reply

    • Hi Ronak. No, I don’t think you can configure a port channel as a DAD link. You can, however, configure multiple interfaces for Dual Active Detection.

      interface range twe1/0/47-48,twe2/0/47-48
      description DUAL ACTIVE DETECTION
      stackwise-virtual dual-active-detection
      end

      I don’t think there’s a limitation as to what ports you can use for DAD. I normally use 1 Gbps interfaces, but if 10 Gbps is all you have, you can configure those SFPs as DAD.

      Reply