Stateful switch over and non-stop forwarding test

This test is to verify how long SSO and NSF takes to recover.

SSO without NSF

Issue redundancy force-switchover command from privilege exec mode. This command forces primary RSP to reboot so that secondary RSP takes over as primary.

The time for secondary RSP to take over as primary is 3 second, if no NSF is configured there will be an additional 12 second blackhole.

SSO with NSF
NSF operation uses CEF, during normal operation, primary RSP’s CEF synchronizes its FIB and adjacency table with secondary RSP’s CEF.

With NSF configured, there will be no blackhole after SSO is completed.

Active(Primary) RSP Syslog messages:

*Aug  7 04:37:15.215: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Aug  7 04:37:17.299: %OIR-SP-3-PWRCYCLE: Card in module 5, is being power-cycled (Switchover)
*Aug  7 04:37:18.239: %C7600_PWR-SP-4-PSOK: power supply 1 turned on.
*Aug  7 04:37:18.243: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*Aug  7 04:37:18.279: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
*Aug  7 04:37:18.279: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
*Aug  7 04:37:18.283: %OIR-SP-6-INSCARD: Card inserted in slot 2, interfaces are now online
*Aug  7 04:37:18.283: %OIR-SP-6-INSCARD: Card inserted in slot 3, interfaces are now online
*Aug  7 04:37:18.283: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
*Aug  7 04:37:20.435: %CLNS-5-NSF_RESTART: ISIS NSF completion

Configure SSO and NSF is easy
SSO is the default switchover method for RSP720, NSF however is enabled in the router configuration mode.

To enable NSF, use nsf cisco command in router configuration mode.

router isis
 net 49.0001.0192.1681.1010.00
 is-type level-1
 nsf cisco

To explicitly enable SSO as the redundancy mode use mode sso command in redundancy configuration mode.

redundancy
 main-cpu
  auto-sync running-config
 mode sso

To verify if redundancy mode is SSO
Use show redundancy states command to show the redundancy mode used.

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 8  -STANDBY HOT
           Mode = Duplex
           Unit = Secondary
        Unit ID = 6

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#

If a redundancy peer is not functional:

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 1  -DISABLED
           Mode = Simplex
           Unit = Primary
        Unit ID = 5

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = Non Redundant
     Maintenance Mode = Disabled
 Communications = Down      Reason: Simplex mode

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#

SSO Peer states

Disable state

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 1  -DISABLED
           Mode = Simplex
           Unit = Primary
        Unit ID = 5

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = Non Redundant
     Maintenance Mode = Disabled
 Communications = Down      Reason: Simplex mode

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#


*Aug  7 06:08:01.787: %FABRIC-SP-STDBY-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.

Standby-cold state

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 4  -STANDBY COLD
           Mode = Duplex
           Unit = Primary
        Unit ID = 5

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#

*Aug  7 06:09:44.983: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode

Standby cold-config state

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = in progress to standby cold-config
           Mode = Duplex
           Unit = Primary
        Unit ID = 5

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#

*Aug  7 06:10:14.455: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
*Aug  7 06:10:14.539: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby
*Aug  7 06:10:15.635: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
*Aug  7 06:10:16.915: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
*Aug  7 06:10:17.710: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online

Standby cold-filesys state

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = in progress to standby cold-filesys
           Mode = Duplex
           Unit = Secondary
        Unit ID = 6

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#
*Aug  7 06:17:00.107: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Aug  7 06:17:11.651: %C7600_PLATFORM-SP-STDBY-4-CONFREG_BREAK_ENABLED: The default factory setting for config register is 0x2102.It is advisable to retain 1 in 0x2102 as it prevents returning to ROMMON when break is issued.
*Aug  7 06:17:25.619: %SPANTREE-SP-STDBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug  7 06:17:25.831: SP-STDBY: SP: Currently running ROMMON from S (Gold) region
*Aug  7 06:17:29.519: %C7600_PLATFORM-SP-STDBY-4-CONFREG_BREAK_ENABLED: The default factory setting for config register is 0x2102.It is advisable to retain 1 in 0x2102 as it prevents returning to ROMMON when break is issued.
*Aug  7 06:17:29.915: %DIAG-SP-STDBY-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
*Aug  7 06:17:30.847: %DIAG-SP-STDBY-6-DIAG_OK: Module 5: Passed Online Diagnostics
*Aug  7 06:18:24.703: %SYS-SP-STDBY-5-RESTART: System restarted

Standby HOT state

C7609-2#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 8  -STANDBY HOT
           Mode = Duplex
           Unit = Secondary
        Unit ID = 6

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 130
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

C7609-2#
*Aug  7 06:18:42.967: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Aug  7 06:18:42.975: %RF-SP-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
C7609-2#
*Aug  7 06:18:41.907: %PFREDUN-SP-STDBY-6-STANDBY: Ready for SSO mode