The result of the Cisco IP phones failover to secondary or tertiary CUCM server is instantaneous in the point of view of enduser, if everything was configured correctly. The test was done with 13 registered devices, of course as the number of registered devices increase the recovery time may vary. Only SCCP and MGCP (used for Cisco 2951 router which is a voice gateway with 4x FXO) are used, no SIP.
A cluster also known as Cisco unified CM group contains one or more CUCM, each cluster supports up to three CUCM servers. Every cluster has only one publisher and the rest is subscribers. You only need to do configuration in publisher, however you need to turn on Cisco CallManager service on every subscribers not only on publisher.
A publisher is also known as the first node, a subscriber is also known as subsequent node. Only the publisher needs to synchronize its clock with the lowest stratum NTP server, subscribers will get their time through NTP from publisher only.
A device pool is associated to a Cisco unified CM group which in turn also attached to the phones. By default Cisco unified CM disable auto-registration, however for huge deployment of many IP phones you can enable auto-registration first then disable it after all IP phones are registered.
IP phones also known as registered devices will register to the primary CUCM server based on what was configured in Cisco unified CM group. In other words, all IP phones which have the same device pool will register themselves to the same CUCM server.
In the Cisco Unified CM group, the top of the list is the primary, the subsequent item of the list is the secondary, the lowest item in the list is the tertiary.
It is possible to do call processing distribution by creating two or more Cisco unified CM groups and two device pools. In this setup, I have two CUCM servers, two device pools are created and two Cisco unified CM groups are created, each Cisco Unified CM groups have reverse orders on which is the primary CUCM server.
Turn on Cisco CallManager service on subscriber
All services are disabled by default on subscriber, services turned on on publisher is not replicated to the subscriber in order to create cluster you need to turn on Cisco CallManager service on subscriber.
To turn on the service, on subscriber go to Cisco Unified Serviceability, click on Tool > Service Activation.
Cisco Unified CM group
Create two Cisco unified CM groups, from CUCM, System > Cisco Unified CM group.
The Cisco unified CM groups are created automatically when you have provisioned the servers in System > Server from CUCM. However you need to manually add the CUCM servers in order yourself.
The publisher server is created during the installation, you need to add subscriber before installing the subscriber.
CUCM is complicated because it has too many dependencies, creating a cluster is one fine example. In order to create Cisco unified CM group, it depends on Cisco Unified CM, in order to create Cisco Unified CM it depends on Server.
Note: Many IP phones failed to failover to the secondary CUCM server is likely due to DNS issue, the DNS has no record of the CUCM hostname mapped to an IP address.
Another issue I encountered is to use hostname in System > Server despite I have no DNS. If you have no DNS servers please put in the IPv4 address of the servers instead. If you insist on using hostname despite you do not have hostname you will find the failover to be so slow that it is unacceptable for factory acceptance test, your IP phones will not register to the CUCM server as expected, in conclusion the result is erratic and the failover is slow and sometime IP phones are not registering to secondary CUCM server at all!
My setup is to create two device pools and two Cisco unified CM groups, so that some IP phones join the first cluster and some phones join the second cluster to form call processing distribution.
DHCP servers and DHCP subnets
I have heard that only the publisher has to be the TFTP and dhcp server. In this setup I have included two dhcp servers which are publisher and subscriber which I am not sure if this is correct or not.
Service parameter adjustment
From CUCM, System > Service Parameters
Station and backup server are referring to primary and secondary CUCM servers.
Apply device pool to the IP phones
From CUCM, Device > Phone, you can attach individual IP phones to device pools and they will register to the primary CUCM server as specified in the Cisco Unified CM group arrangement.
IP phones that are registered in publisher will switch back to publisher from subscriber once publisher is up again, similarly IP phones that are registered in subscriber will switch back to subscriber from the publisher once the subscriber is up again. The failover is seamless in the point of view of user.
Cluster verification with Cisco unified reporting
You can generate report in Cisco Unified reporting to check if the CUCM servers have discrepancy within the same cluster.