Terminology:
Boot IP: The address bound to the physical interface (e.g. ifconfig blah x.x.x.x).
Service IP: The VIP to which clients connect to hit the actual service, can exist on any interface on any cluster member.
Persistent IP: Used to reach a host for management. Also called node VIP and can exist on any interface on a single cluster member.
HWAT - Hardware Address Takeover: MAC address follows the IP when failing over.
IPAT - IP Address Takeover: Moves the service IP between interfaces and cluster members.
There are two methods of doing IPAT, via replacement and by alias.
IPAT via Replacement
This is the older method, it uses HWAT so no gratuitous ARP is required as the MAC address fails over with the service IP. However clearly port security cannot be used! You need two interfaces in the same VLAN, 1 configured with a real IP address (boot IP) and 1 with any IP (standby IP) that need not be routable. When HACMP starts it replaces the real IP address on NIC 1 with a VIP in the same subnet. The failover moves both VIP and MAC onto NIC 2. You can only have 1 service VIP per adapter pair.
IPAT via Aliasing.
The newer and recommended method, it requires a network that can support gratuitous ARP as HWAT is not used. The service IP is the only routable address needed. The 2 NICs are configured with IP addresses on different subnets that need not be routable. The service VIP is an alias address on the interface and fails over as an alias. You can have as many service VIPs as you want on an interface.
Heartbeats.
The boot IPs seem fairly pointless, however network heartbeats are broadcast/multicast from the boot IP so they should be allocated from the same subnet, an example allocation is:
Node service IP 10.0.0.10
Node 1
NIC1 boot IP 192.168.0.1
NIC2 boot IP 192.168.10.1
Persistent IP 10.0.0.101
Node 2
NIC1 boot IP 192.168.0.2
NIC2 boot IP 192.168.10.2
Persistent IP 10.0.0.102
Node 3
NIC1 boot IP 192.168.0.3
NIC2 boot IP 192.168.10.3
Persistent IP 10.0.0.103
Routing.
Any routes should be configured via the service IP subnet and persistent/node IP subnet. You should not use the boot addresses as they may not always be reachable (e.g. if NIC failover). You can use the service IP to manage the system but it might not be on that node if the cluster has failed over so better to use a persistent IP. Service and Persistent IPs can be on same subnet or different ones. If different then you'll either need multiple IPs configured on the VLAN interface or static routing configured on the AIX box as they'll both be in the same VLAN. I would KISS and have both on same VLAN & same subnet.
Read more...