OSPF Sham Links

If you install a backup link between sites in an MPLS VPN then you can run into problems as shown below.

This article follows on from OSPF as a PE-CE routing protocol and uses the same network layout.

I've set everything into area 0 to simplify things a bit, however there is now a serial link between the two customer sites. Topology is shown below:


Before the link is brought up the routing table on CE1 looks as follows, the important route is the 172.16.2.0 (CE2 site LAN) which is currently reached via the MPLS WAN:

CE1#show ip route

    172.16.0.0/24 is subnetted, 2 subnets
C       172.16.1.0 is directly connected, FastEthernet0/1
O IA    172.16.2.0 [110/20] via 10.0.255.2, 00:01:15, FastEthernet0/0
    10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O E2    10.254.254.52/32 [110/10] via 10.0.255.2, 00:01:10, FastEthernet0/0
O IA    10.0.255.4/30 [110/20] via 10.0.255.2, 00:01:15, FastEthernet0/0
C       10.0.255.0/30 is directly connected, FastEthernet0/0

The link is brought up using 192.168.255.0/24 and OSPF is enabled across it. As it's a backup link the cost is set accordingly.

CE1#show run int s0/0
Building configuration...

Current configuration : 142 bytes
!
interface Serial0/0
description Backup Link to Site 2
ip address 192.168.255.1 255.255.255.0
ip ospf cost 30000
clock rate 2000000

CE1#show ip route          
[snip]
    172.16.0.0/24 is subnetted, 2 subnets
C       172.16.1.0 is directly connected, FastEthernet0/1
O       172.16.2.0 [110/30010] via 192.168.255.2, 00:00:51, Serial0/0
    10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O E2    10.254.254.51/32 [110/10] via 192.168.255.2, 00:00:51, Serial0/0
O E2    10.254.254.52/32 [110/10] via 10.0.255.2, 00:00:51, FastEthernet0/0
O       10.0.255.4/30 [110/30010] via 192.168.255.2, 00:00:51, Serial0/0
C       10.0.255.0/30 is directly connected, FastEthernet0/0
C    192.168.255.0/24 is directly connected, Serial0/0

The problem is that the remote site is now being routed via the backup link, which is a 2mbit line, when there is a 100mbit ethernet link available. The reason for this is due to the administrative distance.

There are two routes being advertised to CE1 for the 172.16.2.0 network, one via iBGP and one via OSPF. The administrative distance of iBGP is 200 and OSPF is 110 so the OSPF route is chosen. The bandwidth of the link isn't compared because of the order of route selection:

1. Prefix Length.
2. Administrative Distance (configured then default).
   
3. Routing Protocol Metric.
   

The AD is compared before the metric so the routing process doesn't get as far as looking at the bandwidth of the link or the configured OSPF cost, it's already decided that OSPF is preferable to iBGP and will use it whatever.

The way around this is to use a Sham link. This is basically fudging the routing decision process by running an OSPF link across the MPLS WAN so that you have matching ADs and then can use the metric to decide which route is preferred.


The configuration is simple enough, you need a loopback with a /32 address on each PE router. That address must be advertised into the VRF across BGP. We've already got this with loopback 1 so the configuration is simply:

PE1#show run int loop1
Building configuration...

Current configuration : 96 bytes
!
interface Loopback1
 ip vrf forwarding VPN_ONE
 ip address 10.254.254.51 255.255.255.255
end
!
PE1#show run | section ospf
router ospf 100 vrf VPN_ONE
 router-id 10.254.254.51
 area 0 sham-link 10.254.254.51 10.254.254.52 cost 10

With the opposing configuration on PE2. The OSPF neighbors on PE1 then show an additional neighbor across the Sham Link:

PE1#show ip ospf 100 nei   

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.254.254.52     0   FULL/  -           -        10.254.254.52   OSPF_SL0
172.16.1.1        1   FULL/DR         00:00:34    10.0.255.1      FastEthernet0/0

PE1#show ip ospf 100 int
OSPF_SL0 is up, line protocol is up 
  Internet Address 0.0.0.0/0, Area 0 
  Process ID 100, Router ID 10.254.254.51, Network Type SHAM_LINK, Cost: 10

Now the routing table on CE1 shows that the MPLS WAN is being used for all traffic:

CE1#show ip route
[snip]
     172.16.0.0/24 is subnetted, 2 subnets
C       172.16.1.0 is directly connected, FastEthernet0/1
O       172.16.2.0 [110/40] via 10.0.255.2, 00:07:56, FastEthernet0/0
     10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O E2    10.254.254.51/32 [110/10] via 10.0.255.2, 00:07:56, FastEthernet0/0
O E2    10.254.254.52/32 [110/10] via 10.0.255.2, 00:07:56, FastEthernet0/0
O       10.0.255.4/30 [110/30] via 10.0.255.2, 00:07:56, FastEthernet0/0
C       10.0.255.0/30 is directly connected, FastEthernet0/0
C    192.168.255.0/24 is directly connected, Serial0/0