Chapter 6. Redundancy, Symmetry, and Load Balancing
This chapter covers the following key topics:
Redundancy, symmetry, and load balancing are crucial issues facing anyone implementing high-throughput connections to the Internet. ISPs and corporations connected to ISPs require adequate control over how traffic enters and exits their respective ASs.
Redundancy is achieved by providing multiple alternate paths for the traffic, usually by having multiple connections to one or more ASs. Symmetry means having traffic that leaves the AS from a certain exit point return through the same point. Load balancing is the capability to divide traffic optimally over multiple links. Putting these three requirements together, you can imagine how challenging it is to achieve an optimal routing solution.
No single switch exists that you can turn on that gives you all you need. On the Internet, multiple providers can control and manipulate traffic that transits any AS. Any provider along the way can direct the traffic. The art of balancing traffic depends on coordination between multiple entities.
The general design problem of how best to implement redundancy, symmetry, and load balancing is common to every network. The specific answer, however, depends on the needs and configuration of each particular network. This chapter considers the general design problem within the context of several specific network configurations. You might not see your exact network configuration in these examples, but the general issues and implementation methods they raise provide a model for your analysis and design of your own routing needs.
Before examining specific network scenarios, it is necessary to establish some basic concepts and definitions concerning redundancy.
Although corporations and providers would prefer uninterrupted connectivity, connectivity problems occur for one reason or another from time to time. Connectivity is not the responsibility of one entity. A router's connection to the Internet involves the router, the CSU/DCU, cabling, physical access line, and numerous administrators—each with influence over different parts of the connection. At any time, the connectivity can be jeopardized by human error, software errors, physical errors, or adverse unforeseen conditions (such as bad weather or power outages).
For all these reasons, redundancy is generally desirable. But finding the correct balance between redundancy and symmetry is critical. Redundancy and symmetry can be conflicting design goals—the more redundancy a network has, the more unpredictable the traffic entrance and exit points would be. If a customer has multiple connections—one to a Point Of Presence (POP) in San Francisco and another to a POP in NY—traffic leaving San Francisco might come back from NY. Adding a third connection to a POP in Dallas makes connectivity even more reliable, but it also makes traffic symmetry more challenging. These are the trade-offs that network administrators must consider in implementing routing.
Geographical Restrictions Pressure
In addition to the reliability motivation, companies might feel geographical pressure to implement redundancy. Many contemporary companies are national, international, or multinational in nature. For them, the autonomous system is a logical entity that spans different physical locations. A corporation with an AS that spans several geographical points can take service from a single provider or from different providers in different regions. In figure 6-1, the San Francisco office of AS1 connects to the San Francisco POP of ISP1, and the NY office connects to the NY POP of ISP2. In this environment, traffic can take a shorter path to reach a destination by traveling via the geographically adjacent POP.
Because redundancy refers to the existence of alternate routes to and from a network, this translates into an additional number of routing information that needs to be kept in the routing tables. To avoid the extra routing overhead, default routing becomes an alternate practical tool. Default can provide us with backup routes in case primary connections fail. The next section attempts to define the different aspects of default routing and how it can be applied to achieve simple routing scenarios.
 RFC 1321 The MD5 Message-Digest Algorithm
 RFC 1997 BGP Communities Attribute
Setting Default Routes
Following defaults is a powerful technique in minimizing the amount of routes a router has to learn and providing networks with redundancy in the event of failures and connectivity interruptions. Cisco calls the default path the gateway of last resort. It is important to understand how default routing works, although it makes life easier when configured correctly; life is more difficult when routing is configured incorrectly.
By definition, a default route is a route in the IP forwarding table that is used if a routing entry for a destination does not exist. In other words, a default route is a last resort in case specific route information for a destination is unknown.
Dynamically Learned Defaults
The universally known default route is usually represented by the network mask combination 0.0.0.0/0.0.0.0 (also represented as 0/0). This route can be exchanged as a dynamic advertisement between routers. Any system advertising this route will be representing itself as a gateway of last resort for other systems. Figure 6-2 illustrates such an advertisement.
Figure 6-2 Dynamic default advertisement.
Ch. 11, pp. 368-373. Dynamically Learned Defaults
Dynamic defaults (0/0) can be learned via BGP or via IGP, depending on what protocol is running between two domains. For redundancy purposes and to accommodate potential failures, you should be receiving defaults from multiple sources. In the context of BGP, the local preference can be set for the default to give a degree of preference over which default is primary and which is backup. If one default goes away, the other will take its place.
In the left instance of figure 6-2, a single router is connecting AS1 to AS2 via two connections. If AS1 chooses to accept as few routes as possible from AS2, AS1 can accept only the 0/0 default route. In this example, AS1 is learning 0/0 from two links and giving preference by setting the local preference to 100 on the primary link and 50 (or any number smaller than 100) on the backup link. This would set the gateway of last resort to 126.96.36.199.
In the multiple routers scenario (right instance of figure 6-2), the same behavior can be achieved with multiple routers as long as IBGP is running inside the AS. Local preference, which is exchanged between routers, will determine the primary and backup links.
Statically Set Defaults
It is also possible for an AS to statically set its own defaults by setting its own 0/0 route. Statically set defaults provide more control over routing behaviors because the operator has the option of defining his last resort rather than it being forced on him by some outside entity. Many operators choose to filter dynamically learned defaults to avoid situations where traffic ends up where it is not supposed to be.
Ch. 11, pp. 370-373. Statically Set Defaults
An operator can statically set the default route 0/0 to point to the following:
Figure 6-3 illustrates the first two possibilities. On the left, a router is statically pointing its own 0/0 default toward the IP address 188.8.131.52. On the right, the same router is pointing its default toward an Ethernet interface. In the latter approach, further processing is needed to figure out to whom on the segment the traffic should be sent. Such processing usually involves sending Address Resolution Protocol (ARP) packets to identify the physical address of the next hop router.
Figure 6-3 Statically set defaults.
A system can also set its default based on some network number it learns from another system. In figure 6-4, AS1 is dynamically learning route 184.108.40.206/16 from AS2. If AS1 points its default to 220.127.116.11/16, that network will automatically become the gateway of last resort. This approach uses recursive route lookup to find the IP address of the next hop gateway. In this example, the recursive lookup will determine that 18.104.22.168/16 was learned via the next hop 22.214.171.124, and traffic would be directed accordingly.
Figure 6-4 Pointing default toward a network number.
It is important for defaults to disappear dynamically if what they point to disappears. Cisco enables a statically defined default to follow the existence of the entity to which it is pointing. If the default, for example, is pointing to a network number and that network is no longer reachable (does not show in the IP routing table), the default will also disappear from the IP routing table. This behavior is needed in situations where multiple defaults exist. One default can be used as primary and others as a backup in case the primary default is no longer valid.
Default networks should be selected as far upstream (closer to the Internet) as possible so that they are more representative of the whole link toward the NAP rather than a portion. This is important if the AS you are connected to has a single connection toward the NAP. In figure 6-4, AS1 can set the default toward its provider AS2 by pointing to prefix 126.96.36.199/24 or the supernet 188.8.131.52/16. Pointing the default to 184.108.40.206/24 makes it dependent on the stability of a portion of the link (AS1 to AS2) and not the whole link (AS1 to AS3) toward the NAP. If the link between AS2 and AS3 goes down, AS1 would be still sending traffic toward AS2 rather than directing it to some other default (assuming that AS1 has other providers). A better default choice would be the supernet, 220.127.116.11/16, because its existence is more representative of the whole link toward the NAP and is no longer dependent on any intervening links.
Setting and selecting reliable defaults.
Selected default networks should not be specific subnets. A subnet that is flip-flopping might cause your default to come and go constantly. It is much better to point the default to a major aggregate or supernet that reflects the stability of a whole provider rather than a particular link.
Multiple static defaults can be used at the same time. One way to set multiple static defaults is to point to multiple networks (use aggregates if possible for stability reasons) and establish a degree of preference by using the local preference attribute. This would apply to a single router connected to the provider via multiple connections, or multiple routers running IBGP inside the AS. Both scenarios are illustrated in figure 6-5. These are similar to the scenarios you saw in figure 6-2, the only difference being that the customer is setting its own default rather than relying on the provider to send the 0/0 default route. In this example, the customer will choose 18.104.22.168/16 with the local preference of 100 via the upper link. The lower link will be used as backup in case of failure in the primary link.
Figure 6-5 Statically pointing to multiple networks defaults.
Another way of setting defaults statically involves using the Cisco distance parameter (as described in Chapter 5, "Tuning BGP Capabilities," table 5-1) to establish a degree of preference. This would work only in the case of one router connected to multiple connections because the distance parameter is not exchanged between routers.
If two static default entries are defined with different distances, the default with the lowest distance wins. If the better default goes away, the second default becomes available. If both defaults have the same distance, then traffic will be balanced between the two defaults.
Figure 6-6 illustrates the use of the distance parameter in setting multiple defaults. AS1 is connected to AS2 via two links and is setting its own defaults toward AS2. AS1 uses one link as primary by giving the static default a distance of 50, lower than the distance of 60 given to the backup link. In case of failure in the primary link, traffic will shift toward the backup.
Figure 6-6 Static defaults pointing to multiple connections.
Symmetry refers to the fact that traffic leaving the AS from an exit point comes back through the same point. This is easy to achieve if a single exit and entrance point exists. But, given the mandates of redundancy and the presence of multiple connections, traffic tends to be asymmetrical. When it is, customers and providers notice a lack of control over how traffic flows in and out of their ASs. Traffic leaving the AS from the East Coast might end up taking the "scenic route," coming back from the West Coast and traveling inside the AS multiple hops before returning to its origin.
Actually this is not as bad as it sounds, and in some situations asymmetrical traffic is acceptable depending on the overall physical topology as far as the speed of the links and the number of hops between locations. In general, customers and providers would like to see their traffic come back close to or at the same point it left the AS to minimize potential delays that could be incurred otherwise.
To accommodate symmetry, a primary link should be chosen, and a best effort should be made to enable the majority of traffic to flow on this link. Redundancy would be accommodated by enabling other links to be backup links that will be used if the primary link is problematic.
Load balancing deals with the capability to divide data traffic over multiple connections. A common misconception about balancing is that it means an equal distribution of the load. Perfectly equal distribution of traffic is elusive enough even in situations where traffic flows in a network that is under a single administration. Given the multiple players that traffic has to touch, equal distribution of the traffic is difficult to achieve in most scenarios. Load balancing tries to achieve a traffic distribution pattern that will best utilize the multiple links that are providing redundancy. To achieve this requires a good understanding of what traffic you are trying to balance, incoming or outgoing.
It is important not to think about traffic as a single entity. Traffic is two separate entities, inbound and outbound. With respect to an autonomous system, inbound traffic is received from other ASs, whereas outbound traffic is sent to other ASs.
Suppose that you are connected to two ISPs and traffic is overloading your link to ISP1. Your question should be: What traffic—inbound or outbound? Are you receiving all your traffic from ISP1, or are you sending all your traffic toward ISP1?
The patterns of inbound and outbound traffic go hand in hand with the way you advertise your routes and the way you learn routes from other ASs. Inbound traffic is affected by how the AS advertises its networks to the outside world, whereas outbound traffic is affected by the routing updates coming in from outside ASs. Make sure that you fully understand this behavior because it will be the basis of all future discussions. From now on, whenever we talk about taking steps to affect inbound traffic, we are really talking about applying attributes to outbound routing announcements because how our routes are learned by others affects how traffic is routed inbound. Similarly, whenever we talk about taking steps to affect outbound traffic, we are talking about applying attributes to inbound routing announcements, because how our network learns routes affects how outbound traffic is routed.
Figure 6-7 illustrates how inbound and outbound traffic behaves. As you can see, the path for outbound traffic to reach NetA depends on where NetA is learned from. Because NetA is received from both SF and NY, your outbound traffic toward NetA can go via SF or NY.
Figure 6-7 Inbound and outbound decisions.
On the other hand, the path for inbound traffic to reach your local networks, NetB and NetC, depends on how you advertise these networks. If you advertise NetC over the NY link only, then incoming traffic toward NetC will take the NY link. Similarly, if you advertise NetB over the SF link only, traffic toward NetB will take the SF link.
By now you recognize the general ways in which the design goals of redundancy, symmetry, and load balancing intersect with and potentially conflict with one another. How is it possible to balance traffic among multiple links and still achieve a single entrance and exit point as symmetry mandates? This becomes even harder when multiple links are spread out over multiple routers in the autonomous system. The routing attributes described in Chapter 5, "Tuning BGP Capabilities," are the tools for implementing the desired redundancy, symmetry, and load balancing. It is the responsibility of the operator to choose and configure the correct attributes and filtering to achieve the desired outcome.
This section presents specific scenarios and attempts to configure them in such a way as to optimize redundancy, symmetry, and load balancing. The scenarios are not representative of every possible network configuration, and the design solutions shown here are not the only ones possible. But the lessons they illustrate can be applied to other scenarios and will help you understand and implement better and more efficient designs.
The first scenario is a simple case followed by increasingly complex scenarios. Note that there is a fine line between a customer and provider in many cases because a provider can be the customer of another provider. The principal distinction is this: customers obtain Internet connectivity by connecting to providers, but do not themselves offer connectivity to other customers. Providers offer Internet connectivity services and can themselves be customers of other providers.
The scenarios to be considered in the following subsections are further divided depending on whether the customer is receiving minimal or no routes, partial routes, full routes, or some combination of these from the providers. In the case where the customer is accepting minimal or no routes (called default only), you can assume that the customer can still learn the 0/0 route or a couple of aggregate routes that enables him to statically set a default. Partial routing usually consists of the provider's local routes and the provider's other customers' routes. Full routing means all Internet routes in existence—about 42,000 routes in 1996. A combination of these scenarios can occur where a customer can receive a default route and partial routes from the same provider, or partial routes from one provider and full routes from another and so on.
Scenario 1: Single-Homing
Single-homed customers have sites that connect to the Internet via a single connection to a service provider. Figure 6-8 illustrates such a situation. These customers can usually be adequately served by pointing defaults toward the provider. The provider can also install static routing to reach the customer. This method is the least expensive and the most effective. The customer router does not need to learn any of the Internet routes. This substantially reduces memory usage and processing overhead. In this case, there is no issue of route symmetry because traffic has a single entrance and exit point.
Figure 6-8 Simple single-homed site situation.
Single-homed sites generally rely on a single connection to the Internet. Backup is not an issue. If the connection is lost, the customer can tolerate the outage until it is fixed. Obviously, such an arrangement would not satisfy mission-critical data communication requirements. A single-homed site with no backup access would not be appropriate for applications needing high levels of reliability.
Scenario 2: Multihoming to a Single Provider
A customer with multiple connections to the Internet via the same provider is considered to be multihomed to a single provider. For multihoming to a single provider, assume that BGP is used as a routing protocol. Although it is not necessary in all cases, it is recommended.
Default Only, One Primary, and One Backup Link
In this scenario, the customer configures default routing toward the provider and is not accepting partial or full routes. The customer can run default to both connections. In figure 6-9, the customer wants to use one link as the primary traffic conduit and the other as a backup in case the primary link goes down. (If there were more than two connections to the provider, the customer could set up multiple defaults with varying preference levels.)
Figure 6-9 Basic mulithoming/single provider scenario.
Ch. 11, pp. 373-376. Default Only, One Primary, and One Backup Link
Customer's Outbound Traffic
In the scenario of figure 6-9, where a single router is used to connect to the provider in multiple locations, multiple static defaults with different distance values can be used, as already discussed in figure 6-6. The default with the lower distance will be the primary. The 0/0 default route or few aggregate routes can also be learned dynamically from the provider to enable the customer to set the default. Local preference can be used to prefer one default over the other.
Assume in figure 6-9 that the default to NY is more preferred than the default to SF. In normal operations, the customer will use the NY link as the primary link and the SF link as a backup.
For outbound traffic, load balancing is not an option because all traffic is sent over the primary line, and the secondary is kept as backup.
Absence of load balancing is offset by the fact that the customer's router requires less memory and processing power.
Customer's Inbound Traffic
The customer can advertise its networks to the provider via BGP. The provider will have two paths to reach the customer. Which path it chooses affects the customer's inbound traffic. Usually, the provider's default behavior (assuming that all attributes are the same) is for traffic to flow back to the customer's AS depending on which of the provider's exit points it is closest to. If traffic toward the customer is closer to the NY link, then it will enter the customer's AS via NY. If it is closer to SF, then it will enter via SF.
All the previous factors are outside the customer's control. Customers who want to override these influences and control incoming traffic via one path or the other can do so by advertising their routes with different metrics. The provider will direct its traffic toward the customer based on the metric value. In figure 6-9, the customer is advertising its routes with a metric of 50 toward NY and a metric of 100 toward SF. As such, traffic toward the customer will take the NY route.
Default, Primary, and Backup Plus Partial Routing
This is the same scenario as the default, primary, and backup case except that the customer can accept partial routing from the provider. Figure 6-10 illustrates this environment. This approach gives the customer better flexibility in choosing its exit point because more routing information is provided. As previously, both inbound and outbound traffic patterns are discussed.
Figure 6-10 Multihoming/single provider scenario with partial routing.
Ch. 11, pp. 376-382. Default, Primary, and Backup Plus Partial Routing
Customer's Outbound Traffic
Consider a situation in which customer 1 is connected to the provider via two separate routers. The customer has the option of deciding which path to take for each of the partial routes it accepts from the provider. This is usually done by setting different local preference for different routes coming into the customer's AS. Local preference can be set on an AS_path or prefix basis or both. If set based on an AS_path, then the local preference will apply to all prefixes contained in a particular AS. In case routing decisions need to be made on a prefix basis, the local preference can be set based on each prefix. In figure 6-10, based on the physical location of certain ASs or prefixes, the customer can choose to forward traffic to customer 2 and customer 3 (C2 and C3) on the SF link and to C4 and C5 on the NY link. The customer can achieve this by doing the following:
When presented with multiple routes for the same destination (via external and internal BGP), the customer will prefer the C4 and C5 routes via the NY link (300 > 200). In the same manner, the customer will prefer the C2 and C3 routes via the SF link (300 > 250). For customers other than C2, C3, C4, and C5, the NY link will be preferred (250 > 200).
For all other Internet routes not known to customer 1, default will be taken in the primary backup manner. The 0/0 default route could be dynamically learned from the provider from both ends, or could be statically configured to point to one of the provider's networks (as discussed in the "Setting Default Routes" section of this chapter). Local preference could be used to prefer one default over the other. Based on the way the local preference routes for the C2, C3, C4, and C5 customers were set, all other routes including the 0/0 will be preferred via the NY link (250 > 200).
A totally different approach that doesn't require as much configuration on the customer's side is for the provider to send its metrics toward the customer. This option was discussed in the MED section of Chapter 5. If metrics coming from the provider are representative of how close or how far networks are from the entrance points to the customer networks, then the customer will be able to load balance its outbound traffic accordingly. Traffic toward C4 and C5 will go out on the NY link, and traffic toward C2 and C3 will go out on the SF link. Other traffic will flow depending on what metrics are associated with the routes learned on each link. Although this method requires less configuration, it is also less deterministic on the customer's side because its traffic trajectory is totally dependent on the provider's setup. A combination of both approaches discussed might give the best behavior.
Customer's Inbound Traffic
The customer can influence inbound traffic by advertising different metrics on different links. Some providers encourage their customers to send their internal IGP metrics as BGP metrics (also discussed in Chapter 5). This way, the provider will deliver traffic to the customer via the link closer to the destination. In the example illustrated in figure 6-10, the customer has decided to manually set the metrics to force the following behavior:
When presented with multiple routes for the same destinations, the provider will access the Z and W prefixes over the NY link (200 < 300). In the same manner, the provider will access the X and Y prefixes over the SF link (200 < 250). For all prefixes other than X, Y, W, and Z, the provider will choose the NY link (250 < 300).
Default, Primary and Backup, Full and Partial Routing
For customers multihomed to a single provider, the customer can either get full routes on all its connections to the provider, or the customer can have a combination of full routes on one link and no routes (default) or partial routes on the other links. The same techniques discussed in the preceding sections would apply here: local preference is used to control the customer's outbound traffic, and the metric is used to control the inbound traffic. Also, if internal metrics are exchanged between customer and provider, a certain level of load balancing can be achieved.
Careful! When dealing with outbound traffic, manipulating exit points for specific routes is dangerous. Routing loops can occur if outbound traffic following an IGP default toward the customer's BGP router gets directed toward another router following default to the BGP router. This situation might seem confusing now, but will become more clear in the next chapter.
Automatic Load Balancing
As is probably clear from the previous scenarios, load balancing is not a very intuitive task and requires extensive planning. To help, Cisco IOS software supports dynamic load balancing for identical destinations learned via EBGP by the same router and coming from the same autonomous system. This will reduce configuration efforts.
Ch. 11, pp. 382-385. Automatic Load Balancing
Figure 6-11 illustrates an example in which the same router (NY) is connected to its provider via two links and is getting identical routing on both links. A Cisco router will keep in its IP routing table up to six identical BGP routes to the same destination. When passing on the EBGP updates to the IBGP peers, however, the router will only pass on one best route. The next hop address of the route will automatically be changed to reflect the router's (NY) own IP address instead of having the EBGP next hop address carried into IBGP. Note that this is done automatically only in the case where load balancing is configured dynamically.
Figure 6-11 Router receiving identical routes from two sources.
By default, a Cisco router will load balance on a per destination (Host) basis. Balancing on a destination basis is done in round-robin fashion. One host will be locked to one path, the next host will be locked to the other path, and so on.
Figure 6-11 assumes that the customer is getting two identical routes to network 22.214.171.124/24. Without automatic load balancing, the BGP process prefers one path only. It is up to the administrator to try to affect the BGP decision by changing attributes to balance the traffic between paths.
With automatic balancing, BGP will keep two entries for the 126.96.36.199/24 prefix, one via the SF link and one via the NY link. Outbound traffic from the customer network will then be split over the two links on a round-robin basis, assuming that the customer needs to send traffic to the destinations 188.8.131.52 to 184.108.40.206. Destination 10.1 will be reached via the SF link, destination 10.2 will go over the NY link, destination 10.3 will go over the SF link, and so on.
Load balancing in this manner works only when dealing with identical routing updates coming into the same router from the same provider. This method does not work to load balance in a multiprovider environment.
In the example illustrated in figure 6-11, automatic load balancing works well for outbound traffic. For inbound traffic, you must resort to manipulating metrics to influence the provider's decision.
Balancing Between Two Routers Sharing Multiple Paths
In some situations, two routers share multiple physical paths for backup or higher bandwidth services, as illustrated in figure 6-12.
Figure 6-12 Load balancing between two routers sharing multiple paths.
To balance traffic in this environment, one option is to implement dynamic balancing. This is simply a special situation of the previous automatic load balancing case. Dynamic load balancing, however, will result in extra overhead for the routers. Each router would receive duplicate update messages from the other router. In the case of full routing, the result would be approximately 42,000 routes arriving on each link. Instead, it is possible (and preferable) to achieve load balancing for the situation illustrated in figure 6-12 by using a static approach.
In the normal behavior, BGP keeps the best next hop for each prefix it learns. As seen in table 6-1, RTA will receive two identical BGP routes for NetX. BGP will pick the best route and install it in its IP routing table. In this case, BGP has picked the route via next hop 10.10.10.2. Table 6-2 illustrates RTA's IP routing table where the next hop 10.10.10.2 is reachable via link1. As a result of this configuration, traffic toward networks learned from RTB will be sent over link1. Hence, no load balancing is achieved.
Table 6-1 RTA's BGP table— NetX reachable via 10.10.10.2.
Destination Next Hop
NetX 10.10.10.2 (best) NetX 220.127.116.11
Table 6-2 RTA's IP routing table —NetX reachable via Link1.
Destination Next Hop
NetX 10.10.10.2 10.10.10.0/24 Link1
Ch. 11, pp. 385-387. Balancing Between Two Routers Sharing Multiple Paths
BGP can be fooled by setting the next hop to a virtual interface rather than the physical link and by using the IP routing table to do the actual load balancing.In figure 6-13, RTB can be assigned a loopback interface (virtual interface), and RTA can use that address to set up the BGP neighbor connection. This way, the loopback interface itself and not the IP address of the physical link will be used as a next hop. Some dynamic IGP or static routing can be used to load balance between the links independent of BGP.
Figure 6-13 A single BGP session across multiple physical links.
As seen in table 6-3, RTA will receive its BGP routes from its neighbor 18.104.22.168 and will be able to reach NetX via the next hop 22.214.171.124. Table 6-4 illustrates RTA's IP routing table. Next hop 126.96.36.199 can be reached via link1 and link2. Reachability of the 188.8.131.52/24 network can be achieved via IGP or by pointing multiple static routes toward link1 and link2. The router can now load balance the traffic. Due to the recursive route lookup in this scenario, load balancing is done per network rather than per destinations. Networks learned from RTB can now be reached round robin over multiple links.
Table 6-3 RTA's BGP table— NetX reachable via 184.108.40.206.
Destination Next Hop
Table 6-4 RTA's IP routing table — NetX eachable via Link1 or Link2.
Destination Next Hop
NetX 220.127.116.11 18.104.22.168/24 Link1 22.214.171.124/24 Link2
Scenario 3: Multihoming to Different Providers
A customer connected to multiple providers is considered to be multihomed to different providers. Redundancy and geographical restrictions are strong motivations for multihoming. The outbound traffic behavior for each iteration of this scenario will be considered on a case-by-case basis. For all cases, the inbound traffic behavior is the same and is covered at the end of the section.
Default Only, Primary and Backup
In this case, the customer can follow defaults toward the provider. One link will be used as primary, and the second link as backup. Figure 6-14 illustrates a relevant situation.
Figure 6-14 Multihoming to two providers.
A customer can set the default routes to the two providers statically or can dynamically learn 0/0 from both providers. The customer can prefer one default over another by using the "distance" or local preference. One good method of pointing defaults to both providers is to accept the same network from both providers. The customer will configure its 0/0 default based on that network and can manipulate local preference to choose one link over the other. In case one default goes away because of a link failure toward one provider, the other default will take its place. The customer can either negotiate with the providers to send him only the one network entry, or the customer can filter all updates on his end except for the one entry.
In Figure 6-14, the customer is pointing the default toward the prefix 126.96.36.199/16 it is receiving from both providers and setting the local preference on the NY link to be higher (200). The NY link will be the primary link, and the SF link will be the backup.
Default, Primary, and Backup Plus Partial Routing
The addition of partial routing to the environment introduced in the previous discussion changes the traffic behavior. Figure 6-15 illustrates the new situation. The customer can accept partial routing from one or both providers and run default toward both providers with one default preferred over the other.
Figure 6-15 Multimihoming to two providers plus partial routing.
By accepting partial routing from the providers, a customer does not need to see all Internet routes and can still make a best route decision when routing toward its direct providers. (For some major providers, partial routes could represent a substantial number of routes.) In the case illustrated in figure 6-15, BGP will make the right choice, and the customer will choose the provider link closest to the destination network (shorter AS_path). For other Internet routes, the basic principal of primary and backup can be used. The customer can point to a specific network to be the default, accept that network from both providers, and use local preference to prefer one link over the other.
Default, Primary and Backup, Full and Partial Routing
In multihoming to different providers, accepting full routes from both or either providers is not really necessary unless the customer plans to be a provider itself and pass along full routes to its customers (act as a transit AS). Figure 6-16 illustrates a relevant environment.
Figure 6-16 Multimihoming to two providers with full and partial routing.
Ch. 11, pp. 387-392. Multihoming to Different Providers
The customer can accept full routing from one or both providers depending on how much load balancing he wants to do. In the case of full routing from both (or multiple) providers, the customer can use local preference to decide which networks can be accessed via which provider. Decisions can be made based on AS or prefix information. In some cases, the customer might want to accept full routing from one provider and just do partial/default routing with the other provider. This way, the customer can get the best of both worlds without having to deal with managing full routes from different links. As you will see later, Internet instabilities caused by any provider could cause routers to become very CPU-intensive.
In figure 6-16, the customer is receiving full routes from the NY provider and partial routes from the SF provider. The customer is also pointing a default toward the SF provider. For the SF local and customer routes, the SF link will be used because of the shorter AS_path. For all other routes, the NY link will be used because the SF link is only providing partial routes. In case the SF link goes down, all networks can be reached via NY. In case the NY link goes down, the customer can still reach all Internet routes by following a default toward the SF link.
Customer Inbound Traffic (AS_Path Manipulation)
The inbound traffic is affected by how the customer advertises its networks to the providers. Note that with the multiprovider scenario, sending different metrics from the customer's end will not have any effect. This is because the MED is always terminated at the provider's network and is not carried to the other provider.
To affect the providers' behavior dynamically, the customer can manipulate the AS_path attribute by inserting bogus entries in the AS_path to affect the AS_path length. The providers will receive the same prefix information with different path length and will pick the path with the shortest length. Note that in a multiprovider environment, it is not enough to influence the direct provider only because there is no guarantee that the direct provider will get the traffic itself. Path manipulation will have to influence providers all the way up to the NAP because this is where the balance (as far as path length) will be tipped one way or the other.
Figure 6-17 illustrates how bogus entries in the AS_path affect routing. The customer (AS100) has inserted a bogus entry (100) in its AS_path toward AS300. Providers at the NAP will get the same prefixes with different path length (300 100 100 versus 200 100) and will pick the shorter path via AS200. The bogus entry should be a repeat of the AS that originated the entry (in this case 100).
Figure 6-17 Using bogus AS_path entries to affect routing.
Scenario 5: Customers of Different Providers with a Backup Link
It is not unusual for separate ASs to require Internet interconnection and to have different Internet service providers. Whenever multiple providers are involved and the customers of these providers agree to back up one another, support can get complicated. This section takes the previous discussions one step further and discusses how this backup connectivity is addressed from the provider's point of view.
Ch. 11, pp. 394-399. Customers of Different Providers with a Backup Link
In figure 6-20, AS1 is the customer of ISP1, and AS2 is the customer of ISP2. AS1 and AS2 have also entered a bilateral agreement under which the private link between the two ASs will be used as a backup in the event of a failure of either primary Internet link. Normally, an individual AS does not want to be used as transit for another AS. In the case illustrated in figure 6-20, AS1 wants ISP1 to set its routing configuration so that ISP1 reaches AS2 via ISP2. Similarly, AS2 would prefer ISP2 to set its routing configuration so that ISP2 reaches AS1 via ISP1. In this scenario, for the backup link to work, AS1 advertises AS2's networks to ISP1, and AS2 advertises AS1's networks to ISP2.
Figure 6-20 Customers of multiple providers with a backup link.
The discussions about primary and backup are the same as with the scenario discussed in the preceding section, "Scenario 4: Customers of the Same Provider with a Backup Link." The private link can be a pure backup or can be used for interior traffic between customers.
The requirement to have the provider not use one customer to reach the other customer is more complicated. ISP1 will have to set the local preference for AS2 routes coming from ISP2 to be higher than the routes coming from AS1.This would cause ISP2 to be used under normal operational conditions. The same strategy might be deployed for ISP2.
Providers, however, would like to minimize configuration on their end as much as possible. In cases where a provider has multiple customers coming online every day, tracking the local preference for each can be cumbersome. Providers would also like to set their policies based on AS numbers rather than specific networks.
A couple of approaches can be used to implement the required policies. The first approach is the community approach, which requires coordination between providers and their customers. The second approach is the AS_path manipulation approach. AS_path manipulation is easier to implement, but might not be available in all vendor products.
The Community Approach
The use of the community attribute becomes very effective. Providers want to map certain community values to corresponding local preference values. Routing updates coming from customers having a specific community will automatically be given the corresponding local preference.
Ch. 11, pp. 394-399. Customers of Different Providers with a Backup Link
To keep this scenario manageable, only routing and policy setting from ISP1's point of view is addressed. An identical discussion would apply to ISP2. Traffic flow for the case figure 6-21 illustrated in can be divided into a minimum of three patterns.
Figure 6-21 See Community approach solution.
There can be more flow patterns, depending on how many connections a customer has to its provider, but the basic set of three illustrates required considerations.
Flow patterns from ISP1's point of view can be summarized as follows:
- • Pattern 1—Routes originated by the customer AS1, or customer local routes.
- • Pattern 2—Routes transiting via AS1. These routes come from AS2 and consist of AS2's routes and all other routes that AS2 is receiving from ISP2. ISP1 uses this information to reach AS2 via AS1 as a backup in the event that AS2's link to ISP2 fails. This pattern is referred to as customer transit routes.
- • Pattern 3—All other routes coming from ISP2, or ISP routes. These can include routes learned from AS2.
Having divided the routes into different categories, ISP1 will assign a community value to each pattern and will dynamically map it to the local preference, as listed in table 6-5.
Table 6-5 Dynamic mapping of local preference.
Pattern Community Local Preference
Customer local routes none 100 Customer transit routes 400:40 40 ISP routes 400:60 60
ISP1 will inform all its customers and connected ISPs that its local preference values are dynamically set according to Table 6-5. Customers can then dynamically influence the ISP's decision by sending the corresponding community values. In the example illustrated in figure 6-21, AS1 will send its local routes with no community and the transit routes with community 400:40. ISP2 will send its routes with community 400:60.
According to the preferences summarized in table 6-5, ISP1 prefers AS1's local routes via its direct link to AS1 (preference 100 is the highest). ISP1 prefers all other routes, including AS2 routes, via ISP2 (preference 60 is higher than 40.)
The AS_Path Approach
The AS_path manipulation approach is the same as was discussed for multihoming to different providers, under "Customer Inbound Traffic (AS_Path Manipulation)." It is straightforward and has proven to be one of the the most efficient methods of influencing a provider's routing decisions. Figure 6-22 illustrates an environment in which AS_path manipulation is used to direct routing processes.
Figure 6-22 AS_path manipulation example.
Ch. 11, pp. 398-399. The AS_Path Approach
For the case illustrated in figure 6-22>, assume that all local preference attributes are kept at their default values to avoid overriding the AS_path attribute. With this assumption in mind, ISP1 will use the direct link to AS1 for AS1's local traffic and the direct link to ISP2 to reach ISP2's traffic. This is done based on the shorter AS_path.
For traffic going to AS2, ISP1 has an equal path via ISP2 and AS1. ISP1's AS_path to AS2 via AS1 is 1 2 and the AS_path via ISP2 is 500 2, which are of equivalent length.
To influence ISP1's decision, AS1 must increase the AS_path length when advertising AS2's routes to ISP1 by prepending an additional AS number to the AS_path list. Normally, AS1 will repeat its own AS number. ISP1's new AS_path to reach AS2 via AS1 will be 1 1 2, which is longer than ISP1's AS_path to reach AS2 via ISP2 500 2. As a result, ISP1 will use ISP2 to reach AS2.
Mastering routing at the edges of your domain gives you full control over traffic in and out of your autonomous system. Still, another piece of the puzzle is how the traffic flows inside the AS before it gets out. Not all routers inside the AS run BGP. IGP-only routers usually do not carry a full list of Internet routes due to memory constraints. Running defaults inside the AS to reach external routes is one of the most common ways for internal routers to reach destinations outside the AS. With defaults comes the threat of routing loops if conflicting policies exist between your BGP and your IGP. The following chapter discusses these issues of how to make BGP policies flow hand-in-hand with IGP defaults. The chapter also discusses the use of policy routing in achieving total control over routing behaviors based on the sources of IP addresses rather than the traditional destination-based routing.
Q—I statically defined a default toward my provider by pointing toward a network I am learning via BGP. What happens if that network goes up and down?
A—Your default will appear and disappear. That is why you should not point your default to a specific subnet. Always point to an aggregate or supernet because they are less likely to flip-flop.
Q—I have the option of getting the 0/0 default via BGP or defining a static default. What do you think is best?
A—For the border router, both methods are the same as long as the aggregate you are pointing to is stable. On the other hand, after you receive the 0/0 via BGP, it will get flooded to all your IBGP peers and there is a chance that you will end up sending it out to your other EBGP peers. When you define the default statically, you will have better control.
Q—I need to have a primary link where all my traffic flows and a backup link in case of failure. I also need to load balance my traffic. Is that possible?
A—That is not possible. If you are using your primary link for all inbound and outbound traffic, this would dictate that no other traffic will flow on the other link. These are two contradicting requirements.
Q—My AS is connected to two providers, one in SF and one in NY. I want the traffic from and toward my SJ site to go in and out on the SF link. All other traffic should flow over the NY link. What do I need to do to achieve this behavior?
A—For your inbound traffic toward San Jose, you can use the AS_path manipulation technique to make your path longer for all SJ routes advertised on the NY link. The problem is with your outbound traffic. If you know exactly what networks the SJ users are trying to reach, you can give those destinations better local preference on the SF exit. If the SJ site needs to reach any destination, then setting a better local preference on the SF link will cause all your outbound traffic to leave via the SF link. That doesn't meet your requirement about the NY link carrying all other traffic.
Another way of dealing with this scenario is policy routing, where a router can track source addresses and direct traffic accordingly. This is described in Chapter 7, "Controlling Routing Inside the Autonomous System."
Q—I am prepending AS numbers to my routes to tip the balance of my traffic. I am not seeing any effect. Why?
A—Remember that your updates are exchanged by multiple providers. A provider along the way can use local preference to override your path length. Check with your provider.
Q—Do I have to set BGP policies? Why can't I leave it to BGP to figure out the correct path?
A—You do not have to set policies. Remember, though, that BGP is not taking into account the speed of your links and your user traffic requirements. If you are happy with your traffic pattern the way it is, then you do not need to change any attributes.