Chapter 9. Designing Stable Internets
This chapter covers the following key topics:
- Route Instabilities on the Internet
A list and brief description of the most common causes of route instabilities.- Controlling Route and Cache Invalidation
Cisco offers a soft configuration feature that enables administrators to reconfigure attributes on-the-fly with minimal repercussions for routes and cache.- Route Dampening
BGP's route dampening feature identifies and suppresses unstable routes for better reliability within and outside the AS. - Controlling Route and Cache Invalidation
Establishing and maintaining route stability within and among networks is crucial to ensuring reliable Internet connectivity. A number of design flaws and problems can contribute to destabilizing connections to the Internet. This chapter explores some of the causes of route instability and techniques for reducing it.
Route Instabilities on the Internet
The central symptom of route instability is the disappearance of a route that previously existed in the routing table. This route might disappear and reappear intermittently, a condition sometimes referred to as flapping. What is happening at the routing protocol level is that BGP sends a routing update and then immediately withdraws it. A router that receives UPDATE or WITHDRAWN messages would have to propagate those messages to its peers. If this behavior continues to cascade, routing performance suffers.
Factors that affect route instabilities on the Internet include the following influences:
- Instabilities of IGPs
- Hardware Failures
- Software Problems
- Insufficient Horsepower
- Insufficient Memory
- Network Upgrades
- Human Error
- Backup Link Overloads
- Hardware Failures
Instabilities of IGPs
Dynamically injecting IGPs into BGP can cause unnecessary route flapping. Problems that occur inside a domain can translate into problems outside the domain. As already discussed in Chapter 5, "Tuning BGP Capabilities," static injection of routing into BGP can solve this problem.
Route aggregation at the border routers can also reduce the potential unpleasant side effects associated with IGP injection into BGP. With aggregation, multiple route entries get injected into BGP as a summary aggregate. A route instability in any one element of the aggregate does not affect the stability of the aggregate itself.
Still, some network designers are forced to rely on dynamic routing for valid reasons:
- BGP implementations can only handle a
fixed number of network entries to be advertised statically. The
number of static routes permitted varies from vendor to vendor.
Whatever that limit is, networks that want to go beyond this limit
require that administrators inject the IGP into BGP.
- Some administrators are not too comfortable with the fact that the networks they are statically advertising might become unreachable by the router advertising them. This is understandable, especially in cases where routes are advertised from different points of the AS. Advertising a route that is not reachable can create black holes.
Hardware Failures
Faulty interfaces, faulty systems, or faulty lines can all affect route stability. An interface that is intermittently available might cause routing information to transition. Hardware failures are, to a certain degree, beyond the control of service users. System and link redundancy are important tools for reducing connectivity loss due to failures, but when a physical failure occurs, routing will be interrupted, and any interruption will initiate some kind of cascade effect down the routing path.
Software Problems
Software problems or "bugs" can cause system failures and network instabilities. Development teams try their best to catch these problems before the software is released to customers. Nevertheless, it is almost impossible to forsee every single situation that might occur in live networks. Administrators should experiment with new software or new features in test labs and low impact portions of their network in order to get some level of confidence before the software is fully deployed.
Insufficient Horsepower
The more routing updates and peering sessions the router handles, the more CPU power is required. Think of the router as your basic 4x4 truck, and think of the routing and traffic overhead as the load you carry. Would you be surprised if the truck has trouble moving with a 20-ton load? Picking the correct system with the correct CPU power is very important to satisfy your particular routing needs.
At the initial stages of building BGP tables after the BGP sessions are established, a system's processor can spend more than 90 percent of its time processing updates. When links become unstable and overloaded, the router might end up in a race condition: the CPU is too busy handling updates, which causes BGP sessions to drop, which in turn causes more instabilities.
Insufficient Memory
In addition to the memory needed by a router to run its own operating system, a router must store routing tables, cache tables, databases, and the other bits of software to permit operation. A router that reaches its memory limit might stop functioning, which causes all routes it knows of or advertises to be lost.
In BGP terms, a routing entry consists of the entry in the IP forwarding table and whatever corresponding information is available in the BGP routing table. Today, the Internet has reached more than 42,000 routes. Systems that are taking full routes from the Internet from a couple of providers are barely keeping up with 32 MB of memory. Most providers have upgraded their systems to 64 MB and even 128 MB.
Network Upgrades
Networks are dynamic. Performance improvement, site consolidation, and support expansion all require changes and adaptations. Changes might include upgrades to new versions of software or hardware, additions of more links, additions of more bandwidth, or reconfiguration of a network's layout.
For obvious reasons, administrators prefer to bring a system down for upgrading during some period when it usually experiences minimal usage. The downtime for some networks cannot exceed more than an hour, even at night, because of time zone differences. Despite these difficulties, the upgrade period itself is not usually the time when errors are most significant because administrators always keep a backup plan and can revert to the old setup if the new setup does not work. In case of configuration or software/hardware problems, network instabilities will take effect the next day when everybody is back online. At that point, reverting to the old setup is not likely to be a viable option. Unfortunately, to rectify the situation, administrators sometimes start adding or changing configuration on-the-fly, making the situation even worse.
To reduce the likelihood of causing disruptions, network changes should be simulated if possible in nonproduction environments. In addition, multiple major changes should not be deployed at the same time. It is, for example, unwise for a provider to perform major router software upgrades, switch hardware, and change cabling, all at the same time. Good planning and network simulation is the key to successful network upgrades.
Human Error
Most of the network instabilities caused by human error occur because an administrator circumvents an administration policy or makes a change without knowledge of possible effects. It is easy to make mistakes in complex network configurations. One wrong filter, and an entire AS can be isolated. Administrators should anticipate problems before they occur.
Here's an example of the kinds of errors that can happen: any router can send the default 0.0.0.0 via BGP to its neighbors. If you are not careful, traffic will take the wrong route. As much as it is somebody else's responsibility to send appropriate default routes, it is your responsibility to protect yourself by making sure that you filter any unwanted routes, default or otherwise, that come your way. The list of possible human errors is long: someone might advertise somebody else's networks; a provider might stop advertising your networks; somebody summarizes the wrong networks. The point is, don't expect everyone else to play by your rules. Other administrators can (usually inadvertently) deploy rules that directly conflict with your rules, which can lead to serious performance and connectivity degradation.
Backup Link Overloads
In some cases, a link failure will cause a backup link to be overloaded with traffic. This occurs because the backup link is handling all the additional traffic that is now being routed its way on top of its normal traffic. Even if the link can handle the load, a router might not be able to handle the additional loaddepending on its horsepower. This can cause major performance degradation for the end-user.
In the process of trying to get a handle on network instabilities, BGP implementations have introduced several helpful features. Although these features do not provide a complete solution, they are significant preventative measures of route instability.
BGP Stability Features
Of course, developing effective routing policies and configuring them correctly is at the core of building stability. BGP's attribute selections, as discussed throughout this book, are tools for building that core stability. In addition, BGP functions that can help provide a buffer against route instability effects include:
- Controlling Route and Cache Invalidation
- Route Dampening
Controlling Route and Cache Invalidation
The basis of any BGP conversation is the TCP/IP session that takes place between two neighbors. The neighbor connection itself is based on the OPEN message, which contains parameters such as the BGP version number. In addition, exchanged routing updates carry different attributes such as the metric, communities, and AS_Path. Anytime an administrator changes attributes or policies, BGP implementations require that a BGP TCP session with its neighbor be reset (broken and restarted) for the modified routing behavior to take effect.
Troubleshooting:
Ch. 11, pp. 433-437. Controlling Route and Cache Invalidation
Unfortunately, every time the TCP session is reset, routing is interrupted. When a session is reset, the routing cache gets invalidated, routes disappear, and route instability cascades throughout the Internet. By the time the session is brought back up and routes/cache are re-established, real damage could result.
Cisco Systems introduced a mechanism called soft reconfiguration that enables administrators to reconfigure attributes on-the-fly without killing an already established TCP session. As a result, the routing cache is not cleared, and impact on the route is minimal.
Route Dampening
Another mechanism for controlling route instability is called route dampening. A route that appears and disappears intermittently causes BGP UPDATE and WITHDRAWN messages to be repeatedly propagated on the Internet. The tremendous amount of routing traffic generated can use up all the link's bandwidth and drive up CPU utilization of routers.
Dampening categorizes routes as either well-behaved or ill-behaved. A well-behaved route shows a high degree of stability during an extended period of time. On the other hand, an ill-behaved route experiences a high level of instability in a short period of time. Ill-behaved routes should be penalized in a way that is proportional to the expected future instability of the route. An unstable route should be suppressed (not advertised) until there is some degree of confidence that the route is stable.
Troubleshooting:
Ch. 11, pp. 437-442. Route Dampening
The recent history of a route is used as a basis for estimating future stability. To track a route history, it is essential to track the number of times the route has flapped over a period of time. Under route dampening, each time a route flaps, it is given a penalty. Whenever the penalty reaches a predefined threshold, the route is suppressed. The route can continue to accrue penalties even after it is suppressed. The more frequently a route oscillates in a short amount of time, the faster the route will be suppressed.
Similar criteria are put in place to unsuppress a route and start readvertising it. An algorithm is implemented to decay (reduce) the penalty exponentially. The algorithm bases its configuration on a user-defined set of parameters. The following set of terms and parameters applies to the Cisco implementation:
- PenaltyAn incremented numeric
value that is assigned to a route each time it flaps.
- Half-life-timeA configurable numeric value that describes the amount of time that must elapse to reduce the penalty by one half.
- Suppress limitA numeric value that is compared with the penalty. If the penalty is greater than the suppress limit, the route is suppressed.
- Reuse limitA configurable numeric value that is compared with the penalty. If the penalty is less than the reuse limit, a suppressed route that is up will no longer be suppressed.
- Suppressed routeA route that is not advertised, even if it is up. A route is suppressed if the penalty is more than the suppressed limit.
- History entryAn entry used to store flap information. For the purposes of monitoring and calculating the level of oscillation of a route, it is important to store this information in the router when the route oscillates. When the route stabilizes, the history entry will become useless and will have to be flushed from the router.
- Half-life-timeA configurable numeric value that describes the amount of time that must elapse to reduce the penalty by one half.
Figure 9-1 illustrates the process of assessing a penalty to a route every time it flaps. The penalty is exponentially decayed according to parameters such as the half-life-time. The half-life-time parameter can be changed by the administrator to reflect the oscillation history of a route: a longer half-life might be desirable for a route that has a habit of oscillating frequently. A larger half-life-time value would cause the penalty to decay more slowly, which translates into a route being suppressed longer.
Figure 9-1 Route
dampening penalty assessment.
Stability Inside the AS
The benefits of route dampening are noticed inside as well as outside an autonomous system. When BGP is redistributed (injected) into an IGP, it is important that BGP instability does not affect internal routing in such a way as to cause a meltdown inside the AS. This is where route dampening can be useful. Routes that are flapping will be dampened and prevented from being injected into the AS until they show some degree of stability. Figure 9-2 compares the effects of EBGP flapping on an IGP with and without route dampening.
Figure 9-2 Effects of
EBGP flapping on an IGP.
In figure 9-2, routes R1, R2, and R3 are injected from BGP into the AS. The up and down arrows next to R2 indicate that it is flapping. The routes will be carried via IBGP and/or IGP depending on how the administrator is injecting the routes into the AS. In either case, the oscillations of R2 create major overhead for the border router and also on the interior routers. IGPs will be flooding and removing the route as long as the route is instable. With route dampening, the ill-behaved route will be suppressed (after reaching the suppress limit) and will be prevented from entering the AS.
Instabilities Outside the AS
Route dampening can prevent unstable EBGP routes from being propagated to other peers. This can save on link bandwidth usage and processing overhead within border routers. If you are a provider with multiple customers using your services, it is important not to burden your own network (and the outside world) with instabilities that go on inside a customer's network. In the case where a provider advertises a customer's network as part of an aggregate, this is not an issue. The aggregate will be stable (always advertised) even if most of its elements are not. Nonetheless, within the provider's AS, a customer's instabilities are a concern. When a customer's network cannot be aggregated (due to multihoming or addresses not being part of the provider, then instabilities will be carried to the outside world.
With dampening, the provider's border router will suppress customer routes that are flapping. Suppression will take effect according to dampening rules and dampening parameters discussed earlier in this section.
One possible side effect of this is that the customer will experience some short outages even if his routes become stable. In figure 9-3, route R2 in the customer network is flapping. When the customer's ISP is running route dampening, R2 will be penalized and suppressed according to its level of oscillation. R2 could be dampened for minutes. Even if R2 stops oscillating, the penalty it had accumulated still might be far above the reuse limit, and it has to be decayed before the route can be used. In the meantime, some poor soul on the customer's network is pulling out his or her hair trying to figure out why some subnets are not reachable from the outside world. If administrators are unaware that their routes are being dampened, they might try to remedy the situation by some other means, which makes their routes flap even more and become more penalized. The better approach is to check with the provider on whether he is receiving the routes, and if he is, why they are not being advertised. Providers have strict policies and might not change the dampening behavior per the customer's request.
Figure 9-3 Route
dampening ISP environment.
What the provider can do is "flush" the history information of the routes being dampened to advertise the route. This is, of course, under the condition that the customer will investigate the routing problems causing the routes to fluctuate.
On the other hand, instabilities can be caused by the providers themselves, and the effect can be much larger. If a link carrying full routes between a provider and customer or a provider and another provider oscillates, the border routers will feel the impact.
Suppose that you are getting full Internet routes (about 42,000 routes in 1996) from multiple providers. Now imagine that five percent of theses routes (about 2,100 routes) are toggling every two minutes. Your border router will be unable to handle this load.
Without route dampening, it will be difficult to determine what is really happening. All you know is that the process utilization on your border router is increasing rapidly. With route dampening, all the unstable routes will generate a history entry that shows the level of stability of the routes. After the unstable routes are identified, it is easy to determine where they are coming from by looking at the next hop address. Although route dampening in this case did not help solve the problem, it helped identify who is causing the problem. After you identify the culprit, you can temporarily remove your BGP session with the ISP at fault. Pick up the phone, call the ISP, and start complaining.
In conclusion, route instabilities in the Internet will affect everybody one way or the other. It is everyone's responsibility to minimize route oscillation by being more aware of the things we do and why we do them. Providers are becoming more tough on culprits; there is even talk about charging an additional fee per route flap. This might sound like overkill, but it is getting harder and harder to get the Internet under control. Having a "routing patrol" issue tickets whenever someone breaks the rules may become necessary.
Looking Ahead
We have talked enough about architectures and routing behaviors. For the reader who wants to put things into perspective by learning actual routing implementations, the best is yet to come. The following chapters touch upon key designs and architectures already discussed in the book by presenting actual configuration by using the Cisco IOS software language.
The configuration examples are accompanied by complete explanations on why a certain action is taken and what outcome results from it. Actual displays are taken from Cisco routers to point out multiple BGP attributes and how the configuration has affected the routing tables. By going through the examples in the next two chapters, we hope that you will achieve a high-level of expertise in integrating your networks in the global Internet.