Troubleshooting BGP

Vinit Jain, CCIE No. 22854 (R&S, SP, Security & DC), is a High Touch Technical Support (HTTS) engineer with Cisco providing support to premium customers of Cisco on complex routing technologies. Before joining Cisco, Vinit worked as a CCIE trainer and a network consultant. In addition to his expertise in networks, he has experience with software development, with which he began his career.Vinit holds certifications for multiple vendors, such as Cisco, Microsoft, Sun Microsystems, VMware, and Oracle, and also is a Certified Ethical Hacker. Vinit is a speaker at Cisco Live and various other forums, including NANOG. Vinit pursued his graduation from Delhi University in Mathematics and earned his Masters in Information Technology from Kuvempu University in India. Vinit is married and is presently based out of RTP, North Carolina. Vinit can be found on Twitter @vinugenie.Brad Edgeworth, CCIE No. 31574 (R&S & SP), has been with Cisco working as a systems engineer and a technical leader. Brad is a distinguished speaker at Cisco Live, where he has presented on multiple topics. Before joining Cisco, Brad worked as a network architect and consulted for various Fortune 500 companies. Brad’s other certifications include Cisco Certified Design Professional (CCDP) and Microsoft Certified Systems Engineer (MCSE). Brad has been working in the IT field with an emphasis on enterprise and service provider environments from an architectural and operational perspective. Brad holds a Bachelor of Arts degree in Computer Systems Management from St. Edward’s University in Austin, Texas. Brad can be found on Twitter @BradEdgeworth.

• Foreword xxiiIntroduction xxiiiPart I BGP FundamentalsChapter 1 BGP Fundamentals 1Border Gateway Protocol 1Autonomous System Numbers 2Path Attributes 3Loop Prevention 3Address Families 3BGP Sessions 4Inter-Router Communication 5BGP Messages 6OPEN 6Hold Time 6BGP Identifier 7KEEPALIVE 7UPDATE 7NOTIFICATION Message 8BGP Neighbor States 8Idle 9Connect 9Active 10OpenSent 10OpenConfirm 10Established 10Basic BGP Configuration 11IOS 11IOS XR 12NX-OS 13Verification of BGP Sessions 14Prefix Advertisement 17BGP Best-Path Calculation 20Route Filtering and Manipulation 21IBGP 22IBGP Full Mesh Requirement 24Peering via Loopback Addresses 25EBGP 26EBGP and IBGP Topologies 28Next-Hop Manipulation 30IBGP Scalability 31Route Reflectors 31Loop Prevention in Route Reflectors 33Out-of-Band Route Reflectors 33Confederations 34BGP Communities 37Route Summarization 38Aggregate-Address 39Flexible Route Suppression 40Selective Prefix Suppression 40Leaking Suppressed Routes 40Atomic Aggregate 40Route Aggregation with AS_SET 42Route Aggregation with Selective Advertisement of AS-SET 42Default Route Advertisement 42Default Route Advertisement per Neighbor 42Remove Private AS 43Allow AS 43LocalAS 43Summary 44References 45Part II Common BGP TroubleshootingChapter 2 Generic Troubleshooting Methodologies 47Identifying the Problem 47Understanding Variables 48Reproducing the Problem 49Setting Up the Lab 49Configuring Lab Devices 52Triggering Events 56Sniffer-Packet Capture 57SPAN on Cisco IOS 58SPAN on Cisco IOS XR 60SPAN on Cisco NX-OS 62Remote SPAN 63Platform-Specific Packet Capture Tools 65Netdr Capture 66Embedded Packet Capture 68Ethanalyzer 70Logging 74Event Monitoring/Tracing 77Summary 81Reference 81Chapter 3 Troubleshooting Peering Issues 83BGP Peering Down Issues 83Verifying Configuration 84Verifying Reachability 87Find the Location and Direction of Packet Loss 88Verify Whether Packets Are Being Transmitted 89Use Access Control Lists to Verify Whether Packets Are Received 90Check ACLs and Firewalls in Path 91Verify TCP Sessions 94Simulate a BGP Session 95Demystifying BGP Notifications 96Decode BGP Messages 99Troubleshoot Blocked Process in IOS XR 103Verify BGP and BPM Process State 104Verify Blocked Processes 105Restarting a Process 106BGP Traces in IOS XR 106BGP Traces in NX-OS 108Debugs for BGP 110Troubleshooting IPv6 Peers 112Case Study–Single Session Versus Multisession 113Multisession Capability 114Single-Session Capability 115BGP Peer Flapping Issues 115Bad BGP Update 115Hold Timer Expired 116Interface Issues 116Physical Connectivity 117Physical Interface 117Input Hold Queue 117TCP Receive Queue 119MTU Mismatch Issues 120High CPU Causing Control-Plane Flaps 125Control Plane Policing 127CoPP on NX-OS 129Local Packet Transport Services 134Dynamic BGP Peering 138Dynamic BGP Peer Configuration 139Dynamic BGP Challenges 142Misconfigured MD5 Password 142Resource Issues in a Scaled Environment 142TCP Starvation 142Summary 143References 143Chapter 4 Troubleshooting Route Advertisement and BGP Policies 145Troubleshooting BGP Route Advertisement 145Local Route Advertisement Issues 145Route Aggregation Issues 147Route Redistribution Issues 150BGP Tables 152Receiving and Viewing Routes 154Troubleshooting Missing BGP Routes 156Next-Hop Check Failures 157Bad Network Design 160Validity Check Failure 162AS-Path 162Originator-ID/Cluster-ID 165BGP Communities 167BGP Communities: No-Advertise 167BGP Communities: No-Export 169BGP Communities: Local-AS (No Export SubConfed) 170Mandatory EBGP Route Policy for IOS XR 172Filtering of Prefixes by Route Policy 173Conditional Matching 174Access Control Lists (ACL) 174Prefix Matching 175Regular Expressions (Regex) 177UnderScore _ 179Caret ^ 180Dollar Sign $ 181Brackets [ ] 181Hyphen - 182Caret in Brackets [^] 182Parentheses ( ) and Pipe | 183Period . 183Plus Sign + 183Question Mark ? 184Asterisk * 184Looking Glass and Route Servers 185Conditionally Matching BGP Communities 185Troubleshooting BGP Router Policies 185IOS and NX-OS Prefix-Lists 186IOS and NX-OS AS-Path ACLs 188Route-Map Processing 191IOS and NX-OS Route-Maps 192IOS XR Route-Policy Language 196Incomplete Configuration of Routing Policies 198Conditional BGP Debugs 199Summary 203Further Reading 204References in This Chapter 204Chapter 5 Troubleshooting BGP Convergence 205Understanding BGP Route Convergence 205BGP Update Groups 207BGP Update Generation 212Troubleshooting Convergence Issues 216Faster Detection of Failures 218Jumbo MTU for Faster Convergence 219Slow Convergence due to Periodic BGP Scan 219Slow Convergence due to Default Route in RIB 222BGP Next-Hop Tracking 223Selective Next-Hop Tracking 225Slow Convergence due to Advertisement Interval 226Computing and Installing New Path 226Troubleshooting BGP Convergence on IOS XR 227Verifying Convergence During Initial Bring Up 227Verifying BGP Reconvergence in Steady State Network 228Troubleshooting BGP Convergence on NX-OS 234BGP Slow Peer 237BGP Slow Peer Symptoms 238High CPU due to BGP Router Process 238Traffic Black Hole and Missing Prefixes in BGP table 238BGP Slow Peer Detection 239Verifying OutQ value 240Verifying SndWnd 240Verifying Cache Size and Pending Replication Messages 241Workaround 242Changing Outbound Policy 242Advertisement Interval 243BGP Slow Peer Feature 245Static Slow Peer 245Dynamic Slow Peer Detection 245Slow Peer Protection 246Slow Peer Show Commands 246Troubleshooting BGP Route Flapping 246Summary 250Reference 250Part III BGP Scalability IssuesChapter 6 Troubleshooting Platform Issues Due to BGP 251Troubleshooting High CPU Utilization due to BGP 251Troubleshooting High CPU due to BGP on Cisco IOS 252High CPU due to BGP Scanner Process 253High CPU due to BGP Router Process 255High CPU Utilization due to BGP I/O Process 256Troubleshooting High CPU due to BGP on IOS XR 258Troubleshooting High CPU due to BGP on NX-OS 262Capturing CPU History 265Troubleshooting Sporadic High CPU Condition 265Troubleshooting Memory Issues due to BGP 267TCAM Memory 269Troubleshooting Memory Issues on Cisco IOS Software 269Troubleshooting Memory Issues on IOS XR 274Troubleshooting Memory Issues on NX-OS 278Restarting Process 281Summary 281References 282Chapter 7 Scaling BGP 283The Impact of Growing Internet Routing Tables 283Scaling Internet Table on Various Cisco Platforms 285Scaling BGP Functions 288Tuning BGP Memory 290Prefixes 290Managing the Internet Routing Table 290Paths 292Attributes 293Tuning BGP CPU 295IOS Peer-Groups 295IOS XR BGP Templates 295NX-OS BGP Peer Templates 296BGP Peer Templates on Cisco IOS 297Soft Reconfiguration Inbound Versus Route Refresh 298Dynamic Refresh Update Group 302Enhanced Route Refresh Capability 305Outbound Route Filtering (ORF) 309Prefix-Based ORF 309Extended Community—Based ORF 309BGP ORF Format 310BGP ORF Configuration Example 312Maximum Prefixes 316BGP Max AS 318BGP Maximum Neighbors 322Scaling BGP with Route Reflectors 322BGP Route Reflector Clusters 324Hierarchical Route Reflectors 331Partitioned Route Reflectors 332BGP Selective Route Download 339Virtual Route Reflectors 342BGP Diverse Path 346Shadow Route Reflectors 349Shadow Sessions 355Route Servers 357Summary 364References 365Chapter 8 Troubleshooting BGP Edge Architectures 367BGP Multihoming and Multipath 367Resiliency in Service Providers 370EBGP and IBGP Multipath Configuration 370EIBGP Multipath 372R1 373R2 374R3 374R4 375R5 376AS-Path Relax 377Understanding BGP Path Selection 377Routing Path Selection Longest Match 377BGP Best-Path Overview 379Weight 380Local Preference 380Locally Originated via Network or Aggregate Advertisement 380Accumulated Interior Gateway Protocol (AIGP) 381Shortest AS-Path 383Origin Type 383Multi-Exit Discriminator (MED) 384EBGP over IBGP 386Lowest IGP Metric 386Prefer the Oldest EBGP Path 387Router ID 387Minimum Cluster List Length 388Lowest Neighbor Address 388Troubleshooting BGP Best Path 389Visualizing the Topology 390Phase I–Initial BGP Edge Route Processing 391Phase II–BGP Edge Evaluation of Multiple Paths 392Phase III–Final BGP Processing State 394Path Selection for the Routing Table 394Common Issues with BGP Multihoming 395Transit Routing 395Problems with Race Conditions 397Peering on Cross-Link 402Expected Behavior 403Unexpected Behavior 406Secondary Verification Methods of a Routing Loop 409Design Enhancements 411Full Mesh with IBGP 412Problems with Redistributing BGP into an IGP 413Summary 417References 418Part IV Securing BGPChapter 9 Securing BGP 419The Need for Securing BGP 419Securing BGP Sessions 420Explicitly Configured Peers 421IPv6 BGP Peering Using Link-Local Address 421BGP Session Authentication 424BGP Pass Through 426EBGP-Multihop 427BGP TTL Security 428Filtering 429Protecting BGP Traffic Using IPsec 431Securing Interdomain Routing 431BGP Prefix Hijacking 432S-BGP 439IPsec 439Public Key Infrastructure 439Attestations 441soBGP 442Entity Certificate 442Authorization Certificate 443Policy Certificate 443BGP SECURITY Message 443BGP Origin AS Validation 443Route Origination Authorization (ROA) 445RPKI Prefix Validation Process 446Configuring and Verifying RPKI 449RPKI Best-Path Calculation 460BGP Remote Triggered Black-Hole Filtering 463BGP Flowspec 467Configuring BGP Flowspec 469Summary 479References 480Part V Multiprotocol BGPChapter 10 MPLS Layer 3 VPN (L3VPN) 481MPLS VPNs 481MPLS Layer 3 VPN (L3VPN) Overview 483Virtual Routing and Forwarding 483Route Distinguisher 485Route Target 485Multi-Protocol BGP (MP-BGP) 486Network Advertisement Between PE and CE Routers 487MPLS Layer 3 VPN Configuration 487VRF Creation and Association 488IOS VRF Creation 488IOS XR VRF Creation 489NX-OS VRF Creation 490Verification of VRF Settings and Connectivity 492Viewing VRF Settings and Interface IP Addresses 492Viewing the VRF Routing Table 494VRF Connectivity Testing Tools 495MPLS Forwarding 495BGP Configuration for VPNv4 and PE-CE Prefixes 497IOS BGP Configuration for MPLS L3VPN 497IOS XR BGP Configuration for MPLS L3VPN 499NX-OS BGP Configuration for MPLS L3VPN 500Verification of BGP Sessions and Routes 502Troubleshooting MPLS L3VPN 506Default Route Advertisement Between PE-CE Routers 508Problems with AS-PATH 509Suboptimal Routing with VPNv4 Route Reflectors 514Troubleshooting Problems with Route Targets 520MPLS L3VPN Services 524RT Constraints 534MPLS VPN Label Exchange 538MPLS Forwarding 541Summary 542References 542Chapter 11 BGP for MPLS L2VPN Services 543L2VPN Services 543Terminologies 545Virtual Private Wire Service 548Interworking 549Configuration and Verification 550VPWS BGP Signaling 558Configuration 560Virtual Private LAN Service 561Configuration 562Verification 564VPLS Autodiscovery Using BGP 569VPLS BGP Signaling 580Troubleshooting 586Summary 588References 589Chapter 12 IPv6 BGP for Service Providers 591IPv6 BGP Features and Concepts 591IPv6 BGP Next-Hop 591IPv6 Reachability over IPv4 Transport 596IPv4 Routes over IPv6 Next-Hop 601IPv6 BGP Policy Accounting 604IPv6 Provider Edge Routers (6PE) over MPLS 6076PE Configuration 6116PE Verification and Troubleshooting 615IPv6 VPN Provider Edge (6VPE) 620IPv6-Aware VRF 6226VPE Next-Hop 623Route Target 6246VPE Control Plane 6246VPE Data Plane 6266VPE Configuration 6276VPE Control-Plane Verification 6296VPE Data Plane Verification 633Summary 639References 639Chapter 13 VxLAN BGP EVPN 641