Troubleshooting Cisco Nexus Switches and NX-OS

Vinit Jain, CCIE No. 22854 (R&S, SP, Security & DC), is a technical leader with the Cisco Technical Assistance Center (TAC) providing escalation support in areas of routing and data center technologies. Vinit is a speaker at various networking forums, including Cisco Live events globally on various topics. Prior to joining Cisco, Vinit worked as a CCIE trainer and a network consultant. In addition to his CCIEs, Vinit holds multiple certifications on programming and databases. Vinit graduated from Delhi University in Mathematics and earned his Master's in nformation Technology from Kuvempu University in India. Vinit can be found on Twitter as @VinuGenie.Brad Edgeworth, CCIE No. 31574 (R&S & SP), is a systems engineer at Cisco Systems. Brad is a distinguished speaker at Cisco Live, where he has presented on various topics. Before joining Cisco, Brad worked as a network architect and consultant for various Fortune 500 companies. Brad’s expertise is based on enterprise and service provider environments with an emphasis on architectural and operational simplicity. Brad holds a Bachelor of Arts degree in Computer Systems Management from St. Edward’s University in Austin, Texas. Brad can be found on Twitter as @BradEdgeworth.Richard Furr, CCIE No. 9173 (R&S & SP), is a technical leader with the Cisco Technical Assistance Center (TAC), supporting customers and TAC teams around the world. For the past 17 years, Richard has worked for the Cisco TAC and High Touch Technical Support (HTTS) organizations, supporting service provider, enterprise, and data center environments. Richard specializes in resolving complex problems found with routing protocols, MPLS, multicast, and network overlay technologies.

• Foreword xxviIntroduction xxviiPart I Introduction to Troubleshooting Nexus SwitchesChapter 1 Introduction to Nexus Operating System (NX-OS) 1Nexus Platforms Overview 2Nexus 2000 Series 2Nexus 3000 Series 3Nexus 5000 Series 4Nexus 6000 Series 4Nexus 7000 Series 5Nexus 9000 Series 6NX-OS Architecture 8The Kernel 9System Manager (sysmgr) 9Messages and Transactional Services 11Persistent Storage Services 13Feature Manager 14NX-OS Line Card Microcode 17File Systems 19Understanding NX-OS Software Releases and Packaging 25Software Maintenance Upgrades 27Licensing 28NX-OS High-Availability Infrastructure 28Supervisor Redundancy 29ISSU 34NX-OS Virtualization Features 35Virtual Device Contexts 35Virtual Routing and Forwarding 37Virtual Port Channel 37Management and Operations Capabilities 39NX-OS Advanced CLI 39Technical Support Files 44Accounting Log 45Feature Event-History 46Debug Options: Log File and Filters 47Configuration Checkpoint and Rollback 48Consistency Checkers 49Feature Scheduler, EEM, and Python 50Bash Shell 51Summary 51References 51Chapter 2 NX-OS Troubleshooting Tools 53Packet Capture: Network Sniffer 53Encapsulated Remote SPAN 57SPAN on Latency and Drop 60Nexus Platform Tools 63Ethanalyzer 63Packet Tracer 71NetFlow 72NetFlow Configuration 73NetFlow Sampling 77sFlow 78Network Time Protocol 81Embedded Event Manager 83Logging 87Debug Logfiles 90Accounting Log 91Event-History 92Summary 93References 93Chapter 3 Troubleshooting Nexus Platform Issues 95Troubleshooting Hardware Issues 95Generic Online Diagnostic Tests 98Nexus Device Health Checks 108Nexus Fabric Extenders 124Virtual Device Context 130VDC Resource Template 131Configuring VDC 133VDC Initialization 134Out-of-Band and In-Band Management 137VDC Management 137Troubleshooting NX-OS System Components 142Message and Transaction Services 144Netstack and Packet Manager 148ARP and Adjacency Manager 160EthPM and Port-Client 175HWRL, CoPP, and System QoS 179MTU Settings 192Summary 195References 196Part II Troubleshooting Layer 2 ForwardingChapter 4 Nexus Switching 197Network Layer 2 Communication Overview 197Virtual LANs 200VLAN Creation 201Access Ports 203Trunk Ports 204Private VLANS 207Spanning Tree Protocol Fundamentals 218IEEE 802.1D Spanning Tree Protocol 219Rapid Spanning Tree Protocol 220Multiple Spanning-Tree Protocol (MST) 236Detecting and Remediating Forwarding Loops 241MAC Address Notifications 242BPDU Guard 243BPDU Filter 244Problems with Unidirectional Links 245Summary 252References 254Chapter 5 Port-Channels, Virtual Port-Channels, and FabricPath 255Port-Channels 255Basic Port-Channel Configuration 259Verifying Port-Channel Status 260Verifying LACP Packets 262Advanced LACP Configuration Options 265LACP System Priority 268Port-Channel Member Interface Consistency 271Troubleshooting LACP Interface Establishment 272Troubleshooting Traffic Load-Balancing 272Virtual Port-Channel 274vPC Fundamentals 275vPC Configuration 278vPC Verification 280Advanced vPC Features 288FabricPath 294FabricPath Terminologies and Components 296FabricPath Packet Flow 297FabricPath Configuration 300FabricPath Verification and Troubleshooting 303FabricPath Devices 310Emulated Switch and vPC+ 310vPC+ Configuration 311vPC+ Verification and Troubleshooting 314Summary 320References 320Part III Troubleshooting Layer 3 RoutingChapter 6 Troubleshooting IP and IPv6 Services 321IP SLA 321ICMP Echo Probe 322UDP Echo Probe 324UDP Jitter Probe 325TCP Connect Probe 328Object Tracking 329Object Tracking for the Interface 330Object Tracking for Route State 330Object Tracking for Track-List State 332Using Track Objects with Static Routes 334IPv4 Services 335DHCP Relay 335DHCP Snooping 341Dynamic ARP Inspection 345IP Source Guard 349Unicast RPF 351IPv6 Services 352Neighbor Discovery 352IPv6 Address Assignment 357IPv6 First-Hop Security 362First-Hop Redundancy Protocol 370HSRP 370VRRP 380GLBP 385Summary 391Chapter 7 Troubleshooting Enhanced Interior Gateway Routing Protocol (EIGRP) 393EIGRP Fundamentals 393Topology Table 395Path Metric Calculation 396EIGRP Communication 399Baseline EIGRP Configuration 399Troubleshooting EIGRP Neighbor Adjacency 401Verification of Active Interfaces 402Passive Interface 403Verification of EIGRP Packets 405Connectivity Must Exist Using the Primary Subnet 409EIGRP ASN Mismatch 412Mismatch K Values 413Problems with Hello and Hold Timers 414EIGRP Authentication Issues 416Troubleshooting Path Selection and Missing Routes 419Load Balancing 421Stub 421Maximum-Hops 424Distribute List 426Offset Lists 427Interface-Based Settings 430Redistribution 430Classic Metrics vs. Wide Metrics 433Problems with Convergence 439Active Query 441Stuck in Active 443Summary 446References 447Chapter 8 Troubleshooting Open Shortest Path First (OSPF) 449OSPF Fundamentals 449Inter-Router Communication 450OSPF Hello Packets 450Neighbor States 451Designated Routers 452Areas 453Link State Advertisements 453Troubleshooting OSPF Neighbor Adjacency 456Baseline OSPF Configuration 456OSPF Neighbor Verification 458Confirmation of OSPF Interfaces 460Passive Interface 461Verification of OSPF Packets 463Connectivity Must Exist Using the Primary Subnet 468MTU Requirements 469Unique Router-ID 471Interface Area Numbers Must Match 471OSPF Stub (Area Flags) Settings Must Match 473DR Requirements 474Timers 476Authentication 478Troubleshooting Missing Routes 482Discontiguous Network 482Duplicate Router ID 485Filtering Routes 487Redistribution 487OSPF Forwarding Address 488Troubleshooting OSPF Path Selection 494Intra-Area Routes 494Inter-Area Routes 495External Route Selection 495E1 and N1 External Routes 496E2 and N2 External Routes 497Problems with Intermixed RFC 1583 and RFC 2328 Devices 499Interface Link Costs 500Summary 504References 505Chapter 9 Troubleshooting Intermediate System-Intermediate System (IS-IS) 507IS-IS Fundamentals 507Areas 508NET Addressing 509Inter-Router Communication 511IS Protocol Header 511TLVs 512IS PDU Addressing 512IS-IS Hello (IIH) Packets 513Link-State Packets 515Designated Intermediate System 516Path Selection 517Troubleshooting IS-IS Neighbor Adjacency 518Baseline IS-IS Configuration 518IS-IS Neighbor Verification 520Confirmation of IS-IS Interfaces 523Passive Interface 526Verification of IS-IS Packets 528Connectivity Must Exist Using the Primary Subnet 535MTU Requirements 537Unique System-ID 539Area Must Match Between L1 Adjacencies 539Checking IS-IS Adjacency Capabilities 541DIS Requirements 543IIH Authentication 544Troubleshooting Missing Routes 546Duplicate System ID 546Interface Link Costs 549Mismatch of Metric Modes 553L1 to L2 Route Propagations 556Suboptimal Routing 562Redistribution 566Summary 567References 568Chapter 10 Troubleshooting Nexus Route-Maps 569Conditional Matching 569Access Control Lists 569ACLs and ACL Manager Component 570Prefix Matching and Prefix-Lists 577Route-Maps 581Conditional Matching 582Optional Actions 586Incomplete Configuration of Routing Policies 586Diagnosing Route Policy Manger 586Policy-Based Routing 591Summary 594References 595Chapter 11 Troubleshooting BGP 597BGP Fundamentals 597Address Families 598Path Attributes 599Loop Prevention 599BGP Sessions 600BGP Identifier 601BGP Messages 601BGP Neighbor States 602BGP Configuration and Verification 605Troubleshooting BGP Peering Issues 609Troubleshooting BGP Peering Down Issues 609Demystifying BGP Notifications 619Troubleshooting IPv6 Peers 621BGP Peer Flapping Issues 622BGP Route Processing and Route Propagation 630BGP Route Advertisement 631BGP Best Path Calculation 636BGP Multipath 640BGP Update Generation Process 643BGP Convergence 646Scaling BGP 649Tuning BGP Memory 650Soft Reconfiguration Inbound Versus Route Refresh 654Scaling BGP with Route-Reflectors 657Maximum Prefixes 659BGP Max AS 662BGP Route Filtering and Route Policies 662Prefix-List-Based Filtering 663Filter-Lists 669BGP Route-Maps 673Regular Expressions (RegEx) 676AS-Path Access List 684BGP Communities 684Looking Glass and Route Servers 687Logs Collection 687Summary 687Further Reading 688References 688Part IV Troubleshooting High AvailabilityChapter 12 High Availability 689Bidirectional Forwarding Detection 689Asynchronous Mode 691Asynchronous Mode with Echo Function 693Configuring and Verifying BFD Sessions 693Nexus High Availability 707Stateful Switchover 707ISSU 713Graceful Insertion and Removal 719Custom Maintenance Profile 727Summary 731References 732Part V Multicast Network TrafficChapter 13 Troubleshooting Multicast 733Multicast Fundamentals 734Multicast Terminology 735Layer 2 Multicast Addresses 738Layer 3 Multicast Addresses 739NX-OS Multicast Architecture 741Replication 744Protecting the Central Processing Unit 745NX-OS Multicast Implementation 747IGMP 750IGMPv2 751IGMPv3 752IGMP Snooping 756IGMP Verification 761PIM Multicast 771PIM Protocol State and Trees 772PIM Message Types 773PIM Interface and Neighbor Verification 780PIM Any Source Multicast 785PIM Bidirectional 799PIM RP Configuration 811PIM Source Specific Multicast 841Multicast and Virtual Port-Channel 848vPC-Connected Source 849vPC-Connected Receiver 861vPC Considerations for Multicast Traffic 870Ethanalyzer Examples 871Summary 871References 872Part VI Troubleshooting Nexus TunnelingChapter 14 Troubleshooting Overlay Transport Virtualization (OTV) 875OTV Fundamentals 875Flood Control and Broadcast Optimization 877Supported OTV Platforms 878OTV Terminology 878Deploying OTV 881Understanding and Verifying the OTV Control Plane 885OTV Multicast Mode 887OTV IS-IS Adjacency Verification 888OTV IS-IS Topology Table 898OTV IS-IS Authentication 905Adjacency Server Mode 907OTV Control Plane Policing (CoPP) 912Understanding and Verifying the OTV Data Plane 913OTV ARP Resolution and ARP-ND-Cache 915Broadcasts 917Unknown Unicast Frames 918OTV Unicast Traffic with a Multicast Enabled Transport 919OTV Multicast Traffic with a Multicast Enabled Transport 924OTV Multicast Traffic with a Unicast Transport (Adjacency Server Mode) 932Advanced OTV Features 937First Hop Routing Protocol Localization 938Multihoming 939Ingress Routing Optimization 940VLAN Translation 941OTV Tunnel Depolarization 942OTV Fast Failure Detection 944Summary 946References 947Part VII Network ProgrammabilityChapter 15 Programmability and Automation 949Introduction to Automation and Programmability 949Introduction to Open NX-OS 950Shells and Scripting 951NX-SDK 964NX-API 968Summary 975References 9759781587145056 TOC 4/24/2018