Cisco Device Troubleshooting
Layered decision trees, symptom-based checks, expected results, evidence collection, and escalation preparation.
Search troubleshooting symptoms:
Do not start by changing configuration. First identify the failure boundary and collect the current state.
A change made before verification can hide the original fault and create a second fault.
Universal troubleshooting method
1. Define scope
One host, one VLAN, one site, or all traffic?
One host, one VLAN, one site, or all traffic?
2. Check Layer 1
Link, cable, optics, speed, duplex, errors
Link, cable, optics, speed, duplex, errors
3. Check Layer 2
VLAN, MAC, trunk, STP, EtherChannel
VLAN, MAC, trunk, STP, EtherChannel
4. Check Layer 3
Address, mask, gateway, ARP, route
Address, mask, gateway, ARP, route
5. Check policy
ACL, NAT, firewall, port security
ACL, NAT, firewall, port security
6. Check return path
Remote route, reverse ACL, state
Remote route, reverse ACL, state
7. Prove and document
Retest, save evidence, prevent recurrence
Retest, save evidence, prevent recurrence
Start with these read-only commands
show clock show version show logging show interfaces description show ip interface brief show interfaces status show vlan brief show interfaces trunk show spanning-tree show etherchannel summary show mac address-table show ip arp show ip route show access-lists show ip nat statistics
Decision tree: a device cannot ping X
| Step | Test | Expected result | If the test fails |
|---|---|---|---|
| 1 | Confirm the exact source and destination IP addresses. | The addresses are correct and not duplicated. | Correct addressing or identify which interface/source the test actually uses. |
| 2 | Check the source interface with show ip interface brief. |
Required interface is up/up with the correct address. |
Resolve shutdown, cabling, SVI state, encapsulation, or line protocol failure. |
| 3 | Ping the source device's own interface address. | Local IP stack responds. | Check address assignment, VRF, local control-plane policy, or interface state. |
| 4 | If the destination is remote, ping the next hop or default gateway. | The directly connected next hop responds. | Check VLAN, trunk, STP, ARP, subnet mask, and the neighbor interface. |
| 5 | Run show ip route <destination>. |
A route exists with the intended next hop and exit interface. | Add or repair the route, adjacency, OSPF process, or default route. |
| 6 | Run a sourced ping. | The test succeeds from the address the real traffic uses. | Check return routing, ACLs, NAT, and source-specific policy. |
| 7 | Run traceroute from both sides when possible. | The path reaches the destination or identifies the final responding hop. | Investigate the first hop after the last successful response. |
| 8 | Check ACL counters and NAT translations. | Required permit entries and translations increment. | Correct ACL order/direction, NAT match criteria, or inside/outside roles. |
| 9 | Verify the return path on the destination side. | The destination has a route back to the source or translated source. | Repair return routing or stateful-policy handling. |
Ordered Cisco checks for a failed ping
show ip interface brief show interfaces <source-interface> show ip route <destination-ip> show ip cef <destination-ip> show ip arp ping <next-hop-ip> ping <destination-ip> source <source-ip-or-interface> traceroute <destination-ip> show access-lists show ip nat statistics show ip nat translations show logging
Interpret interface states correctly
| Status | Meaning | Most likely causes | Next checks |
|---|---|---|---|
| up/up | Physical and line protocol are operational. | The problem is probably above basic link state. | Check VLAN, addressing, routes, ACLs, NAT, and the remote path. |
| down/down | No usable physical signal. | Disconnected cable, bad optic, powered-off peer, wrong transceiver, disabled remote port. | show interfaces, LEDs, cable/optic swap, remote interface state. |
| admin down | The local interface has shutdown. |
Intentional shutdown, unused port template, or missed no shutdown. |
Review description and change record before enabling. |
| up/down | Layer 1 works, but line protocol does not. | Encapsulation mismatch, keepalive issue, inactive SVI, serial clocking, EtherChannel mismatch. | Inspect detailed interface output and peer configuration. |
| err-disabled | The switch disabled the port after a protection event. | BPDU Guard, port security, link flap, UDLD, channel mismatch, DHCP rate limit. | show errdisable recovery, show logging, feature-specific show commands. |
Layer 2 and switching failures
Host cannot reach its gateway
- Check host IP, mask, gateway, and duplicate address risk.
- Check switch interface state and description.
- Verify static access mode and access VLAN.
- Verify the VLAN exists and is active.
- Verify STP is forwarding on the host port and uplinks.
- Verify the host MAC appears on the expected port and VLAN.
- Verify the gateway SVI or router subinterface is up.
show interfaces status show interfaces <port> switchport show vlan brief show spanning-tree interface <port> detail show port-security interface <port> show mac address-table interface <port> show interfaces trunk show ip interface brief
VLAN works on one switch but not another
- Confirm the VLAN exists on both switches.
- Confirm the interswitch link is operationally trunking.
- Check the allowed VLAN list in both directions.
- Check native VLAN consistency.
- Check STP state for the affected VLAN.
- Check EtherChannel state when the trunk uses a port-channel.
show vlan brief show interfaces trunk show interfaces <uplink> switchport show spanning-tree vlan <vlan-id> show etherchannel summary
STP blocks the wrong path
- Identify the actual root bridge for the VLAN.
- Check bridge priorities and root path costs.
- Check interface cost and port-priority overrides.
- Confirm the topology matches the intended root placement.
- Check for root guard or loop guard inconsistency.
show spanning-tree vlan <vlan-id> show spanning-tree root show spanning-tree interface <port> detail show spanning-tree inconsistentports
EtherChannel member is suspended or standalone
- Compare channel protocol and mode on both ends.
- Compare speed, duplex, trunk mode, native VLAN, and allowed VLANs.
- Check whether configuration belongs on the port-channel interface.
- Check LACP or PAgP neighbor information.
show etherchannel summary show etherchannel port-channel show lacp neighbor show pagp neighbor show interfaces trunk show running-config interface port-channel <id>
Layer 3 and routing failures
Route is missing
- The outgoing interface must be operational.
- The next hop must be recursively reachable.
- A floating static route installs only when lower-AD alternatives disappear.
- OSPF must advertise and accept the prefix.
- A more specific route can change the selected path.
show ip route show ip route <destination> show ip cef <destination> show ip protocols show running-config | include ^ip route
OSPF neighbors do not reach FULL
| Check | Must be compatible |
|---|---|
| Layer 3 connectivity | Same subnet and working link |
| Area | Same area ID |
| Timers | Hello and dead intervals |
| Network type | Compatible behavior |
| Authentication | Same method and key |
| MTU | Mismatch can stop database exchange |
| Router ID | Unique |
| Passive state | Transit interface must not be passive |
show ip ospf neighbor show ip ospf interface <interface> show ip ospf interface brief show ip ospf database show ip protocols show logging
Inter-VLAN routing fails
- Verify every VLAN exists.
- Verify access ports are in the intended VLAN.
- Verify trunks allow all required VLANs.
- For router-on-a-stick, verify subinterface VLAN tags.
- For a multilayer switch, verify
ip routing. - Verify each SVI or subinterface is up with the correct gateway address.
- Check ACLs between VLANs.
Asymmetric or intermittent routing
- Run sourced pings from every relevant interface.
- Compare traceroute in both directions.
- Check equal-cost paths and per-flow hashing.
- Check stateful firewalls and NAT for asymmetric paths.
- Check route flaps and interface transitions in logs.
DHCP, NAT, ACL, and SSH failures
DHCP client gets no address
- Verify the client VLAN and link.
- Verify the DHCP pool network, exclusions, and remaining addresses.
- If the server is remote, verify
ip helper-addresson the client gateway interface. - Verify routing between relay and server.
- Check DHCP snooping trust on the server-facing path.
- Check conflicts and bindings.
show ip dhcp pool show ip dhcp binding show ip dhcp conflict show running-config interface vlan <id> show ip dhcp snooping show ip dhcp snooping binding
NAT or PAT does not translate
- Verify the inside and outside interface roles.
- Verify the NAT ACL matches the real inside source.
- Verify a route exists to the outside destination.
- Generate new traffic after clearing only when safe.
- Check translation and statistics counters.
- Verify the return traffic reaches the outside interface.
show ip nat statistics show ip nat translations verbose show access-lists show ip route show running-config | include ip nat
ACL blocks valid traffic
- Confirm the actual protocol, source, destination, and ports.
- Read the ACL top to bottom.
- Check for a broader deny before the intended permit.
- Check interface and direction.
- Check counters before and after a controlled test.
- Remember the implicit deny.
show access-lists show ip interface <interface> show running-config interface <interface>
SSH fails
- Verify IP reachability to the management address.
- Verify hostname and domain name exist.
- Verify RSA keys and SSH version 2.
- Verify a valid local user or AAA method.
- Verify VTY lines use
login localor the intended method. - Verify
transport input ssh. - Check VTY access-class source restrictions.
show ip ssh show ssh show users show running-config | section line vty show access-lists
Safe use of debug commands
Debug can affect production performance and flood the terminal. Use a narrow debug during a controlled window.
Start logging capture first. Stop debugging immediately after collecting the required evidence.
terminal monitor debug ip ospf adj ! Reproduce the problem once undebug all terminal no monitor
Evidence to collect before escalation
- Exact source IP, destination IP, protocol, destination port, and timestamp.
- Device hostname, model, serial number, IOS or IOS XE release, and uptime.
- Relevant interface descriptions, states, counters, speed, duplex, and optics information.
- VLAN, trunk, STP, MAC table, and EtherChannel state.
- IP addresses, ARP or neighbor entries, route lookup, and sourced ping results.
- ACL entries and counters, NAT rules and translations, DHCP bindings, or OSPF neighbors.
- Relevant log messages with timestamps.
- Running configuration for only the affected interfaces and features.
- A topology sketch showing the failure boundary and last known working hop.
- Changes made, rollback performed, and the result after each action.
Embedded CLI tutorial: host cannot reach its gateway
This procedure was moved from the general use-case page because it belongs in the troubleshooting workflow.
19. Troubleshoot a host that cannot reach its gateway
Use when: Use this ordered check to classify the failure as physical, VLAN, STP, security, gateway, or addressing.
Before you start: Know the host switchport, intended VLAN, host IP, and gateway IP.
| Step | Command | Why you use it |
|---|---|---|
| 1 | show interfaces status | Confirm the port is connected and inspect VLAN, speed, and duplex. |
| 2 | show interfaces gigabitEthernet1/0/10 | Check link state and error counters. |
| 3 | show interfaces gigabitEthernet1/0/10 switchport | Confirm access mode and VLAN. |
| 4 | show vlan brief | Confirm the VLAN exists and the port is assigned. |
| 5 | show spanning-tree interface gigabitEthernet1/0/10 detail | Confirm the port is forwarding and not inconsistent. |
| 6 | show port-security interface gigabitEthernet1/0/10 | Check security violations or error-disable state. |
| 7 | show mac address-table interface gigabitEthernet1/0/10 | Confirm the switch learns the endpoint MAC. |
| 8 | show interfaces trunk | Confirm the VLAN is carried toward the gateway. |
| 9 | show ip interface brief | Confirm the gateway SVI is up/up. |
| 10 | show ip arp | Check whether the gateway learns the host IP-to-MAC mapping. |
| 11 | ping <host-ip> | Test Layer 3 reachability from the gateway. |
Verification
Collect the output before changing the port. The first failed check identifies the troubleshooting layer.
Root cause and prevention template
| Field | Record |
|---|---|
| Symptom | What failed from the user's perspective? |
| Scope | Which hosts, VLANs, sites, protocols, and times were affected? |
| Failure boundary | What was the last successful test and first failed test? |
| Root cause | Which exact configuration, hardware, software, or dependency caused the fault? |
| Corrective action | What restored service? |
| Verification | Which tests prove normal operation? |
| Prevention | Monitoring, configuration standard, documentation, testing, or hardware change. |