Fix BGP Neighbor Session Auth Failure: MD5 Mismatch

You've configured BGP authentication, but the neighbor stays idle.

I've been there. You paste the password, double-check it case by case, and yet the session never comes up. The fix isn't obvious: you need to clear the TCP session after fixing the password. This isn't a BGP quirk — it's how TCP MD5 works.

The Fix

Here's the sequence that works for Cisco IOS and IOS-XE (I'm assuming you're on 15.x or later; same logic applies to NX-OS).

Check the password on both sides. It's case-sensitive. On Cisco, look at show run | section router bgp and find the neighbor <ip> password line. On Juniper, show configuration protocols bgp group <name> neighbor <ip> shows the authentication-key. Verify they match exactly.
If they don't match, fix the one that's wrong. But don't stop there — the session won't come up until you kill the existing TCP connection.
Clear the BGP session using the hard (not soft) option. On Cisco:
```
clear ip bgp <neighbor-ip> hard
```
On Juniper:
```
clear bgp neighbor <neighbor-ip> hard
```
The hard flag tears down the TCP connection and re-establishes it with the new MD5 digest. A soft reset only re-advertises routes — it won't re-authenticate.
Watch the session come up. Run show ip bgp summary (Cisco) or show bgp summary (Juniper). Within a few seconds, the state should flip from Active to Established.

Why This Works

What's actually happening here is that BGP uses TCP MD5 signatures (RFC 2385) for authentication. The password is converted into an MD5 hash and embedded in every TCP segment sent to the neighbor. The neighbor computes the same hash and compares. When the passwords match, the TCP connection is allowed.

The reason step 3 works is that the TCP connection itself stores the MD5 key in its socket structure. Changing the BGP configuration updates the key for future connections, but the existing TCP socket keeps using the old one. BGP won't tear down the TCP session on its own when the password changes — it just sits there, sending packets with a stale hash. The neighbor sees the mismatch and drops the packet silently. No error message, no log, nada. You have to force a fresh TCP handshake.

I learned this the hard way in 2018 on a Cisco 4451 running 16.9.1. I spent 45 minutes re-typing passwords before a senior engineer walked over and said, "Clear it hard." That fixed it in 5 seconds.

Less Common Variations

1. Password Contains Special Characters

Some platforms (especially older Cisco IOS versions like 12.4) truncate passwords longer than 25 characters or silently reject certain characters. On Cisco, the password is limited to 80 characters, but I've seen it break at 64 on certain 15.x trains. Test with a simple string like bgptest1 first. If that works, the issue is the password length or characters. Use only alphanumeric plus ! @ # $ % ^ & * to be safe.

2. MTU Issues with MD5

Adding MD5 authentication increases the TCP header by 18 bytes (the option field). If your path MTU is borderline — say, you're using GRE tunnels with a 1476-byte MTU — the extra bytes can push the packet past the interface MTU, causing fragmentation or drops. The BGP session will flap randomly, not just refuse to come up. Check with ping <neighbor> size 1476 df-bit. If it fails, lower the BGP MTU using ip tcp path-mtu-discovery (Cisco) or tcp-mss 1400 (Juniper).

3. Time Sync Issues

This one's rare but real. MD5 doesn't use timestamps, but some implementations (Juniper Junos before 18.2) used the TCP timestamp option in conjunction with MD5. If the clocks on both routers are more than a few seconds apart, the timestamp checksum fails. I saw this once on a pair of MX240s where NTP was broken. Sync them with NTP, then reset the session.

4. TTL Security (GTSM) Conflict

Generalized TTL Security Mechanism (GTSM) drops packets where the TTL doesn't match expected hops. If you've enabled neighbor <ip> ttl-security hops 1 (Cisco) or ttl 1 (Juniper), and the neighbor is more than one hop away (e.g., over an MPLS LSP), the packets never reach BGP. The log will show BGP-3-NOTIFICATION: sent with no clear error. Disable GTSM or increase the hop count.

Prevention

Stop making the same mistake twice. Here's what I do now:

Use a password manager to store BGP keys. I use a KeePass database shared between team members. No more typing by hand.
Write a quick script to compare configs. On Cisco, show run | include neighbor.*password gives you the line. On Juniper, show configuration | display set | match authentication-key. Diff the outputs before and after a change.
Always clear the session hard after any authentication change. Build this into your change plan — don't assume the state will fix itself.
Log authentication failures. On Cisco, debug ip bgp updates is too noisy. Instead, use event manager to trap BGP-3-NOTIFICATION syslog messages and alert you. On Juniper, set traceoptions file bgp-auth with flag authentication so you see the exact reason for the teardown.

That's it. The fix is quick once you know the trick. The next time a BGP neighbor won't come up, don't re-type the password three times. Just check the hash, fix it, and clear it hard.