SDN Controller Down: Fix East-West Link Disconnect

Quick Answer

Check the TLS certificate expiry on both controllers. Renew if expired. Then restart the opendaylight service on each node.

Context & Why This Happens

I know this error makes you want to throw your laptop out the window. I've been there. This particular OpenFlow error (OPENFLOW_ERR_0042) shows up when your SDN controllers — say, OpenDaylight or ONOS — lose their east-west link. That's the sync channel between controllers in a cluster.

The most common trigger? A TLS certificate that expired during a maintenance window. I've seen it on OpenStack deployments running Newton and older, where the self-signed certificates were set to 365 days. Suddenly, the controllers stop talking. The log says something like "SSL handshake failed" or "peer certificate not trusted."

Other reasons: network firewall rules blocking port 6644 (the default east-west port), or a clock skew over 5 seconds between controllers. But 9 times out of 10, it's the cert. Let's fix it.

Step-by-Step Fix

Step 1: Check Certificate Expiry

1. SSH into controller 1.
2. Run:

   openssl x509 -in /etc/opendaylight/certs/controller.crt -noout -enddate

3. Check the output. If the date is in the past, you've found the problem.
4. Do the same on controller 2 and 3 (if you have a 3-node cluster).

Step 2: Renew the Certificate

On each controller where the cert expired, regenerate it. I recommend using the same script from your deployment (e.g., generate-cert.sh from OpenDaylight's bin/ directory). If you don't have that, do this:

cd /etc/opendaylight/certs
openssl req -x509 -newkey rsa:2048 -keyout controller.key -out controller.crt -days 3650 -nodes -subj "/CN=$(hostname -f)"

Yes, I'm setting 10 years. I learned the hard way that 1-year certs are a trap. You won't remember to renew them.

Step 3: Restart Services

On each controller, restart the SDN controller service:

systemctl restart opendaylight

Wait 2 minutes for the cluster to re-form. Then check the logs:

tail -f /var/log/opendaylight/karaf.log | grep -i "east-west"

You should see a line like "East-West connection established with peer 10.0.0.2:6644".

Step 4: Verify Sync

On controller 1, run:

opendaylight:cluster-status

Look for "Cluster state: ACTIVE" and each node showing "LEADER" or "FOLLOWER". If you see "SPLIT_BRAIN" or "ISOLATED", you've got more work to do. But 90% of the time, the cert was the bottleneck.

Alternative Fixes If Main Fix Fails

If the cert was fine or renewing it didn't help, here's what else could be wrong:

• Firewall blocking east-west port: Check port 6644 (or whatever you configured in etc/opendaylight/datastore/initial_config.xml). Use telnet <peer-ip> 6644 to test connectivity. If it times out, update your firewall rules.
• Clock skew: Run chronyc tracking on each controller. If the skew exceeds 5 seconds, fix NTP. On Ubuntu 18.04, I've seen the default chrony config drift badly.
• Controller IP changed: If you moved the controller to a new subnet, update the initial_config.xml with the new IPs. Then restart the service.
• Java truststore issue: If the cert is valid but not trusted, import it into the Java truststore:

  keytool -import -trustcacerts -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -noprompt -alias odl-eastwest -file /etc/opendaylight/certs/controller.crt

Prevention Tip

Set a calendar reminder 30 days before your certificate expiry. I use a cron job that emails me every month:

0 9 1 * * /usr/bin/openssl x509 -in /etc/opendaylight/certs/controller.crt -noout -enddate | mail -s "SDN cert check" you@company.com

Also, consider using a proper internal CA (like FreeIPA or HashiCorp Vault) that automates renewals. Manual cert management is a ticking time bomb in any SDN cluster.