I'm having a strange issue with network connectivity between an SBC3003 and HUB5000 on the same wireless network. Connectivity just stops working for some reason.
When I first factory reset the HUB5000, or afterwards when I reboot my router, the SBC3003 can connect to the device no problem. After a few hours, connectivity is lost and nothing I can do will restore it except another factory reset of the HUB5000, or rebooting my router. Weird thing is, only the SBC3003 exhibits this issue. No other device on my network loses connectivity to the HUB5000. The SBC3003 can't ping the HUB5000 over IPv4 or v6; every other device I tested can do both and can resolve the hostname no problem.
I've tried rebooting the SBC3003 and the HUB5000, restarting avahi and networking on the SBC3003, renaming the devices, and using static addresses.. Nothing fixes it except factory reset on the HUB5000, or rebooting my router which isn't something I can do multiple times per day.
I'm convinced this has something to do with mDNS, but I'm not having any luck with a fix. All devices have the latest code. Has anyone else seen this? Any ideas?
Are you able to reach the SBC and the HUB5000 from a PC on the same network, using their IP addresses?
When the SBC loses ability to ping the HUB5000, can it ping other devices on the network? Are you using the SBC remotely over SSH or with a keyboard/screen?
It seems like it may be some king of routing error, or maybe an issue with WiFi. Have you tried using ethernet? Are you using the WiFi dongle we sell on with SBC? It could be that the dongle is having issues.
I think mDNS related because nothing else makes sense to me as a network engineer.
Yes, I can ALWAYS reach the SBC and HUB from any other PC on my network at all times using IPv4, IPv6, or by resolving the hostname.
When the SBC loses connectivity with the HUB, that's the only thing on my network it loses connectivity with. I connect to the SBC via SSH.
One weird symptom I noticed is that when the SBC can't connect to the HUB, then other PCs on my network resolve the HUB's hostname to the IPv6 address by default. When the SBC can connect to the HUB, the HUB hostname resolves to the IPv4 address. This is another reason I think it's mDNS related.
Unfortunately, with no ability to get a Linux shell on the HUB, my troubleshooting options are extremely limited.
Yes, I am using the wifi dongle sold by Phidgets on the SBC. I'm going to connect both devices to the wired network today and see if things still behave the same way.
I've got at least 50 mDNS "services" advertised on my network, so I can't rule out a bug with the router or something like that, but I have a hard time reconciling that with the fact that ONLY the SBC to HUB connectivity is impacted and nothing else.
I don't think this could be an mDNS issue, as you are having trouble even using IP addresses directly. mDNS is just used for resolving .local hostnames and for service registration/discovery.
It sounds like it could be an IPv6 issue. When the HUB5000 can't be accessed from the SBC, what is it's IPv4 address? You said you tried setting up static IPv4 addresses, and there were still issues, so it's probably not a DHCP issue.
Have a look at the system logs on the SBC and the HUB5000 (accessible via their web interfaces), and hopefully there are some network related entries. If the IPv4 address is getting revoked by Avahi on the HUB5000 for some reason, that would show up in the log.
Well, this has been a saga, but I believe my issue to be resolved for now.
I still don't know exactly what the issue was. At some level, DHCP + wifi with my specific router/modem/AP combo device is implicated.
I finally noticed that, after a router reboot, the SBC lost connectivity with the HUB on a very precise 6 hour schedule. This was exactly half of the DHCP lease time on my network (12 hours) which is the expected time for most DHCP clients to renew their lease (confirmed via packet capture this was happening). For whatever reason, this renew process and momentary loss of connectivity caused the SBC to lose its ARP entry for the HUB and not be able to re-establish it.
In my quest to fix the issue, I ran cat5 cables from both the SBC and HUB to the nearest switch on my network which was a Cisco Nexus 3172. It didn't take long for me to determine neither device would play nice with a GLC-T transceiver; I couldn't even get a link to come up. I had some no-name SFP ethernet transceivers which established a link with the Phidgets, but they wouldn't pass broadcast traffic at all. Ugh. Not a Phidget problem, but I do wonder when adding a 1gbps ethernet PHY is on the roadmap for the SBC and the HUB?
Anyways, I ran another set of cat5 cables all the way to the other side of my house to my core Cisco Catalyst 3850 switch. No problem getting a link there. I put the 2 Phidgets on their own VLAN with DHCP and routing (inc. multicast) provided by the Cat 3850.
So far we're going on a little over 48 hours with no loss in connectivity.
At some point soon I will connect a Cisco or Ubiquiti access point to my new Phidget VLAN and see if I can replicate the issues I was experiencing. I've got a feeling I was getting bit by some some nonsense bug with my router/modem/AP combo device which won't be present when that device is removed from the equation.
