Car enthusiasts want to know how quickly they can make a quarter mile. Weightlifters are forever trying to add one more plate to the bar. Internet denizens have their own favorite number to brag about: the result from a speed test.

The ritual is familiar. Close a few browser tabs, click the big “Go” button, and watch the needle climb. Perhaps you pay for gigabit service and see 940 megabits per second, which produces a satisfied nod. Perhaps you see 299 megabits and begin obsessing over network hardware. But before you get too excited either way, try another test. There is a fair chance it will give you a different answer.

That does not necessarily mean one test is lying. “Internet speed” is not a single physical quantity waiting to be measured. A speed test measures the performance of a particular device, over a particular local connection, through a particular ISP route, to a particular server, at a particular time using a particular test method. Change any of those things and the answer can change too.

The Usual Suspects

Speedtest by Ookla is probably the best-known test. It selects a nearby server, although you can choose another. It attempts to saturate the connection with multiple simultaneous transfers. That makes it good at answering the question most consumers are asking: approximately how much aggregate bandwidth can this Internet connection deliver?

Running several connections matters. A single TCP connection must gradually increase its sending rate while reacting to round-trip time, packet loss, receive-window limits, and congestion-control behavior. On a high-bandwidth or high-latency path, one connection may not fill the available pipe. Several parallel connections can ramp up independently and make it easier to reach the link’s aggregate capacity. That number is valid, but it represents something like a busy household, a large segmented download, or several applications operating at once. It does not necessarily predict the speed of one file transfer from one distant server.

Google’s built-in search speed test (search “speed test”) uses Measurement Lab’s Network Diagnostic Tool, or NDT. M-Lab describes NDT as a single-stream measurement of bulk-transport capacity. That makes it an interesting counterpoint to Ookla. A single flow may expose latency, loss, or TCP-window limitations that a multi-stream test can partially conceal. You can also use M-Lab’s own speed test directly.

While you may get similar numbers between the two approaches, you also may not get similar numbers, especially on high-latency connections where Ookla’s multiple streams will help hide latency.

Netflix’s Fast.com is deliberately simple. Open the page, and it immediately begins transferring data from Netflix infrastructure. By default it emphasizes download performance, since its original purpose was to answer a practical question: can this connection deliver Netflix video properly? Selecting “Show more info” adds upload speed and both unloaded and loaded latency.

The use of Netflix servers is significant. Fast.com measures the route between you and Netflix’s content-delivery network, while Ookla may test against a server operated by your ISP only a few network hops away. A superb Ookla result and a poor Fast.com result do not prove deliberate throttling, but they do tell you that the destinations — or the routes to them — are behaving differently.

Cloudflare offers two related tests. Its Radar Network Quality Test provides a quick summary, while speed.cloudflare.com  gives an extremely detailed breakdown. The latter reports download and upload throughput, idle and loaded latency, jitter, packet loss, server location, and application-oriented quality estimates.

Loaded latency is especially useful. An otherwise fast connection can become miserable when a large upload or download fills an oversized queue in the modem or router. Your idle ping might be 12 milliseconds, but under load it may jump to several hundred milliseconds. That is the classic symptom usually called bufferbloat.

If you want more options, there is testmy.net, which allows you to test upload and download speeds separately, and speedof.me, which keeps a history for you, among others. It isn’t always obvious which ones are measuring a single connection vs multiple ones, so you may have to dig through whatever documentation you can find.

Your WiFi Is Part of the Test

A browser speed test cannot automatically tell you what’s hurting your speed. A laptop connected through marginal WiFi may report 180 megabits per second even though the router has a flawless gigabit Internet connection.

In fact, once incoming Internet service reaches several hundred megabits per second, WiFi is frequently the limiting factor. The link rate displayed by the operating system is not the same thing as usable throughput. Wireless protocols have framing overhead, acknowledgments, contention, retransmissions, and half-duplex operation. The advertised 866, 1200, or 2400 megabit link rate is therefore not a promise that application data will move at that rate.

The numbers printed on WiFi boxes add another layer of optimism. A router sold as “AC1800,” for example, does not provide an 1800-megabit connection to one device. The figure is normally the sum of the maximum advertised PHY rates on separate radios — perhaps 1300 Mb/s on 5 GHz plus 450 Mb/s on 2.4 GHz — with some rounding for marketing. A conventional WiFi client connects to one band at a time, so it cannot combine those rates. The total is better understood as the router’s theoretical aggregate capacity while serving multiple devices across both bands. Even then, protocol overhead, contention, signal quality, and client limitations make actual data throughput considerably lower. Newer WiFi 7 equipment can sometimes combine links using Multi-Link Operation, but that exception does not make the old ACxxxx arithmetic any less misleading.

WiFi also uses shared airtime. Devices on the same channel — including neighboring access points that can hear one another — must contend for opportunities to transmit. A slow or distant client takes longer to send a given amount of data and can consume disproportionate airtime while doing so. Modern access points may provide airtime fairness and other mitigations. One old device does not invariably drag every client down to its rate, but it can still reduce the capacity available to the rest of the network. Interference has a similar effect. A weak signal, a crowded channel, microwave noise, or an overlapping neighboring network causes frames to be delayed or retransmitted. Those retries consume airtime without delivering additional data.

Repeaters and wireless mesh backhaul add another complication. A simple same-channel repeater must receive each packet and then transmit it again over the same shared medium. In the worst case, each repeated hop can roughly halve the available throughput. Modern tri-band mesh systems can avoid much of that penalty by using a dedicated backhaul radio, and Ethernet backhaul avoids it almost entirely.

This means it is entirely reasonable to buy gigabit Internet service and obtain only 300 or 500 megabits per second from a WiFi laptop. Whether that represents a problem depends on the client, radio band, channel width, signal level, backhaul, and local RF environment.

For a meaningful ISP test, begin with a computer connected directly to the router by Ethernet. Stop large transfers and temporarily disable any VPN. Record the chosen server, latency, upload speed, and download speed rather than preserving only the most flattering number. Then run the same tests over WiFi. The difference is an approximate measurement of what the wireless portion of the network is costing you.

Remove the Internet From the Experiment

Better still, remove the ISP from the test completely. OpenSpeedTest is a self-hostable, browser-based test. Run its server on a wired computer, NAS, or container, then visit it from laptops, phones, and tablets around the house. Because the traffic remains on your LAN, a slow result points toward WiFi, switching, cabling, or the client rather than the Internet connection.

It is possible to run this on the uhttpd server used with OpenWRT, although you’ll need to coax it to measure upload speeds since the server can’t handle the default method. The trick is to create a CGI script that accepts a large amount of data successfully and then configure uhttpd to run that.

A browser-based local test is convenient, but for serious diagnosis it is hard to beat iperf3, the client/server tool we recently used while testing mesh routers. On one machine (say, 192.168.1.100), start the server:

iperf3 -s

From another machine, run:

iperf3 -c 192.168.1.100

By default, iperf3 uses one TCP connection. Add -P 4 to try four parallel streams, or -R to reverse the direction so that the server sends and the client receives. Those variations can tell you something. If four streams are much faster than one, the network may have enough aggregate capacity but a single TCP flow is being limited by latency, loss, window growth, CPU performance, or offload behavior. If the reverse test is much faster, examine the weaker machine’s transmit path, drivers, antennas, or CPU.

iperf3 can also generate UDP traffic at a specified rate and report packet loss and jitter. That is often more informative for evaluating a wireless link than merely chasing the largest TCP number.

Can Linux Make It Faster?

Linux offers an impressive array of network tuning knobs, which naturally tempts us to turn them. But first, you need to understand what needs tweaking.

Check the negotiated Ethernet rate and interface counters:

ethtool eth0
ip -s link show eth0

A gigabit adapter that has negotiated 100 megabits per second usually has a cabling, connector, or switch-port problem. Increasing TCP buffers will not repair it. Rising interface errors and drops point toward a physical, driver, or congestion problem. TCP retransmits (view with ss -ti) may indicate loss elsewhere on the path.

You can inspect the active queue discipline with:

tc qdisc show

Linux supports queue disciplines such as fq_codel, which combines per-flow queueing with active queue management. It attempts to prevent one large transfer from building an enormous queue and delaying unrelated interactive packets. The kernel documentation specifically lists fq_codel as a sensible queue discipline that works without extensive configuration.

It can be selected as the default for newly created interfaces with:

sudo sysctl -w net.core.default_qdisc=fq_codel

That may improve queueing on traffic leaving the Linux machine. It does not, however, fix a large queue in the cable modem or Internet router. Queue management must be applied at the bottleneck. If the ISP link is limited to 20 megabits upstream, controlling a queue on a gigabit Ethernet interface after it has already handed packets to the router is too late.

For a home connection, the most effective bufferbloat treatment is usually Smart Queue Management on the router. OpenWrt’s SQM system supports both fq_codel and CAKE. CAKE generally provides better performance. However, fq_codel requires less CPU overhead.

High-latency paths introduce a different problem. TCP must keep enough data in flight to fill the bandwidth-delay product. Modern Linux generally autotunes TCP buffers, so the old advice to assign enormous fixed values to tcp_rmem and tcp_wmem is less universally useful than it once was. Before changing them, use ss -ti during a transfer and look for retransmissions, round-trip time, congestion-window size, and whether the receiver window is actually limiting the connection.

Linux also supports selectable TCP congestion-control algorithms:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control

Algorithms such as BBR can improve throughput and queue behavior on some long-distance or lossy paths. But changing the algorithm affects connections sent by that Linux machine; it does not control the remote speed-test server, repair poor WiFi, or eliminate a queue in the router. Congestion-control tuning is therefore a useful experiment for a server, VPN endpoint, or long-haul transfer machine — not a universal solution to slow networking.

Finally, inspect hardware offload features when a Linux system cannot keep up with a fast LAN:

ethtool -k eth0

Advanced network tuning is a bit beyond the scope of this post, but there are plenty of roadmaps down this rabbit hole.

The Lesson

The lesson here is that there is no universally correct speed-test result. Ookla tests how effectively multiple transfers can fill a route to one of its servers. M-Lab examines a single bulk flow. Fast.com tests the path to Netflix. Cloudflare pays unusual attention to latency under load and overall connection quality. OpenSpeedTest and iperf3 can determine whether the Internet connection is even the problem.

Run enough tests, and you will eventually obtain a number worth bragging about. Run the right tests, though, and you may find ways to truly increase real-world performance. If you want to chase that extra 1 kbit per second speed, be our guest — we know how it is. But the truth is that if the Internet is doing what you want it to do, then it is fast enough.

In the previous installment on UDP broadcasting and service discovery, the basics of both were explored, including an implementation in the form of NyanSD and its protocol. Contained in the comment section was a very good demonstration of why one of the most exciting aspects of software development is the opportunity to share your latest creations with other people. This being the ability to get solid feedback on all the points – including any potential boneheaded omissions – that you really should address, whether intentional or accidental.

The most pertinent point raised was definitely that of broadcast addresses and IPv4 subnets, with the latter topic especially being something that the sysadmins at the office would talk about all the time, but which us software developers were always happy to ignore as something that didn’t concern us. Turns out the joke was on me and everyone else – like our esteemed readers – who thought that they could escape the fascinating world of subnets, as today we’ll take an in-depth look at what subnets are and how they are relevant to the world of UDP network discovery.

I somewhat alluded in the first article to the topic of ‘which broadcast address to use’ as being somewhat of a rough topic to figure out, which is clearly why I just stuck to a blatantly ‘works for me’ /24 subnet that usually will work on networks, until it does not.

Subnet And Conquer

The short version of ‘what is a subnet’ is to point at the subnet mask that we have been mostly mindlessly mashing into networking configuration dialogs along with the IPv4 address for many decades now. Usually this takes the form of 255.255.255.0, which is just the human-readable version of the actual bitmask. Here the loopback interface already tends to use 255.0.0.0 as its netmask, which is a detail that tends to be easy to gloss over as this is just one of those local OS things.

Putting netmasks in the crudest and simplest terms, they are a bitmask that is used to identify how an IPv4 pool of addresses is split up by defining which bits of the 32-bit IPv4 address identify a subnet. Normally we call the trailing part of an IPv4 address (the .123) the host identifier, with the preceding section the network identifier.

By masking part of this host ID and using it to create a subnet identifier, we can then use this for additional routing, just at the cost of a reduced number of possible host IDs within that subnet.

As an example, the common 255.255.255.0 mask identifies the first 24 bits (3 bytes) of the 32-bit (4-byte) IPv4 address, hence the mask being referred to as /24. With this mask, the remaining host ID bits allow for 256 hosts, of which two are not used for hosts: the first (e.g. 192.168.0.0) and last (e.g. 192.168.0.255) in the range. The last host ID in the range forms the broadcast address for that subnet.

This is why, for a /24 subnet, you can generally get away with just slapping a .255 on the end of an interface’s address, but also why for other subnet configurations it’s likely to explode violently.

To get briefly back to the loopback’s /8 style netmask, this means a single subnet with a maximum of 16,777,214 hosts, which ought to be sufficient for local system networking shenanigans. Its opposite extreme would be the /31 style netmask, which with just two potential host IDs is practically useless.

IPv6 subnetting is similar, but due to the much larger address pool and differences in the protocol this is a whole other kettle of fish that is as likely to send a network administrator’s heart racing in excitement as it is to make the average software developer run away screaming. This can be a fun topic for another day, perhaps.

This overview of IPv4 subnetting also skips over details like the different classes of IPv4 subnets beyond the Class A type here, but those are happily left to sysadmins and kin for now.

Sub-casting

In order to thus obtain the broadcast address for a given network interface you need to know two things: the IPv4 address and its associated netmask. From this you can then tell three things: the subnet ID, the broadcast address in that subnet, and the current host ID. Of these we only really care about the the second item.

Although you can obtain the broadcast address yourself by applying the netmask to the address, the OS’s APIs tend to happily give you the precomputed broadcast address. If that’s not your style or not an option, a manual procedure is to:

1. Determine the number of host ID bits using the netmask.
2. Set all bits to 1 in these bits to get the highest possible host ID.
3. Use this value along with the original masked (i.e. network ID) bits to obtain the broadcast address.

If we thus start with a 192.168.0.0/24 network, we end up with 192.168.0.255, while for a 192.168.0.0/26 network with just six bits available the maximum value is 64, ergo we get 192.168.0.63, since we start counting at 0.

With this we can now broadcast UDP packets on any interface without any (major) worries.

Local Broadcast Address

A small glitch in the whole above story is that there’s actually another broadcast address, one which is always the same for each interface and can be considered to make the whole preceding explanation completely irrelevant. This being the local, or limited, broadcast address, which is either the best thing since sliced bread or the worst sin ever committed in the history of IP networking, depending on whom you ask.

This cheat code takes the form of the address 255.255.255.255 and if you send a packet on a UDP socket to it, you’ll get happy UDP responses from any service that is listening on the specified port. This raises the point of why you’d not just use this broadcast address on all interface, rather than bother with all the earlier described nonsense.

The only major difference between this local broadcast address and the earlier described directed broadcast address is that the latter can also be used to target a foreign network, instead of just the local network. This makes it a very attractive option if you just want to query the local network with UDP broadcast packets.

As for why you’d not want to use a local broadcast address, I couldn’t really find any references or citations on why this would be the case. Both would appear to be perfectly valid approaches to broadcasting, each with its own pros and cons.

Bugs

One final topic was my mistaken hardcoding of a /24 style broadcast address in NyanSD. Here reader ziew helpfully pointed me towards the Poco::Net::NetworkInterface::broadcastAddress() function, which seemed perfect. Unfortunately Poco’s implementation at least on Windows 10 appears to be rather broken.

After getting only 0.0.0.0 as broadcast address from this function, I had a bit of a look at what was happening, including checking what I got as subnet mask both for the default index parameter and for the next index. Across two different Windows 10 installations and both GCC in MSYS2 as well as MSVC 2017/2022 with various versions of Poco the returned values were… interesting enough to file a bug report on the Poco issue tracker.

Clearly this isn’t going to be fixed just yet, but on the bright side the horrific atrocity that I committed by hardcoding a /24 broadcast address will still work on basically every home LAN out there that NymphCast is likely to be used on.

Maybe I could just switch to a local broadcast address and that’d be even better. Feel free to torch down this idea in the comments, just be sure to provide solid reasoning and cite your sources.

A Complex Topic

Writing out the above pretty much clarifies I think why past me got a bit overwhelmed when trying to ‘just do a UDP broadcast thing’. Even just scratching the surface of IPv4 subnets and not even venturing into IPv6 territory makes one already feel a bit antsy.

Certainly, one could totally argue that anything other than a /24 network is unlikely to be encountered outside of certain government and business networks with either very specific needs, very enthusiastic sysadmins, or both, but it’s always better to design software with such real-life scenarios in mind.

I’ll admit it: I miss the simplicity of /etc/hosts. There was something elegant about it. You wanted laserprinter to mean 192.168.1.40, so you opened a text file and wrote:

192.168.1.40 laserprinter

Done. No cloud account, no discovery daemon, no dashboard with material-themed icons. Just a name and an address. The trouble, of course, is that /etc/hosts is only simple when you have one machine. The moment you have a desktop, a laptop, a Raspberry Pi, a NAS, a test box, and a phone or two, every little network change becomes a tiny distributed-database problem. Which copy of /etc/hosts is authoritative? Did you update the laptop? What about the machine you only boot once a month?

One Solution

Modern LANs solved this with mDNS, using Avahi on Linux. It resolves addresses that end in .local. Instead of asking a central DNS server “who is thing.local?”, a machine sends a multicast query on the local network: “who has thing.local?” The device that owns the name answers. This is why your Linux box named spock and usually be reached as spock.local on your LAN.

There are limits. mDNS is link-local; it is meant for the local LAN, not the whole Internet and shouldn’t route across subnets. Each device is supposed to publish its own name. That works fine when the device cooperates. But what about devices that do not publish mDNS? Or little embedded things that barely even have an IP address?

That is where I wanted the best of both worlds: keep a small authoritative /etc/hosts file on one Linux box, but publish selected entries onto the LAN using mDNS.

Publishing House

Avahi includes a handy tool for this:

avahi-publish-address widget.local 192.168.1.50

While that process is running, Avahi advertises widget.local as an mDNS address record. Kill the process, and the record goes away. So you could just write a script to publish all the addresses for things that won’t do it themselves and launch in in local.rc or a systemd unit. But that seems inelegant. I wanted to just pick things out of the /etc/hosts file. But not everything. Here is a simple publisher, installed as /usr/local/sbin/localip_pub:

#!/bin/sh
# Scan /etc/hosts for .local addressess and publish them using avahi

tmp="$(mktemp)"
pids=""

cleanup() {
   rm -f "$tmp"
   for p in $pids; do
      kill "$p" 2>/dev/null
   done
   wait
}


mdns_name_exists() {
   timeout 2 avahi-resolve-host-name -4 "$1" 2>/dev/null |
      awk -v h="$1" '
         BEGIN {
            sub(/\.$/, "", h)
            rc = 1
         }
      {
         n = $1
         sub(/\.$/, "", n)
      }
      n == h && $2 ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/ {
      rc = 0
   }
   END {
      exit rc
   }'
}

trap cleanup INT TERM EXIT

awk '
   NF < 2 { next } # skip short lines
   $1 ~ /^127\./ { next } # skip local address
   # This assumes  .local
   # it will reject anything with an alias or subdomain or trailing comment
   # it also rejects any full comment lines or IPv6 addresses
   # although you could certainly patch it for IPv6 if you wanted to
   /^[[:space:]]*[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+[[:space:]]+[^.]+\.local[[:space:]]*$/ {
      ip=$1
      name=$2
      if (!seen[ip]++) print name, ip
   }
' /etc/hosts > "$tmp"


while read name ip; do
   if mdns_name_exists "$name"
   then
      echo found $name
      continue
   fi
echo avahi-publish-address "$name" "$ip"
avahi-publish-address "$name" "$ip" &
pids="$pids $!"
done < "$tmp"
wait

The script is intentionally conservative. It ignores loopback, ignores IPv6, and only publishes single names that already end in .local. So consider this hosts snippet:

192.168.1.51 oscope.local
192.168.1.52 server.example.com

The script will publish oscope.local, but leaves server.example.com alone. That avoids accidentally dumping every fully qualified name in your hosts file into mDNS.

But Wait...

The wait at the end of the script matters. Each avahi-publish-address process has to stay alive for the record to remain published. If the shell script simply started them in the background and exited, systemd would consider the service finished and might clean up the child processes. By waiting, the script remains the service’s main process.

Here is the matching systemd unit:

<h1>/etc/systemd/system/localip_pub.service</h1>

[Unit]
Description=Publish selected /etc/hosts names via Avahi/mDNS
Requires=avahi-daemon.service
After=network-online.target avahi-daemon.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/localip_pub
Restart=on-failure
RestartSec=5s
KillMode=control-group

[Install]
WantedBy=multi-user.target

Install and start it:

sudo chmod 755 /usr/local/sbin/localip_pub
sudo systemctl daemon-reload
sudo systemctl enable --now localip_pub.service

After editing /etc/hosts, restart the publisher:

sudo systemctl restart localip_pub.service

Of course, if you want to use this, it has to be a machine that knows to query mDNS. Ironically, your workstation probably does, but if it is configured to use /etc/hosts first, that won't matter there. But it does lead to a practical annoyance: not every application on every platform uses mDNS the same way. Android itself only relatively recently grew better support for normal application-level mDNS resolution, so behavior can vary depending on Android version, app, and resolver path. Chrome on Android seems to resolve .local names correctly in my testing.

Firefox was a little more subtle. At first it looked like Firefox on Android simply did not understand mDNS, but the real culprit was Secure DNS. If Firefox is configured to use DNS-over-HTTPS, it may bypass the local resolver path that knows how to handle .local names. Turning Secure DNS off, or adding exceptions for the local names you care about, lets those names resolve normally. That is not really an Avahi problem; it is an application side effect. To be fair, even if you stood up a LAN DNS server to solve the problem, Firefox would have probably still have skipped it in favor of its "secure" DNS.

Conflicting Data

There is another gotcha: name conflicts are not the only kind of conflict. You might expect trouble if you try to publish widget.local when some other device already owns widget.local, but address conflicts can be troublesome too. If some device is already publishing mDNS information associated with a particular IP address, trying to publish a second name for that same address may fail. In my case, avahi-publish-address did not fail gracefully; it wandered into a collision path and crashed. The safe approach is to check before publishing, avoid duplicate names and duplicate addresses, and let the device itself publish its own mDNS name when possible.

Still, for a small LAN, this is a nice compromise. /etc/hosts remains the simple text file I wanted, but Avahi turns selected entries into something other machines can discover without copying that file everywhere. It is not a replacement for real DNS, and it is not quite as seamless as the old single-machine hosts file. But for those odd devices that sit quietly on the network with an IP address and no useful name, it is a handy little bridge between the old world and the new one. Were there other ways to solve this? There always are. But this works for me.

We've looked under the covers at mDNS. The system can be a lifesaver when you have too many Raspberry Pis.

If you’re building a homelab rig, you could just use off-the-shelf hardware in standard cases and slap it all in a rack like the normies do. Or, you could follow the example of [Justin Garrison] and build a more oddball setup.

This particular homelab is, at its heart, built from familiar components. There are two Raspberry Pi 5s, two Raspberry Pi 4s, a GMKtec NucBox M6 Mini with an ASUS GeForce RT 2060 GPU, a LattePanda IOTA, an NVidia DGX Spark, and an HP Z4 G4 mini PC. These machines are all laced together with a TP-Link LS108GB PoE switch. [Justin] has the mini PC running the control plane components, with the rig as a whole running Talos and Kubernetes workloads. What makes this build particularly appealing, though, is the aesthetics of the rig. [Justin] documents how he hacked this hardware to fit into a bunch of old Linksys router cases, which provides a pleasant early 2000s look to the build. This included a bit of hackery to get status LEDs flickering as they should be. [Justin] also took the time to make the power buttons accessible.

If you want to stunt on your friends with a rad homelab, you either have to go for maximum power, or maximum style. This build would be the latter. Video after the break.

If you asked us to build you a KVM last week, we’d likely have reached for a Raspberry Pi. Now, thanks to [JonathanRowny], we’d seriously consider an ESP32-P4, because his IP KVM seems pretty capable.

He’s using the P4 hardware to its fullest, getting the supported 1080p graphics, and doing so in an interesting way– he’s got a commercial adapter board to try and translate HDMI signals to the camera input on his dev board. Conveniently enough, it’s the same ribbon-cable pinout as the RPi, which is not guaranteed by the CSI standard. Writing a driver to take that signal proved the hardest part– aside from the usual chip revision confusion that plagues this chip– and we can’t help but wonder if the client on the other side of the KVM-IP link might have an easier time doing the image processing that was required for a good image. Regardless, he’s got the code as it is now up on GitHub under the Apache license. 

As of this this writing, there’s no audio, and ironically for an ESP32 project networking is wired-only– but much more importantly, there is no security. So it’s a work in progress, but great to see the P4 in the wild doing something other than emulation. Not that we haven’t seen the P4 at work before–the Tanmatsu handheld also makes use of Expressif’s most powerful chip for a handy little terminal. Between the KVM and the handhelds, we cannot help but wonder how many of the projects that were once the provenance of a Pi will get squeezed into these overpowered microcontrollers. Sure, they can’t even match the original Pi in horsepower, never mind a modern Pi5, but how many times have you seen a Linux SBC seriously under-taxed in a project like this?

If you’re swapping Pi for P4– or doing anything else interesting– please let us know on the tips line.

Local area networks (LANs) that use technologies like Ethernet and Wi-Fi are incredibly useful for letting devices talk with each other. Yet a core problem here is knowing which devices are where on the network, as anyone who has ever tried to add a network printer or network share to their system can probably attest to. Unless you happen to know the IP address of the LAN device, the port, and protocol, the target device may as well be located on the Moon without further help, such as automatic network discovery in lieu of waddling over to the device and reading the label listing its IP address.

Over the decades quite a few ways have been developed to enable such network discovery, with many of them using UDP broadcast as the first step. By broadcasting a global message on the entire LAN, any device that has an actively listening UDP socket on that particular port can parse said message and decide whether it’s feeling sociable enough to reply.

The topic of UDP broadcasting is however not as straightforward as it may sound if you’re just getting started, including the existence of many opinions on the ‘right way’. There is also a massive divide between a sprawling service discovery protocol like mDNS and a light-weight one like that one that I had to implement a few years ago for an open source project.

Network Broadcasting

The obvious advantage of a broadcast message is that a client device that seeks its protocol soul mate on the LAN doesn’t need to ping all possible IP address and subnets. Instead,  a broadcast message is designed so that all connected networking devices know that it should be forwarded to all other known devices. Thus with a single message from the client, in theory, only a single message will then neatly land at every single other connected system.

Of course, this ignores happy joy fun things such as convoluted network configurations, such as those involving overlapping Wi-Fi repeaters and subsequent routing, but in general we can assume that this is how it works. Various edge cases and fascinating complications of these will be considered in a later section.

Much of this service auto-discovery is tossed under the header of ‘zero-configuration networking‘, or zeroconf for people who don’t like typing. The best part about zeroconf is probably that there are so many standards here, ranging from DNS-SD to mDNS, UPnP, SLP and others. Perhaps unsurprisingly, one of the major issues here is that platform support here is spotty, with mDNS – despite being one of the most universal – not having much support outside of MacOS/OS X with Bonjour and Linux/BSD with Avahi.

Thus while trying to add the auto-discovery of NymphCast receivers and media servers by NymphCast clients, I found myself asking the daunting question of whether I was at risk of being about to embark on reinventing the proverbial wheel. After all, nobody wants to become the subject of an xkcd comic.

UDP Discovery Basics

As it turns out, I ought not to have been too worried, as despite looking everywhere I could find nothing along the lines of the NyanSD network service discovery (NSD) protocol that I ended up implementing and integrating into NymphCast. What I wanted after all was the most no-frills NSD possible that could be easily integrated, while working the same across just about any desktop, server and embedded platform imaginable.

All that’s needed for this is a way to create an appropriate UDP socket, and a way to either broadcast a query and receive the response, or to listen for incoming UDP packets. Here you can figure out the platform-native method for each target platform, or not reinvent the wheel and use an existing networking library for C++ like Poco. This is what I used for NyanSD, along with my ByteBauble utility to handle endianness conversions.

For the UDP server — the listening side — the procedure is fairly standard, with a regular UDP listening socket. As UDP is a connectionless protocol, there is not a lot of preamble here, just a UDP socket instance (here Poco::Net::DatagramSocket), which is bound to the target port and regularly polls for any fresh UDP packets to process. This can all be seen in the single source file for NyanSD which covers both the client and server side code.

Where things get spicy is with the client that sends the broadcast query and waits for any replies. If we were to just shove the query data into the socket along with the request to toss it over to a regular IP address, not a lot would happen. To make it into a broadcast request we need a few things:

1. Let the network subsystem know that we want to do broadcast things.
2. Create the special broadcast address for the target network interface.

With Poco the first point is easily handled by simply calling setBroadcast(true) on the UDP socket instance. For BSD sockets this sets the appropriate flag on the socket, which is essentially repeated across all OS implementations due to how prevalent the BSD socket library is.

The second point can be summarized for IPv4 as a curt ‘make it end with .255’. For example 192.168.0.255 when the client network interface’s IP address is 192.168.0.42. If there are multiple interfaces on the client system, you can go through the list one by one to broadcast on each of them before filtering out potential duplicate returns.

As for how to do broadcasting with IPv6: you don’t, as this protocol relies on multicast and special multicast receiver groups, which is another kettle of fish and of not much relevance for LANs.

Complications

If you look at the NyanSD API, it may give the impression that the query process is incredibly straightforward, with the sendQuery() function neatly returning a stack of remote systems that responded to our query. While these are definitely all the responses, it’s important to remember that NyanSD queries every single network interface. This means that the responses are likely to contain duplicates, which may even come from the loopback address when a service runs locally.

The filtering of this is captured in the NymphCast client library (libnymphcast) where the findServers() function in the main source file calls the isDuplicate() and isDuplicateName() functions, as well as the removeLoopback() function that nukes any responses that match a remote service found via a non-loopback interface. This last filtering is essential for NymphCast when e.g. using playback groups that would otherwise get confused by a stray loopback address.

Although one may think that such in-depth filtering is unnecessary if all you have is a single Wi-Fi or Ethernet interface in your system, one of the curveballs that I encountered during real-life testing was apparently related to Wi-Fi repeaters. For some reason it seems that the way that the repeaters did their broadcasting led to erroneous duplication of packets and thus multiple returns from a single system.

Depending on your exact use case and network configuration you may encounter any such issues and perhaps an exciting new one.

NyanSD Findings

Over the years that NyanSD has been used in the NymphCast project, it has proven to be one of the most reliable and probably nearly zero-fuss components. I have so far used it on Windows, various Linux distributions, FreeBSD, Haiku, Android, and the ESP32 via FreeRTOS and ESP-IDF. What this experience has proven to me most of all is that service discovery doesn’t have to be complicated.

The basic UDP protocol is simple and reliable enough that, barring a very sick LAN, there shouldn’t be any issues here. Assuming you get your filtering sorted of the responses, it’s probably the last part of a project to worry about.

One thing that I’m also very happy with in NyanSD is that there’s no set port in the protocol, like how mDNS always uses port 5353. What this means is that I can have NyanSD listen with a UDP socket on the same port as the NymphCast server’s TCP socket, which also means that different services with their own port can be targeted directly rather than every NyanSD-enabled service on the network getting blasted by every NyanSD query.

I did also do some work on a NyanSD daemon as a more central services database, but so far I have had no real need for it in a practical deployment. I guess that such a thing could be very useful if the port of a service is not set in stone, but generally that’s the one aspect of network services that tends to be boringly predictable.