In a recent post, I mentioned that I wanted to build some tools for a stripped-down Linux running on a 3D printer with a MIPS CPU. I had two options: build a toolchain to cross-compile, or use Zig, which, in theory, has built-in toolchains for MIPS. I had to jump through hoops to get Zig to work, and I did mention Crosstool-Ng, so you might wonder why I didn’t start there. Turns out, it had its own set of hoops to work through.

What is Crosstool-Ng?

Crosstool-NG is a build system for making cross-compilation toolchains: compilers, assemblers, linkers, C libraries, kernel headers, and all the other pieces needed to build software on one machine that will run on a different kind of machine. Instead of manually matching a particular GCC version with the right binutils, glibc, or musl release, Linux headers, patches, and configuration options, you select the target architecture and let Crosstool-NG download, patch, configure, and build the stack. The result is a self-contained toolchain with commands such as mipsel-linux-musl-gcc or arm-none-eabi-gcc, ready to produce binaries for the target system.

The four-part name is in a particular format that is often used in the cross compiling world. For example, consider arm-none-eabi-gcc. The tool here is gcc and, as you might expect, there will also be arm-none-eabi-as and arm-none-eabi-ld, among other things. The first part, arm in this case, will be the target architecture.

The second part of the name can mean a few different things. In theory, it is a vendor name but it is sometimes “none” which often means “generic” or, in the case of a linux target, “linux,” which isn’t technically a vendor.

The third part is the calling convention and, often, some idea of the library. For example, arm-linux-gnueabihf-gcc would mean the GNU library using the ARM EABI and hardware floating point. These are sometimes called “target triples” because, historically, it was CPU-VENDOR-OS, but now there are usually four or even five parts if the calling convention includes the OS, like linux-musl, for example.

That sounds simple, but cross-toolchains are unusually sensitive to version combinations and ABI details. Endianness, floating-point conventions, instruction-set variants, threading support, and C library choices all have to agree. So saying “Arm” or “MIPS” doesn’t mean much. You need to account for all the possible variations in the CPU and the libraries. Crosstool-NG does not eliminate those decisions, but it turns them into a reproducible configuration rather than a long sequence of hand-built components. I had two problems that I eventually resolved.

Problem One: Versions

One nice thing about Crosstool-Ng is that it pulls the right versions of everything for you. The problem is, when you install it from your system repositories, you are probably getting a crazy old version of the tool itself. I couldn’t find the right entries in the configuration when I did that, so I eventually uninstalled and picked up the latest version right from the source.

If that was the only problem, I would have been lucky.

Problem Two: Infinite Combinations

The CPU on the printer is an odd bird. As I noted last time, the executables use the r2 instruction set but also use the nan2008 convention which is usually found in r6. While Crosstool-Ng is good at letting you specify exactly what you want, it isn’t always clear on how you specify every detail.

To be fair, just like with Zig, some of that may be on me. I don’t use Crosstool-Ng or Zig every day, so maybe I was making either or both of them too hard. The bad news: It took me three or four attempts to get the right toolchain. The good news: It was a lot easier than manually downloading a bunch of stuff, trying to fix it up, building it, and still having to do it three or four times.

Configuration

Sort of like buysbox or building a custom kernel, the configuration for Crosstool-Ng uses the command: ct-ng menuconfig. This gives you a menu where you can set options about what you want and where you want it stored.

The problem is that the nan2008 setting I needed isn’t part of a standard mips32r2 setup. I suspect that if I had needed mips32r6, everything would have just worked. But, of course, I’m not that lucky.

In the target settings, I needed to match all the specifications, of course, but I also needed to add -mnan=2008 to both the CFLAGS and LDFLAGS as you can see in the figure.

So what’s so hard about that? Just those changes won’t produce a working toolchain for my printer. The C compiler also needed --with-nan2008 (in the C Compiler options screen under extra target CFLAGS) and the same option needed to be placed in the C Library screen, too.

Of course, it is like a word search puzzle. Once you see the answers, they look obvious. But when you are searching through pages of options, it is easy to miss one. It isn’t like there is a checkbox for “Use nan2008” that does it all for you because using nan2008 with mips32r2 is “strange.”

The Proof is in the Build

Once everything was set correctly, I was able to produce a toolchain (ct-ng build) that could compile busybox and even a small text editor. Everything ran fine on the printer.

To build busybox, I used:

make V=1 CC="mipsel-unknown-linux-musl-gcc -march=mips32r2 -msoft-float -static -Os" STRIP='mipsel-unknown-linux-musl-strip' -j6

Unlike Zig, no patching needed. The Zig version was about 9 kB larger than this version, so not much different there. Both were just over a megabyte total. I could probably have used hardware floating point to get a smaller executable, but given that I don’t think any of this is using much floating point at all, it didn’t seem to matter very much.

I had also threatened to compile a text editor. Turns out most have dependencies on things like ncurses, which are a pain to bundle. So I grabbed a copy of the tutorial editor kilo and extended it to look a little like emacs. Works great. Great place to start if you need a static editor that doesn’t take much space.

Lesson Learned

If the CPU on the printer had been more conventional, I think either approach would have worked fine. I prefer the Crosstool solution in this case, because I’m not lying by patching the ELF header. In this case, I don’t think that lie hurts anything, but a program that did a lot of floating-point math might not work correctly, whereas I think the one produced by Crosstool would be fine even for a floating-point program.

On the other hand, like most Unix and Linux things, there are always more ways to solve any problem. If your problem is wedging executables on an alien Linux box, there are two perfectly fine ways to solve it.

What if you want to do something in Linux for a lot of files? [Numerator] was tired of using xarg and other ways to handle this job and created bashumerate.

Some examples in the post of the “other ways” include:

find . -name '*.log' | xargs rm
find . -name '*.sh' -exec wc -l {} \;

You can, also, use a for loop, and if you are a programmer at heart, you may well do this:

for f in *.txt; do
  wc -l "$f"
done

Bashumerate handles all of the common cases in one tool and uses the same syntax for multiple kids of enumerations.

For example, the above translates to:

enumerate -f '*.sh' -- 'wc -l {}'

The -f means enumerate files. You can also enumerate lines, numbers in a range, or lists. What’s even more interesting is that you can add your own source. As an example, there’s an add-in function that enumerates running docker containers.

The source is all in bash, so it should be very portable. Will you try it? What’s your favorite way to enumerate in shell? Let us know in the comments. You know how we love strange bash tricks.

Having Linux on so many devices is both a blessing and a curse. Sure, it is great that you can hack on things and modify them or even totally repurpose them. But it also means you have a fleet of Linux devices you have to manage and keep track of.

My current “main” 3D printer is a Flashforge AD5X: a nice, cheap machine that does four colors with the purge/exchange method. It sort of runs Klipper. I say sort of because Flashforge has Klipper running on a Linux host in the box, but it is massively crippled and modified. I’m sure it works for most folks. I’m also sure that if you know nothing about Linux, Klipper, or 3D printing, the experience is probably better thanks to all the cloud point-and-click interfaces. But, of course, I check none of those boxes.

I’ve had the printer for probably a year or more. Almost immediately, I put a “mod” on the printer to give it a more true Klipper interface and gave me things like shell access. There are several that I think will do this, but I used Zmod, which doesn’t totally replace the printer’s firmware; it just sort of patches it and extends it. You can easily bypass or even remove it and go back to the stock printer, although I would not want to.

In my case, the issue was a printer, but the same idea might apply to any embedded Linux system, from a router to a thermostat. Sure, it runs Linux, but is it Linux you can change?

The Problem

The Flashforge firmware and Zmod both will run on the AD5X’s little sister, the AD5M. However, the AD5M has a significantly less capable processor board than the AD5X. That means that Linux on the boxes is very stripped down. From Flashforge’s point of view, no one should be in the Linux OS anyway, and the author of Zmod probably figures every byte used is a byte taken away from the user or other advanced Zmod features.

It may seem like a first-world problem, but there were two things that irked me about the printer’s Linux. There was no less or more command for poking around files. There was also only vi as an editor. I did a few hacks to make myself happy. I wrote a pager in shell script, for example. I would try to remember to use my desktop emacs and tramp to edit files on the box. But it was a shame that there were some very basic tools lacking. Besides that, even the tools that were there like ls lacked help commands in case you want some strange option you can’t remember.

No Install

To save space, the printer doesn’t really have programs like ls, cat, and grep. Instead, it has a single busybox executable. This is common on small systems. You get one copy of the libraries and a single executable that will do all the work you need. You can invoke, for example, grep by running “busybox grep” or, if you make a symlink to busybox named grep, the user may never realize that you don’t really have grep installed.

However, busybox has to be built. You can’t easily install packages to it. So I could just run some package manager and install less or anything else. My plan was to produce a new busybox package myself to supply at least the missing commands and maybe some of the more basic ones, too. How hard could it be?

How Hard, Indeed

The first issue was figuring out exactly what the printer was running. The OS was a buildroot compile as shown by /etc/os-release:

NAME=Buildroot
VERSION=2020.02.1-g0b1f992-dirty
ID=buildroot
VERSION_ID=2020.02.1
PRETTY_NAME="Buildroot 2020.02.1"

In theory, you could probably try to rebuild everything from that information, but it seemed like a bad idea. Who knows what changes they’ve made or what strange dependencies were out there?

The next piece of the puzzle was the architecture. It turned out to be MIPS, which has many variations, a problem that would come back to haunt me later. To figure it all out, I grabbed busybox from the machine and copied it to my regular computer. This let me read the elf file:

# readelf -h busybox
ELF Header:
Magic:   7f 45 4c 46 01 01 01 00 03 00 00 00 00 00 00 00
Class:                             ELF32
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       3
Type:                              EXEC (Executable file)
Machine:                           MIPS R3000
Version:                           0x1
Entry point address:               0x403fa0
Start of program headers:          52 (bytes into file)
Start of section headers:          775880 (bytes into file)
Flags:                             0x70001405, noreorder, cpic, nan2008, o32, mips32r2
Size of this header:               52 (bytes)
Size of program headers:           32 (bytes)
Number of program headers:         10
Size of section headers:           40 (bytes)
Number of section headers:         30
Section header string table index: 29

Forensics

Just knowing that the CPU was MIPS was hardly enough. Note that the processor could have been big endian, but this one was little endian. What’s more, the flags show a few things, and the important ones would turn out to be mips32r2, which defines a particular set of instructions, and nan2008, which means everything is using a more modern floating point stack compared to some older MIPS setups.

The next step is to find a toolchain that can run on my Linux box and produce executables for such a machine. There were two main candidates that I thought about: zig and crosstool-ng.

Zig

You may remember that Zig is a completely different language from C. So it might not make sense that I want to compile busybox, a C program with Zig. However, Zig has an interesting feature. It can pretend to be gcc, and maybe compilers for some other languages too. The idea is that if you have a lot of C code, you could start changing over to Zig without having to convert everything today or even ever. Using CLang and LLVM, Zig purports to be able to just drop in a project as the system C compiler.

So why does that matter for this project? Zig also supports a lot of targets, including MIPS, and provides run time libraries for them. So if I just tell the build system to use Zig, everything will be fine, right?

Right…

Naturally, this didn’t work as well as I expected. It could be that I don’t understand Zig well enough, but passing --target mipsel-linux-musleabihf -mcpu=mips32r2 to zig cc couldn’t produce a working ‘hello world’ program. The reason? Zig appears not to supply a nan2008 library for MIPS. Or maybe I just didn’t know how to enable it. As far as I can tell, Zig might have done the right thing for mips32r6, but that, unsurprisingly, led to illegal instruction problems. Besides, I wanted to stick as close to the existing setup as possible.

I could have made Zig rebuild its libraries, but then I might as well build a full toolchain. However, I wondered if floating point would matter much for what I was doing, so I decided to cheat. I let Zig build an executable, and then I simply patched the output header to say it used nan2008. It worked for a simple program.

Busybox

Configuring busybox is a matter of using a menuconfig system similar to building a kernel. I mainly asked it to make a static copy in options and then just picked a bunch of things I wanted, especially less and more.

Many programs, including less, have options to help you minimize the size. For example, you could disable regular expressions or make it not read $LESS for options. I left nearly everything on for most of the programs.

Once it was ready to go, I needed to run make with the Zig compilers and then do the patch to fool the printer into running it.

The command line that worked the best turned out to be: make V=1 CC="zig cc -target mipsel-linux-musleabihf -mcpu=mips32r2 -static -Os" STRIP='llvm-strip' -j6

Of course, you could adjust the -j6 to suit how many CPUs you have. I like to use about 1/2 of mine when compiling. Then the patch is as simple as:

printf '\024' | dd of=busybox bs=1 seek=37 count=1 conv=notrunc

Did it Work?

Once I transferred the file, renamed to busybox-ad5x, over to the printer, I was able to successfully run things like busybox less or even busybox ls. Everything seemed to work. You can ask busybox to install a bunch of symlinks for everything it knows how to do, but I wanted to start small, so I only made symlinks for things that didn’t already exist in the native busybox.

Once I didn’t find problems, I eventually replaced even the old busybox utilities with the new ones. I even made busybox-ad5x a symlink and renamed this version as busybox-zig because I wanted to try one more experiment: using crosstool-ng to make a proper toolchain.

Next Time

But that’s a whole other topic for another day. For now, Zig — with a little patching — got the job done. There are many ways to get things done under Linux. Turns out it is very hard for a vendor to totally lock down a system, and if there is a way in, then there is almost always some way to tune things to your liking. Busybox is a great tool for stripped-down systems. Even floppy-based boots.

I’ll admit it: I miss the simplicity of /etc/hosts. There was something elegant about it. You wanted laserprinter to mean 192.168.1.40, so you opened a text file and wrote:

192.168.1.40 laserprinter

Done. No cloud account, no discovery daemon, no dashboard with material-themed icons. Just a name and an address. The trouble, of course, is that /etc/hosts is only simple when you have one machine. The moment you have a desktop, a laptop, a Raspberry Pi, a NAS, a test box, and a phone or two, every little network change becomes a tiny distributed-database problem. Which copy of /etc/hosts is authoritative? Did you update the laptop? What about the machine you only boot once a month?

One Solution

Modern LANs solved this with mDNS, using Avahi on Linux. It resolves addresses that end in .local. Instead of asking a central DNS server “who is thing.local?”, a machine sends a multicast query on the local network: “who has thing.local?” The device that owns the name answers. This is why your Linux box named spock and usually be reached as spock.local on your LAN.

There are limits. mDNS is link-local; it is meant for the local LAN, not the whole Internet and shouldn’t route across subnets. Each device is supposed to publish its own name. That works fine when the device cooperates. But what about devices that do not publish mDNS? Or little embedded things that barely even have an IP address?

That is where I wanted the best of both worlds: keep a small authoritative /etc/hosts file on one Linux box, but publish selected entries onto the LAN using mDNS.

Publishing House

Avahi includes a handy tool for this:

avahi-publish-address widget.local 192.168.1.50

While that process is running, Avahi advertises widget.local as an mDNS address record. Kill the process, and the record goes away. So you could just write a script to publish all the addresses for things that won’t do it themselves and launch in in local.rc or a systemd unit. But that seems inelegant. I wanted to just pick things out of the /etc/hosts file. But not everything. Here is a simple publisher, installed as /usr/local/sbin/localip_pub:

#!/bin/sh
# Scan /etc/hosts for .local addressess and publish them using avahi

tmp="$(mktemp)"
pids=""

cleanup() {
   rm -f "$tmp"
   for p in $pids; do
      kill "$p" 2>/dev/null
   done
   wait
}


mdns_name_exists() {
   timeout 2 avahi-resolve-host-name -4 "$1" 2>/dev/null |
      awk -v h="$1" '
         BEGIN {
            sub(/\.$/, "", h)
            rc = 1
         }
      {
         n = $1
         sub(/\.$/, "", n)
      }
      n == h && $2 ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/ {
      rc = 0
   }
   END {
      exit rc
   }'
}

trap cleanup INT TERM EXIT

awk '
   NF < 2 { next } # skip short lines
   $1 ~ /^127\./ { next } # skip local address
   # This assumes  .local
   # it will reject anything with an alias or subdomain or trailing comment
   # it also rejects any full comment lines or IPv6 addresses
   # although you could certainly patch it for IPv6 if you wanted to
   /^[[:space:]]*[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+[[:space:]]+[^.]+\.local[[:space:]]*$/ {
      ip=$1
      name=$2
      if (!seen[ip]++) print name, ip
   }
' /etc/hosts > "$tmp"


while read name ip; do
   if mdns_name_exists "$name"
   then
      echo found $name
      continue
   fi
echo avahi-publish-address "$name" "$ip"
avahi-publish-address "$name" "$ip" &
pids="$pids $!"
done < "$tmp"
wait

The script is intentionally conservative. It ignores loopback, ignores IPv6, and only publishes single names that already end in .local. So consider this hosts snippet:

192.168.1.51 oscope.local
192.168.1.52 server.example.com

The script will publish oscope.local, but leaves server.example.com alone. That avoids accidentally dumping every fully qualified name in your hosts file into mDNS.

But Wait...

The wait at the end of the script matters. Each avahi-publish-address process has to stay alive for the record to remain published. If the shell script simply started them in the background and exited, systemd would consider the service finished and might clean up the child processes. By waiting, the script remains the service’s main process.

Here is the matching systemd unit:

<h1>/etc/systemd/system/localip_pub.service</h1>

[Unit]
Description=Publish selected /etc/hosts names via Avahi/mDNS
Requires=avahi-daemon.service
After=network-online.target avahi-daemon.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/localip_pub
Restart=on-failure
RestartSec=5s
KillMode=control-group

[Install]
WantedBy=multi-user.target

Install and start it:

sudo chmod 755 /usr/local/sbin/localip_pub
sudo systemctl daemon-reload
sudo systemctl enable --now localip_pub.service

After editing /etc/hosts, restart the publisher:

sudo systemctl restart localip_pub.service

Of course, if you want to use this, it has to be a machine that knows to query mDNS. Ironically, your workstation probably does, but if it is configured to use /etc/hosts first, that won't matter there. But it does lead to a practical annoyance: not every application on every platform uses mDNS the same way. Android itself only relatively recently grew better support for normal application-level mDNS resolution, so behavior can vary depending on Android version, app, and resolver path. Chrome on Android seems to resolve .local names correctly in my testing.

Firefox was a little more subtle. At first it looked like Firefox on Android simply did not understand mDNS, but the real culprit was Secure DNS. If Firefox is configured to use DNS-over-HTTPS, it may bypass the local resolver path that knows how to handle .local names. Turning Secure DNS off, or adding exceptions for the local names you care about, lets those names resolve normally. That is not really an Avahi problem; it is an application side effect. To be fair, even if you stood up a LAN DNS server to solve the problem, Firefox would have probably still have skipped it in favor of its "secure" DNS.

Conflicting Data

There is another gotcha: name conflicts are not the only kind of conflict. You might expect trouble if you try to publish widget.local when some other device already owns widget.local, but address conflicts can be troublesome too. If some device is already publishing mDNS information associated with a particular IP address, trying to publish a second name for that same address may fail. In my case, avahi-publish-address did not fail gracefully; it wandered into a collision path and crashed. The safe approach is to check before publishing, avoid duplicate names and duplicate addresses, and let the device itself publish its own mDNS name when possible.

Still, for a small LAN, this is a nice compromise. /etc/hosts remains the simple text file I wanted, but Avahi turns selected entries into something other machines can discover without copying that file everywhere. It is not a replacement for real DNS, and it is not quite as seamless as the old single-machine hosts file. But for those odd devices that sit quietly on the network with an IP address and no useful name, it is a handy little bridge between the old world and the new one. Were there other ways to solve this? There always are. But this works for me.

We've looked under the covers at mDNS. The system can be a lifesaver when you have too many Raspberry Pis.

Among the many forgotten might-have-beens of the games console world, the Atari Jaguar occupies a special place. It was the final gasp of Atari Corporation, the Jack Tramiel-era incarnation of the famous pioneering game console brand that brought us the ST line of computers, and like Marlon Brando’s Terry Malloy character from On the Waterfront, it coulda been a contender. But the early ’90s games business wasn’t kind to the console from Sunnyvale, and it was squeezed from behind by the SNES and Genesis/MegaDrive, and in front from the PlayStation. Thirty years later then, can it run Linux? [Cakehonolulu] is here to show us how.

With only 2 megabytes of RAM and space for 8 megabytes of ROM, this is hardly a powerhouse. But its 16-bit 68000 processor is a supported Linux architecture, albeit with the -nommu flag on compilation. The “Jerry” DSP chip has the required serial port and timer to boot a first Linux kernel, and after a bit of hackery to make it jump to the ROM location, something boots. There’s no init process until the flat executable file for a -nommu kernel is navigated, but with that past a BusyBox userspace and a graphics driver for the “Tom” graphics chip gives it a chunky on-screen console. The code can be found in a GitHub repository, for the curious.

It seems to be the moment for 68k consoles to receive the Linux treatment, as it’s only a few weeks since we saw it on a MegaDrive. Other ’90s consoles aren’t far behind though, with the Nintendo 64 falling to the penguin a few years ago. Meanwhile, the Dreamcast had Linux running decades ago.

Jaguar image: Evan-Amos, Public domain.

If you’re building a homelab rig, you could just use off-the-shelf hardware in standard cases and slap it all in a rack like the normies do. Or, you could follow the example of [Justin Garrison] and build a more oddball setup.

This particular homelab is, at its heart, built from familiar components. There are two Raspberry Pi 5s, two Raspberry Pi 4s, a GMKtec NucBox M6 Mini with an ASUS GeForce RT 2060 GPU, a LattePanda IOTA, an NVidia DGX Spark, and an HP Z4 G4 mini PC. These machines are all laced together with a TP-Link LS108GB PoE switch. [Justin] has the mini PC running the control plane components, with the rig as a whole running Talos and Kubernetes workloads. What makes this build particularly appealing, though, is the aesthetics of the rig. [Justin] documents how he hacked this hardware to fit into a bunch of old Linksys router cases, which provides a pleasant early 2000s look to the build. This included a bit of hackery to get status LEDs flickering as they should be. [Justin] also took the time to make the power buttons accessible.

If you want to stunt on your friends with a rad homelab, you either have to go for maximum power, or maximum style. This build would be the latter. Video after the break.

Although there’s probably no good reason to want to run image editing software from 1996 other than for nostalgia’s sake, if you ever wanted to run the GIMP version 0.54 from back when Windows 98 was still called Windows 97, you can do so now from the comfort of a modern-day Linux desktop. What enables this is a Flatpak version of a beta release, assembled by [balooii] for everyone’s enjoyment.

It wasn’t a simple matter of compiling the old software’s code and packaging it up, with the repository for the project containing a series of patches that were required to make this possible. Also of note is that this is the first version of GIMP with full surviving source code. Back then, GIMP used the Motif widget toolkit. Later on, it switched to the GIMP Toolkit (GTK).

Bundled with this Flatpak release are a lot of plugins and tutorials that were created at the time, making it a veritable time capsule of a more innocent era. As noted by [balooii], this version of GIMP was very much Beta software, with all of the UI quirks you’d expect. It also features the multiple unconnected windows (not MDI) approach to its UI – dropped in more recent GIMP releases —  that has enraged proponents of the single window approach, as used by all commercial competitors, including Paint Shop Pro and Photoshop.