PXE Boot Adventures

The installfest for VTLUUG was this past weekend, and I was tasked with setting up a PXE boot server. For those unfamiliar with this, PXE (Preboot eXecution Environment) is a built-in feature on most newer motherboards that allows users to obtain a bootloader over the network for remote OS installations. The only requirements on the network side are a router capable of directing DHCP clients to a TFTP server and assigning a file path. Client machines can then download a bootloader (e.g. syslinux) image which provides a menu of kernels to choose from. Typically, only the minimum number of required files are stored on the TFTP server, so that clients can download the rest via faster protocols such as NFS or HTTP.

Setting up the server was not all that difficult, but there were a few setbacks. I started with a 64-bit Debian VM on my laptop with atftp-server and followed a basic tutorial I found online. Setup for this was fairly straightforward for the more common distributions with text installers- -you only needed to mount the netinstall ISO, copy the files to the TFTP root (/var/lib/tftp in my case) and make a bootloader entry pointing to the kernel for each OS.

Unfortunately, this did not always work. Distros like Ubuntu with fancy graphical installers could not be served entirely of of TFTP and required an NFS share for the rest of the content on the ISO. This was fairly easy to do with a "nfsroot=" string appended to the kernel line in the syslinux configuration. Eventually, I was able to get most common distributions (Ubuntu, CentOS, Fedora, Debian, etc) up and running on my VM, but Arch would not boot.

Since I needed a PXE server for another organization, I had everything up and running a few weeks in advance, and just rsynced all of my data to another Debian x64 server on campus. I figured I'd just open the TFTP port for the duration of installfest so everyone could boot remotely. The night before installfest, I found out that TFTP booting would not work at all over NATs, becaus e the UDP ports used were chosen randomly and therefore can't be forwarded in advance.

With less than 12 hours remaining, our options were limited to:
  • Setting up an iPXE server and handing out USB drives
  • Making another PXE boot server
  • Hauling a server across campus and up serveral flights of stairs
  • Learning IPsec and setting up a point-to-point VPN
  • Using the original (albeit outdated) VM on my laptop
We ended up picking a combination of these and had a rather interesting network configuration. I brought a router to assign DHCP leases and pointed the TFTP boot server to a bridged interface to my VM. We also setup an iPXE server elsewhere on campus, which was much easier to install (yet unsupported by nearly all motherboards). To solve this, I was able to set a menu entry on my PXE VM to point to the iPXE kernel, and we had a few flash drives with the iPXE bootloader to hand out. Unfortunately, the iPXE installation had intermittent connectivity issues, possibly due to DHCP timeouts on my router's dnsmasq server.

Installfest was a much larger success than in years past, probably due to our promotion at Gobblerfest and spamming of all the listservs. We ended up with 27 successful installations in a few hours, mainly composed of Fedora, Ubuntu, and Arch Linux. Less popular distros included Sabayon, OpenBSD, FreeBSD, DragonflyBSD, and Rebecca Black Linux (yes, that's a thing).