Whilst doing research for something the other day I came across what might just be one of the coolest additions to the Pi 3! The ability to make it PXE boot from the network to remove the requirement for any form of local storage, now you have been able to get the Pi to boot from the LAN using Uboot for some time, but this always required you to have an SD card in the Pi to deal with the lack of ROM, but this has been taken away in the Pi 3 and boot from LAN is an option.
The nice people over at RaspberyPi.org have put together a bit of information on the subject so here are a couple of useful links should you wish to dive into it a little deeper before just getting stuck in
Its probably worth pointing out that they have noted the following issues when trying to get this to work
- When the boot ROM enables the Ethernet link, it first waits for the link to come up, then sends its first DHCP request packet. This is sometimes too quick for the switch to which the Raspberry Pi is connected: we believe that the switch may throw away packets it receives very soon after the link first comes up.
- The second bug is in the retransmission of the DHCP packet: the retransmission loop is not timing out correctly, so the DHCP packet will not be retransmitted.
The solution to both these problems is to find a suitable switch which works with the Raspberry Pi boot system. We have been using a Netgear GS108 without a problem.
- Finally, the failing timeout has a knock-on effect. This means it can require the occasional random packet to wake it up again, so having the Raspberry Pi network wired up to a general network with lots of other computers actually helps!
That said, I’m running a couple of LinkSys 1Gb switches here and never had any issues, all of my setup and testing was done on a VLAN’ed off section of this and there was very little other stuff running on the network at the time.
Gotchas
For a change I’m going to put the gotchas in a section at the top of the post rather than talking about them as I go through the process, mainly because these are big ones that are going to cost you hours of time to work thru, I’ll update the list as I work on the project with any more I find.
Prepare your “Client” Pi 1st – You need to tweak the standard Image to prep your “Client” Pi (cPi) before you can untweak it and then us it to prepare your “Server” Pi (sPi)
File copies next – During the setup process your going to be installing packages on your “Server” Pi (sPi) so if you miss time your file copies when you boot your “Client” Pi (cPi) anything you have installed, changed, removed will change the file copy and therefore effect the cPi boot result
So those few things pointed out lets make a start…..
From now on I’m going to refer to the “Clent” Pi as cPi and the “Server” Pi as sPi
Client Preparation (cPi)
Before you can PXE boot a Raspberry Pi you need to boot it from the SD card to allow you to enable the USB boot mode. USB boot mode will set a bit in the One Time Programmable (OTP) memory in the Raspberry Pi System on Chip (SoC) that enables network booting. Once this change has been made the SD card is no longer required in the cPi.
I’ve made the assumption here that you have already downloaded a copy of Raspbian and got it over to an SD card you can boot from.
So pop it into the cPi and boot it up, once booted login with the default username and password
Let’s grab e latest set of updates at this point by entering the follwoing command
sudo apt-get update
That done its time to enable USB blot mode on the cPi, this is done by setting the program_usb_boot_mode flag in config.txt, this can simply be achived by the following command
echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt
Now reboot the cPi
sudo reboot
Once the cPi has rebooted we want to make sure that the changes have taken place as we expected, so login to the cPi and run the following command
vcgencmd otp_dump | grep 17:
The output should be
17:3020000a
Remember in the gotchas section I said put everything back before you copy the files? well now is the time to do that! So lets revert the /boot/config.txt back to its original settings
sudo nano /boot/config.txt
and remove the line program_usb_boot_mode from the end of the file, save and close the file by pressing “CTRL+X” followed by “Y” to save the changes and then shut down the cPi with the following command
sudo poweroff
OK so thats the setup for the cPi complete!
Server Preparation (sPi)
Put the SD card into the sPi and power it up, we are going to need to expand the file system at this point to make way for the file copies that are need so you cPi has the files it needs available to boot over the LAN, so lets expand the root filesystem to utilise all of the available space on our SD card
sudo raspi-config
Follow the on-screen instructions to expand the file system and let the sPi reboot when complete, login to the sPi so we can make a start on the setup. At this point I’m going to switch user so all commands are run as root, being a little lazy and not wanting to remember to do it each time I run a command so use the following command
sudo -su
Lets prepare the root filesystem for the cPi to boot from first, we do this by making a complete copy of the sPi filesystem and dumping it out to a directory that we will later configure as an NFS share for the cPi to use, to make that copy we are going to use rsync so lets grab a copy
apt-get install rsync
Once the installation has completed create the directory that we will use later and copy the file system into it
mkdir -p /nfs/client
rsync -xa --progress --exclude /nfs / /nfs/client
We now need to regenerate the SSH host keys on the filesystem copy by chrooting into it
cd /nfs/client
sudo mount --bind /dev dev
sudo mount --bind /sys sys
sudo mount --bind /proc proc
sudo chroot .
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server
exit
sudo umount dev
sudo umount sys
sudo umount proc
We now need to configure our sPi to have a static IP address as we are going to reference it later in the configuration, your going to have to change some of these settings to match your network configuration
nano /etc/network/interfaces
Change the line iface eth0 inet manual as per the below adding the additional lines needed to configure the “address” “netmask” and “gateway” for your network
auto eth0
iface eth0 inet static
address 192.168.1.250
netmask 255.255.255.0
gateway 192.168.1.1
Once done save and close the file by pressing “CTRL+X” followed by “Y”
We now need to disable the DHCP client, enable standard Debian networking and do a reboot for it to take effect
systemctl disable dhcpcd
systemctl enable networking
reboot
Once rebooted log back in and switch back to root sudo -su
At this point, you won’t have working DNS, so you’ll need to add a DNS server to /etc/resolv.conf. On most home networks this is the gateway address of your router so lets get that sorted
echo "nameserver 192.168.1.1" | sudo tee -a /etc/resolv.conf
We need to make the file immutable, otherwise dnsmasq will interfere with it, so use the following command to solve the issue
chattr +i /etc/resolv.conf
We are going to use dnsmasq to send some additional parameters to the cPi on boot so grab a copy
apt-get update
apt-get install dnsmasq
Now we need to stop dnsmasq interfering with DNS resolution, and perform a reboot for the changes to take effect
rm /etc/resolvconf/update.d/dnsmasq
reboot
Once rebooted log back in and switch back to root sudo -su
We need to modify the dnsmasq configuration to enable DHCP to reply to the device
echo | sudo tee /etc/dnsmasq.conf
nano /etc/dnsmasq.conf
Replace the .conf file with
port=0 dhcp-range=192.168.1.255,proxy log-dhcp enable-tftp tftp-root=/tftpboot pxe-service=0,"Raspberry Pi Boot"
I had a bit of a problem here as my DHCP server was on a different network segment received via a relay agent
This extract from the DNSMASQ manual is relevant:
For directly connected networks (ie, networks on which the machine running dnsmasq has an interface) the netmask is optional. It is, however, required for networks which receive DHCP service via a relay agent.
So your dhcp-range line should be may need to look something like this
dhcp-range=192.168.1.0,proxy,255.255.255.0
Now we need a tftp boot directory to server a few files to the cPi prior to it connecting the NFS share created earlier to get the filesystem
mkdir /tftpboot
chmod 777 /tftpboot
systemctl enable dnsmasq.service
systemctl restart dnsmasq.service
Your going to need a few files from the /boot directory on the SD card so we may as well take everything, and then restart dnsmasq
cp -r /boot/* /tftpboot
systemctl restart dnsmasq
If you power up your cPi now you should get something, but not a great deal as the root file system is not quite ready yet, as we have not exported it so lets fix that problem
apt-get install nfs-kernel-server
echo "/nfs/client *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
systemctl enable rpcbind
systemctl restart rpcbind
systemctl enable nfs-kernel-server
systemctl restart nfs-kernel-server
We also need to edit the cmdline.txt file that we copied over to the tftp server so that it know where to find the file system
nano /tftpboot/cmdline.txt
Its going to look something like this
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 roootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
and we want something that looks like this
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.1.250:/nfs/client1 rw ip=dhcp rootwait elevator=deadline
Save and close the file by pressing “CTRL+X” followed by “Y”
The last thing we need to do is edit the /nfs/client/etc/fstab file and remove a couple of lines
nano /nfs/client/etc/fstab
what you have will look like this
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime 0 1 # a swapfile is not a swap partition, no line here # use dphys-swapfile swap[on|off] for that
make it look like this
proc /proc proc defaults 0 0
Save and close the file by pressing “CTRL+X” followed by “Y”
That’s it your done, boot up cPi and see what happens. It can take a little time for anything to happen and you may need to power cycle cPi a couple of time before you get anything, but I did find that once you had it working it would get more reliable on the boots.
All this was for a very good purpose! and for those of you who know me personally you’ll soon be seeing why!
Enjoy
UM
