How I made a Linux distro that runs `wget | dd`
To replace your Linux installation with a new OS, you can download a
pre-prepared disk image directly to your disk by running a command like this:
curl https://something.example/foo.img > /dev/sda # (or whatever your disk is called)
No need to save it anywhere first, just pipe it directly to the disk. Then you
can reboot into that OS, and congratulations, you’ve installed a new OS. This
works even for most EFI machines without any further efibootmgr commands
because
EFI firmware automatically discovers the new EFI system partition!
Why?
This command is possible because /dev/sdX, /dev/nvmeX, and other such disks
can be directly accessed from the filesystem, following the ancient Unix concept
of “everything is a file.” You can write to the /dev/sdX file, and that
directly writes to the disk!
No, like, why would you want to do this instead of installing Linux any other normal way?
Okay, so the reason I initially did this was because I didn’t want to pay
Contabo an extra $1.50/mo to have object storage just to be able to spawn VPSes
from premade disk images.
I thought it was a neat trick, a funny shitpost that riffs on the eternal
curl | sh debate. I could write a blog post about it, I tell you about how you
can do it yourself, one thousand words, I learn something, you learn something,
I get internet points, win win.
The problem is that it didn’t stop there. I kept asking one more question. I
kept peeling one more layer off the onion. I kept turning one more page in the
grimoire… and before I knew it, I ended up with a four part blog series that
doesn’t end where you expect it to end.
Why don’t we start from the beginning?
How to flash a Raspberry Pi the cool way
Nowadays, the Raspberry Pi Foundation gives you a piece of software that they
built that does everything automatically, but back in my day, you had to do it
this way.
There’s a Stack Exchange answer
that lists the exact series of steps.
- First, you go to the website in your web browser.
- Then, you click on the button to download the image.
- Then, quoting directly from that answer:
Copy the contents of the image file onto the SD card by running
sudo dd bs=1M if=your_image_file_name.img of=/dev/sdx
Yeah, they also said to sha256sum the image, but let’s be honest, nobody
fucking does that, not even Caligula
users.
lazier?
Now, a logical first improvement to this step is, rather than navigating through
the web browser, you can just wget or curl that file from the command line
to begin with if you already know the URL. So, you can run:
wget -O disk.img https://www.raspberrypi.com/whatever
sudo dd bs=1M if=disk.img of=/dev/sdx
lazier.
But of course, nothing says you have to download it to a file first. dd will
read from stdin if you don’t give it an if= argument. So you can pipe into
dd and that achieves the same effect without writing any intermediate files.
curl https://www.raspberrypi.com/whatever | sudo dd bs=1M of=/dev/sdx
lazier!
But of course, nothing says you have to use dd. That just makes your writes to
the disk more efficient because of block alignment and caching nonsense. You can
just redirect stdout like so:
sudo -i
curl https://www.raspberrypi.com/whatever > /dev/sdx
And congratulations, you have derived the initial shitpost premise from first
principles.
it was compressed oops
I glossed over the fact that the Stack Exchange article also tells you that the
disk image comes as a zip file that you need to unzip first. But that’s a nice
segue, because it turns out there are plenty of other variations on this:
# You may need to unzip your thing
curl https://something.example/foo.img.gz | gunzip | dd bs=1M of=/dev/sda status=progress
# You can use wget
wget -O- https://something.example/foo.iso | dd bs=1M of=/dev/sda status=progress
# You can upload via SSH
gzip -vc disk.img | ssh my-server.example -- sh -c 'gunzip -vc | dd bs=1M of=/dev/sda status=progress'
I mean frankly, there’s an infinite number of ways to write directly from the
network to the disk!
sillier?
Okay, so now, let’s say you have a VM running in Contabo, booted off of
/dev/sda, that you wanted to reimage with your own OS image. What do we need
to do to adapt this method to that?
Making your own bootable OS image
Of course, you do need to figure out how to make such an OS image first.
Luckily, you can do this for any OS (even Windows!) by installing it in a VM
first, and then using the raw disk image that results from that.
To do that with QEMU, you need to first make a raw disk, preferably of a fairly
small size (you should be able to expand it once you’ve copied it).
truncate 10G disk.img
Then, you can run your OS with installer like so:
qemu-system-x86_64
-hda ./disk.qcow2 \
-m 2G \
-enable-kvm \
-nic user \
-serial mon:stdio
Go through all the setup steps, and you’re done. You can now send disk.img off
to your webserver.
Of course, I use NixOS btw so this entire process was automated! I just did
import <nixpkgs/nixos/lib/make-disk-image.nix>
and that got me a disk image. If you use Nix as well, you can learn how to use
it by reading its documented source code.
Unmounting the disk
Now that we’ve made the disk image, we need to unmount the victim disk. This is
a very easy process. You just type in umount /dev/sdwhatever or
umount /dev/nvmewhatever, like
root@localhost:~# umount /dev/sda1
umount: /: target is busy.
Oh. Right. The disk we’re trying to replace is the OS’s main disk. The one the
OS is running off of.
Well, what can we try instead?
write to the mounted disk anyways. fuck you
The OS may stop you from unmounting /dev/sda1, but it won’t stop you from
writing to /dev/sda1 or /dev/sda even if there’s something mounted! How do
you think parted, gparted, and fdisk work on live systems?
I ran the following command to upload, and typed in my password:
astrid@chungus infra ❯ gzip -vc result/nixos.img | ssh root@myhost.example -- bash -c 'gunzip -vc > /dev/sda'
root@myhost.example's password:
Now, I tried this out expecting that it probably wouldn’t work. Sure, programs
have to get loaded into memory to run, so maybe this might work. But also, given
that a whole bunch of other things are happening on the machine at the same time
besides the rewrite, doing this may cause those other processes to trigger a
kernel panic.
But theory means nothing in the face of practice.
The command had no output, but iftop was indicating that something was
happening:
191Mb 381Mb 572Mb 763Mb 954Mb
└───────────────────────┴───────────────────────┴───────────────────────┴───────────────────────┴───────────────────────
chungus.lan => myhost.example. 32.9Mb 22.3Mb 16.6Mb
<= 424Kb 302Kb 228Kb
After waiting for a little while, the program terminated with the following
output:
astrid@chungus infra ❯ gzip -vc result/nixos.img | ssh root@myhost.example -- bash -c 'gunzip -vc > /dev/sda'
root@myhost.example's password:
77.8% -- replaced with stdout
What happened here?
Well, we tried to overwrite the OS while it was in use, and that caused it to
crash 77.8% of the way through! A whole bunch of things could have happened,
and the exact crash detail could be interesting, but the sum of it is that we
did something stupid and caused something stupid.
Maybe it’s a good idea to unmount the device before writing to it after all.
Still, though, we have to think about the implications here. How do you unmount
your OS’s disk while keeping the OS running to be able to overwrite itself?
This may sound like some kind of paradoxical Buddhist meditation riddle, but the
solution is actually quite simple: just boot into a new OS where the old one
isn’t mounted!
Rescue images to the rescue
Most Linux distros’ installers have all the requisite programs preinstalled,
along with networking. Arch and NixOS installers are perfect for this purpose!
Fedora installer can even be used for this too. You can piggyback off of them
without even running the actual installer included with them! As long as you
have networking and terminal access, you can run curl > /dev/sda to your
heart’s content.
This was, in fact, the option I went with to overwrite my Contabo VMs. Contabo
lets you boot into a Debian-based rescue image instead of whatever’s installed
on the disk they give you, so I booted into that, and managed to
wget -O- > /dev/sda and get my new OS installed!
I did this once, and I was happy.
Then I did it a second time, and thought, “isn’t there a way to automate this?”
Also, do we really need a second disk?
No really, just think about it. All of the utilities needed to overwrite the
disk already exist on the disk before you overwrite it. In theory, you could
just copy all of those utilities into RAM, unmount everything else, and
overwrite the disk from RAM to achieve an in-place swap.
