Iso Creation

Overview / Credits

If you'd prefer to use a simpler approach, check out Hirbod's article on Limeline.

This article is heavily based on Hirbod Behnam's articles. Start by reading Creating hybrid image file from compiled Linux kernel and initramfs with GRUB. We'll be turning his articles into a bash script.

What is an Iso?

Our iso supports dd meaning it can be directly copied.

Iso is basically a mini operating system.

Creating the iso

We'll start by creating a file with all zeros.

build_iso.sh
mkdir -p build_iso
cd build_iso

if [ -f "disk.img" ]; then
    echo "Deleting existing disk image"
    rm "disk.img"
fi

dd if=/dev/zero of=disk.img bs=1M count=928

dd is a copy commands
if is the input file. /dev/zero is a special linux file with all zeros.
of is the output file
bs is the block size. This is how many bytes are being copied from /dev/zero at a time
count is the amount of blocks to be copied. We copy 928 blocks each are 1M giving us a file of 928M.

At this point the file is a clean slate with all zeros. You can view the binary of the file with

sudo apt install ghex
ghex disk.img

Or in the command line with

hexdump -C disk.img

Partitioning

What is partitioning

When you plug a USB drive into your computer, usually it'll show as one drive being plugged in. Partitioning allows you to split that USB (or hard drive) into multiple logical drives.

When you plug in a USB with 2 partitions, you'll see 2 drives (For example you may plug in a USB and see a D:\ and E:\ drive on windows).

Every USB device and hard drive has a partition table which outlines how many partitions the drive has and where each one is located.

When you buy a new USB drive from the store, it looks like it's empty. However it'll have a partition table with one partition which is allocated the amount of space on the drive.

What does a GPT partition table look like?

The partition table stores this information for each partition

Start
End
GUID (Not needed, a unique identifier)
File System Type (GUID)

Just like there are different file formats which store files differently (for example .jpg vs .png), there are different file systems types.

For example, a Windows file system (like NTFS) stores information about what users have rights to files differently than how a linux file system (like EXT4) does.

When Linux reads the GPT partition table and sees a File System Type of NTFS, it'll know the partition is layed out as a windows drive and use different code to read the contents of the partition.

Different operating systems store files differently. The File System Type allows operating systems to

What layout will we use

Partition	Description	Size	File System Type
BIOS boot partition	Used for support older motherboards	1M	BIOS boot partition
EFI Partition	Used for most motherboards released after 2013	48M	EFI system partition
FAT32	Stores Grub + Our Linux OS + The files for our Linux OS	~900M	Linux filesystem

Older motherboards boot differently than new ones. This is often called legacy boot. Motherboards created after 2013 should support UEFI boot. The first BIOS boot partition adds support for these older boards.

Our second partition is an EFI partition. When the system boots, the motherboard has code to start computer. It will read the first drive in the boot order and search it for an EFI partition. It'll eventually load the file /EFI/BOOT/BOOTX64.EFI and hand off the booting process to it. (Slight over simplification) We'll add GRUB to the second partition

Our final partition will contain GRUB and everything we'll use in our installer. We'll essentially have our own Distro on this partition. A distro designed just to install our distro.

Creating the partitions with gdisk

TODO: Add more on Gdisk?

GDisk is an interactive command line tool. This means it expects user input. We can mimic the user typing into the program by using | in our script.

build_iso.sh
echo -e "
o
Y
n
1
2048
+1M
ef02

n
2
4096
+48M
ef00

n
3
102400
1898495
8300

p
w
Y
" | gdisk disk.img

As a challenge, try recreating the above script without looking at it.

Loop devices

We've now partitioned our drive, however only the partition table is on it. (Is there anything else at all after the first chunk of data?)

Next we need to initialize our file system. Filesystems need to have tables which store the starting locations of files and directories, the names of files, how large the files are, and more. To do this, we'll use the mkfs command.

The mkfs command takes in an offset and we can pass in the starting locations of each partition.

Another approach is to create a Loop Device.

If we copied our .img onto a USB, our operating system would recognize the partitions and we could access each partition as its own device. Loop devices allow us to simulate this. We'll setup a loop device for each partition and then write to them.

We use losetup with an offset (how many bytes into the file to start)

If these loop devices are already in use, you may need to use different ones. You can check the current loop devices with losetup -l or losetup -l | grep /dev/loop

TODO Later: Add variable to configure Bash variables

sudo losetup /dev/loop0 disk.img
sudo losetup --offset $((4096*512)) --sizelimit $(((102399-4096+1)*512)) /dev/loop1 disk.img
sudo losetup --offset $((102400*512)) --sizelimit $(((1898495-102400+1)*512)) /dev/loop2 disk.img

We can delete the loop devices we just created too. This is useful when rerunning the script, otherwise we'll get an error saying the loop device is already setup.

# Add this in before running losetup!
sudo losetup -d /dev/loop0
sudo losetup -d /dev/loop1
sudo losetup -d /dev/loop2

Now we'll initialize our file system

sudo mkdosfs -F 32 /dev/loop1
sudo mkdosfs -F 32 /dev/loop2

Mounting

At this point, we've simulated a drive with a readable filesystem. We can now mount the drive. In windows when you plug in a drive, you'll see D:\ or E:\ which you can open to view the drives contents. On linux, we create a new folder which links to the drive.

sudo mkdir /mnt/efi /mnt/drive
sudo mount /dev/loop1 /mnt/efi/
sudo mount /dev/loop2 /mnt/drive/

The mount devices can be deleted using

# Add this in before deleting the loop devices
sudo umount /dev/loop1
sudo umount /dev/loop2

We can view our mount devices

mount | grep /mnt/

We've been using sudo (the super user) to create / modify our disk.img. We'll change the permissions on the file so any user can read / write to it.

chmod 777 disk.img

Installing Grub

You'll need to install some packages to build grub

sudo apt install automake
sudo apt install autoconf-archive
sudo apt install gawk

build_grub.sh
cd downloads

if [ -d "grub-2.14" ]; then
    echo "Skipping grub download"
else
    wget https://ftp.gnu.org/gnu/grub/grub-2.14.tar.xz
    tar xf grub-2.14.tar.xz
fi

mkdir -p ../compile_grub

cp -r grub-2.14 ../compile_grub/grub

cd ../

echo ""
echo "Starting Grub Build"

cd compile_grub/grub



#Qemu requires extra steps to make this work
sudo make clean
sudo ./configure --target=x86_64 --with-platform=efi --disable-werror
sudo make -j $(nproc) AWK=gawk
sudo ./grub-install --target=x86_64-efi --directory=grub-core --efi-directory=/mnt/efi/ --bootloader-id=GRUB --modules="normal part_msdos part_gpt multiboot" --root-directory=/mnt/drive/ --no-floppy /dev/loop0

Grub installs to /mnt/efi/EFI/GRUB/grubx64.efi. We'll move it to /mnt/efi/EFI/BOOT/BOOTX64.EFI so it's auto detected on boot.

sudo mkdir -p /mnt/efi/EFI/BOOT/
sudo mv /mnt/efi/EFI/GRUB/grubx64.efi /mnt/efi/EFI/BOOT/BOOTX64.EFI

Copying on files

We'll copy on the initramfs we made compiling the kernel and a grub config

cd ../../
sudo cp inputs/bzImage inputs/initramfs.cpio.gz /mnt/drive/boot/
sudo cp inputs/grub.cfg /mnt/drive/boot/grub/grub.cfg

Grub config:

set debug="efi,linux,loader"
set pager=1

set default=0
set timeout=10

insmod gzio
insmod efi_gop
insmod font
loadfont /boot/grub/fonts/unicode.pf2

menuentry 'MyOs' --class os {
    linux /boot/bzImage
    initrd /boot/initramfs.cpio.gz
}

Running our iso

We want to test our iso in EFI mode

#Make sure ovmf is installed
#sudo apt install ovmf
qemu-system-x86_64 -bios /usr/share/qemu/OVMF.fd -m 4096 -drive format=raw,file=disk.img

We can also run it and debug

sync
cd build_iso
qemu-system-x86_64 -bios /usr/share/qemu/OVMF.fd -m 4096 -serial stdio -d guest_errors,cpu_reset -drive format=raw,file=disk.img

TODO: Add qemu-nbd for running in docker

Debugging

Without knowing the correct tools, finding issues is very difficult. Here are some ways to see what's going on in the image.

On the grub boot menu, you can press l to enter the grub shell.

Example commands

# See detected devices
ls

ls (hd0)
ls (hd0,gpt3)/boot

Boot Manually

linux (hd0,gpt3)/boot/bzImage  
boot

Chain loader

chainloader (hd0,gpt3)/boot/bzImage  
boot

Add grub debugging to config: set debug="efi,linux,loader"
set pager=1

Installation instructions for end users

End users can use Ventoy, Rufus (if coming from windows), balenaEtcher and more

Overview / Credits​

What is an Iso?​

Creating the iso​

Partitioning​

What is partitioning​

What does a GPT partition table look like?​

What layout will we use​

Creating the partitions with gdisk​

Loop devices​

Mounting​

Installing Grub​

Copying on files​

Running our iso​

Debugging​

Installation instructions for end users​

Additional Reading​