## Alpine Linux Installation on ZFS Root with grub on UEFI - References \[ [ref1](https://bit-strickerei.de/posts/alpine-zfs-root/) | [ref2](https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_ZFS_with_native_encryption) | [ref3](https://wiki.alpinelinux.org/wiki/Setting_up_ZFS_on_LUKS) \] - Boot from alpine-extended \[ [download](https://alpinelinux.org/downloads/) \] - login as `root` without any password - setup network interfaces and start networking: ``` setup-interfaces /etc/init.d/networking start ``` - add, config, and enable openssh: ``` apk add openssh echo "PermitRootLogin yes" >> /etc/ssh/sshd_config /etc/init.d/sshd start ``` - config root password and show IP so we can ssh in: ``` echo "root:abc" | chpasswd # set root password to "abc" ip addr show # show ip address ``` - now we can ssh into the machine - pick a mirror from [alpine-mirrors](http://nl.alpinelinux.org/alpine/MIRRORS.txt) and get the latest version ``` MIRROR=http://dl-cdn.alpinelinux.org/alpine # example from mirror above, no trailing slash ARCH=x86_64 # architecture, 32-bit x86 is "i386" and amd64 is "x86_64" VERSION=v3.11 # get the latest version from https://alpinelinux.org/ -- do not include last dot ``` - setup apk repos and update: ``` echo "$MIRROR/$VERSION/main" >> /etc/apk/repositories echo "$MIRROR/$VERSION/community" >> /etc/apk/repositories apk update ``` - setup udev to get `/dev/disk/by-id` ``` apk add util-linux udev zfs setup-udev ``` - load zfs module and check version ``` modprobe zfs # should return nothing lsmod | grep zfs # should list a few modules zfs --version # shows version ``` - load sgdisk and dosfstools and list disks: ``` apk add sgdisk dosfstools ls -l /dev/disk/by-id # shows disk by id ls -l /dev/disk/by-id/ | sed -E 's/^.*[ \t]+([^ \t]+[ \t]+->[ \t]+.*)$/\1/g' # without extraneous info ``` > ``` > lrwxrwxrwx 1 root root 9 Mar 15 16:44 ata-VBOX_CD-ROM_VB2-01700376 -> ../../sr0 > lrwxrwxrwx 1 root root 9 Mar 15 16:44 ata-VBOX_HARDDISK_VB0cc84e8a-30407bab -> ../../sda > lrwxrwxrwx 1 root root 9 Mar 15 16:44 ata-VBOX_HARDDISK_VB5ee32bc3-a631fa00 -> ../../sdb > ``` - We will use `/dev/sda` id as `$DISK1` and `/dev/sdb` id as `$DISK2` for the following example: ``` DISK1=/dev/disk/by-id/ata-VBOX_HARDDISK_VB0cc84e8a-30407bab DISK2=/dev/disk/by-id/ata-VBOX_HARDDISK_VB5ee32bc3-a631fa00 ``` - wipe both disks: ``` sgdisk --zap-all $DISK1 sgdisk --zap-all $DISK2 ``` each command should return: > ``` > Creating new GPT entries in memory. > GPT data structures destroyed! You may now partition the disk using fdisk or > other utilities. > ``` - create EFI partitions: ``` sgdisk -n1:1M:+512M -t1:EF00 $DISK1 udevadm settle mkfs.fat -F 32 -n EFI ${DISK1}-part1 udevadm settle sgdisk -n1:1M:+512M -t1:EF00 $DISK2 udevadm settle mkfs.fat -F 32 -n EFI ${DISK2}-part1 udevadm settle ``` - create ZFS partition for boot: ``` sgdisk -n2:0:+1G -t2:BF00 $DISK1 sgdisk -n2:0:+1G -t2:BF00 $DISK2 udevadm settle ``` - create ZFS partition for root: ``` sgdisk -n3:0:0 -t3:BF00 $DISK1 sgdisk -n3:0:0 -t3:BF00 $DISK2 udevadm settle ``` - create boot pool (bpool) with features incompatible for grub disabled ``` BPOOL=bpool zpool create -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@userobj_accounting=enabled \ -o feature@zpool_checkpoint=enabled \ -o feature@spacemap_v2=enabled \ -o feature@project_quota=enabled \ -o feature@resilver_defer=enabled \ -o feature@allocation_classes=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=none -R /mnt -f $BPOOL mirror ${DISK1}-part2 ${DISK2}-part2 ``` - create root pool (use `ashift=13` for Samsung SSD): ``` RPOOL=rpool zpool create -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=none -R /mnt \ -f $RPOOL mirror ${DISK1}-part3 ${DISK2}-part3 ``` - create containers for $BPOOL and $RPOOL ``` zfs create -o mountpoint=none -o canmount=off $BPOOL/BOOT zfs create -o mountpoint=none -o canmount=off $RPOOL/ROOT ``` - read [create your datasets](https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Create_your_datasets) ``` zfs create -o mountpoint=legacy -o atime=off $BPOOL/BOOT/default zfs create -o mountpoint=/ -o atime=off $RPOOL/ROOT/alpine zfs create -o mountpoint=/home -o atime=off $RPOOL/HOME zfs create -o mountpoint=/var/log -o atime=off $RPOOL/LOG ``` - mount boot filesystem ``` # mount -t zfs $RPOOL/ROOT/alpine /mnt mkdir -p /mnt/boot mount -t zfs $BPOOL/BOOT/default /mnt/boot ``` - Set the bootfs property on the boot filesystem. This command will only be successful when `dnodesize` property on `$BPOOL/BOOT/default` is set to `legacy`: ``` zfs set dnodesize=legacy $BPOOL/BOOT/default zpool set bootfs=$BPOOL/BOOT/default $BPOOL ``` #### Installing Alpine Linux in a chroot \[ [ref](https://wiki.alpinelinux.org/wiki/Alpine_Linux_in_a_chroot) \] - Find out the latest `apk-tools-static` version. ``` echo ${MIRROR}/latest-stable/main/${ARCH} ``` - Go to the above URL and find the file `apk-tools-static-?????.apk`, As of this writing (2020-03-12) the version is `apk-tools-static-2.10.4-r3.apk` - Download and unpack the `apk-tools-static` package to mount directory: ``` wget ${MIRROR}/latest-stable/main/${ARCH}/apk-tools-static-2.10.4-r3.apk tar -xzf apk-tools-static-*.apk ``` - Install the alpine base onto chroot ``` ./sbin/apk.static -X ${MIRROR}/latest-stable/main -U --allow-untrusted --root /mnt --initdb add alpine-base ``` - bind mount `/dev`, `/proc`, `/sys` -- see [set up the chroot](https://wiki.alpinelinux.org/wiki/Alpine_Linux_in_a_chroot#Set_up_the_chroot) ``` mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys ``` - copy DNS settings and apk repos to to `/mnt` ``` cp /etc/resolv.conf /mnt/etc/resolv.conf cp /etc/apk/repositories /mnt/etc/apk/repositories ``` - save environment vars for later use: ``` cat > /mnt/root/vars.sh << EOF DISK1=$DISK1 DISK2=$DISK2 RPOOL=$RPOOL BPOOL=$BPOOL ARCH=$ARCH MIRROR=$MIRROR VERSION=$VERSION EOF ``` - chroot ``` chroot /mnt /usr/bin/env DISK1=$DISK1 DISK2=$DISK2 RPOOL=$RPOOL BPOOL=$BPOOL /bin/ash -l ``` - mount `/dev/cdrom` and add necessary stuff ``` mount /dev/cdrom /media/cdrom apk add linux-lts udev util-linux zfs-lts openssh nano ``` - perform init process ``` rc-update add devfs sysinit rc-update add dmesg sysinit rc-update add mdev sysinit rc-update add hwdrivers sysinit rc-update add hwclock boot rc-update add modules boot rc-update add sysctl boot rc-update add hostname boot rc-update add bootmisc boot rc-update add syslog boot rc-update add mount-ro shutdown rc-update add killprocs shutdown rc-update add savecache shutdown ``` - enable zfs services ``` rc-update add zfs-import sysinit rc-update add zfs-mount sysinit ``` - Edit the `/etc/mkinitfs/mkinitfs.conf` file and append `nvme zfs` module to the `features` parameter: ``` # if you don't need nvme: sed -i -E 's/^(features=.*)([^ \t])"$/\1\2 zfs"/g' /etc/mkinitfs/mkinitfs.conf sed -i -E 's/^(features=.*)([^ \t])"$/\1\2 nvme zfs"/g' /etc/mkinitfs/mkinitfs.conf cat /etc/mkinitfs/mkinitfs.conf # check the work ``` - rebuild initramfs ``` mkinitfs $(ls /lib/modules/) ``` - generate fstab for bpool ``` mv /etc/fstab /etc/fstab.old cat >> /etc/fstab << EOF $BPOOL/BOOT/default /boot zfs rw,nodev,noatime,xattr,posixacl 0 0 EOF ``` - generate mount points for EFI partitions: ``` mkdir -p /efi1 mkdir -p /efi2 ``` - add 2 EFI partitions to `/etc/fstab` ``` echo UUID=$(blkid -s PARTUUID -o value ${DISK1}-part1 | sed -E 's/^.*[ \t]UUID="([^ \t]+)".*$/\1/g') \ /efi1 vfat nofail,defaults 0 0 >> /etc/fstab echo UUID=$(blkid -s PARTUUID -o value ${DISK2}-part1 | sed -E 's/^.*[ \t]UUID="([^ \t]+)".*$/\1/g') \ /efi2 vfat nofail,defaults 0 0 >> /etc/fstab ``` ``` echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK1}-part1 | sed -E 's/:.*$//g') \ /efi1 vfat nofail,defaults 0 0 >> /etc/fstab echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK2}-part1 | sed -E 's/:.*$//g') \ /efi2 vfat nofail,defaults 0 0 >> /etc/fstab ``` - check `/etc/fstab` to be of this format: > ``` > bpool/BOOT/default /boot zfs rw,nodev,noatime,xattr,posixacl 0 0 > UUID=F857-B91A /boot/efi1 vfat nofail,defaults 0 0 > UUID=F889-5FAE /boot/efi2 vfat nofail,defaults 0 0 > ``` - mount EFI partitions: ``` mount /efi1 mount /efi2 ``` - install grub-efi ``` apk add efibootmgr grub-efi ``` > `grub-probe` will report an error. we will fix later - install grub to `/efi1` ``` mkdir -p /boot/grub ZPOOL_VDEV_NAME_PATH=1 grub-probe /boot # grub-probe should return "zfs" ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/efi1 --bootloader-id=GRUB #ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/boot/efi2 --bootloader-id=GRUB2 ``` - modify `/etc/default/grub` - add `root=ZFS=$RPOOL/ROOT/alpine` to `GRUB_CMDLINE_LINUX` ``` [[ -f /etc/default/grub.original ]] && cp /etc/default/grub.original /etc/default/grub cp /etc/default/grub /etc/default/grub.original # add GRUB_CMDLINE_LINUX="root=$RPOOL/ROOT/alpine" echo "GRUB_CMDLINE_LINUX=\"root=""$RPOOL""/ROOT/alpine rootfstype=zfs\"" >> /etc/default/grub # add stuff to to GRUB_CMDLINE_LINUX_DEFAULT echo "GRUB_CMDLINE_LINUX_DEFAULT=\"modules=sd-mod,usb-storage,ext4 nomoodeset\"" >> /etc/default/grub # remove crap from GRUB_CMDLINE_LINUX_DEFAULT # sed -i -E 's/^(GRUB_CMDLINE_LINUX_DEFAULT=).*$/\1=""/g' /etc/default/grub ``` - make new `/boot/grub/grub.cfg` ``` ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg ``` - **it will report an error.** **BUMMER** **2 choices below** - 1. run script to fix what's manually done in 2. ``` cp /boot/grub/grub.cfg /boot/grub/grub.cfg.original sed -i -E 's;^([ \t]*linux[ \t]+/BOOT/default@/vmlinuz[^ \t]*[ \t]+root=).*$;\1'"$RPOOL"'/ROOT/alpine rootfstype=zfs modules=sd-mod,usb-storage,ext4 nomodeset;g' /boot/grub/grub.cfg sed -i -E '/^\/dev\/sd[a-z]3[ \t]+ro[ \t]+root='"$RPOOL"'\/ROOT\/alpine.*$/d' /boot/grub/grub.cfg ``` - 2. Manually fix it. edit near line "echo 'Loading Linux lts ...' followed by `linux /BOOT/default@/vmlinuz...` so it reads: ``` echo 'Loading Linux lts ...' linux /BOOT/default@/vmlinuz-lts rootfstype=zfs root=$RPOOL/ROOT/alpine ro modules=sd-mod,usb-storage,ext4 nomodeset echo 'Loading initial ramdisk ...' ``` - rsync both partitions to make both bootable ``` apk add rsync rsync -Rai --stats --human-readable --delete --verbose --progress /efi1/./ /efi2 ``` - add EFI boot entry to machine ``` efibootmgr -c -g -d $DISK2 -p 1 -L "GRUB-2" -l '\EFI\GRUB\grubx64.efi' ``` - (optional) disable media use for apk installation: ``` sed -i -E 's/^(\/media.*)$/#\1/g' /etc/apk/repositories ``` - (optional) permit root login on ssh: ``` sed -i -E 's/^(#PermitRootLogin.*)$/PermitRootLogin yes\n\1/g' /etc/ssh/sshd_config ``` - (optional-required for ssh login) set root password: ``` echo "root:abc" | chpasswd ``` - set ZFS pool cachefile: ``` zpool set cachefile=/etc/zfs/zpool.cache $RPOOL ``` - exit chroot ``` exit ``` ### Done with chroot, prepare to reboot - unmount all ``` mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} ``` - unmount zfs and export pools ``` umount /mnt/boot zfs unmount $RPOOL/LOG zfs unmount $RPOOL/HOME zfs unmount -a ``` - remove live USB/CD and reboot ``` # remove live media reboot ``` #### Post Installation Setup - login as `root` with or without password (depending on whether you set root password) - setup networking: ``` setup-interfaces /etc/init.d/networking restart ``` - Hopefully ethernet access will be on: try `ping -c 3 google.com` - If you haven't enabled sshd or set root password, go back and do it: - ssh login to machine with `root` and password - can mount live media for faster apk updates: ``` mount /dev/cdrom /media/cdrom ``` - fix `grub-efi` install error from earlier: ``` cp /boot/grub/grub.cfg /boot/grub/grub.cfg.working apk fix # this will fix grub-efi error cp /boot/grub/grub.cfg.working /boot/grub/grub.cfg ``` - add some useful software: ``` apk add which htop man man-pages ``` - setup the rest of alpine ``` # cp /etc/apk/repositories /etc/apk/repositories.save setup-udev setup-keymap setup-hostname setup-timezone setup-proxy setup-ntp # setup-apkrepos # setup-sshd setup-ntp # use chrony # setup-alpine # when asked "which disk to use?" answer "none" # cp /etc/apk/repositories.save /etc/apk/repositories # don't wanna mess with repo file ``` - change root to run `bash` instead of `ash`: ``` apk add bash bash-completion shadow sudo usermod -s /bin/bash root ``` - logout and login again to take effect - add another user ``` useradd -s /bin/bash -m -U username echo "username:1234" | chpasswd ``` - add that user to `wheel` group so user can `sudo` ``` EDITOR=nano visudo ``` - uncomment either a) `# %wheel ALL=(ALL) ALL` or b) `# %wheel ALL=(ALL) NOPASSWD: ALL` - if you uncomment a), password is needed for `sudo`, if you uncomment b), password is not needed for `sudo` - add a user to `wheel` group so that user can `sudo` ``` usermod -aG wheel username ``` - login as a non-root user and use it to disable direct root login: ``` sudo passwd -l root ``` (`sudo passwd -u root` will unlock root access) - reboot and login as root