Skip to content

Instantly share code, notes, and snippets.

@ixmatus

ixmatus/Readme.md

Forked from techhazard/Readme.md
Created July 30, 2017 18:10
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ixmatus/7dcd56c8e878e8d98ee6d266f7949d11 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ixmatus/7dcd56c8e878e8d98ee6d266f7949d11 to your computer and use it in GitHub Desktop.
Download ZIP
Nixos with ZFS on encrypted LUKS as root filesystem
Raw
Readme.md

NixOS with ZFS on LUKS

After some effort (and asking for help on the nix-dev mailing list) I installed ZFS on an encrypted partition. The relevant configuration is below.

Installing

I do not have a custom iso yet, so you'll need two USBs. One for the NixOS iso, and one for these files. You'll have to mount the second stick manually.

  1. Boot into the nixos environment and find the uuid or id of the disk you want to install to. Do not use /dev/sda but /dev/disk/by-..., use lsblk and blkid.
  2. export it to the environment as rootdisk and run it:
# whole disk please, no partition
export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"
  1. use keyfile and/or configure passphrase usage (see sections below)
  2. run it:
bash /path/to/automated_install.sh

Install with keyfile

It is possible to use a keyfile (e.g. on a usb stick)

export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"

export keyfile="/dev/disk/by-id/usb-Some-Usb-Stick"
# optional, default is 4096
export keysize="8192"

bash /path/to/automated_install.sh

Automated install with passphrase

It is possible to pass the passphrase in an environment variable. This is generally unwise, but since we are in a temporary live enviorment I consider it safe enough. You can also put it as passphrase="your passphrase here" in automated_install.sh instead. If you add a keyfile as well, both are added.

export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"

export passphrase="your passphrase here"

bash /path/to/automated_install.sh

Automated install without passphrase

If you only want to add a keyfile and not set a passphrase, set use_passphrase to no. This is not recommended.

export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"

export use_passphrase="no"

export keyfile="/path/to/keyfile"

bash /path/to/automated_install.sh

Misc commands

I always run these command right after booting the install usb.

# I use programmer dvorak instead of qwerty
loadkeys dvorak-programmer

To Do

  • use nixos-rebuild to make an iso containing the files
  • find the location of automated_install.sh in the built iso.

Resources

I used the following resources:

Installing with old version (tested, working)

use this version of the files: old version

The commands in init.sh I run manually, (so no sed :-P)

The zfscreate.sh is used to set up a single-disk ZFS root filesystem inside of an encrypted LUKS container.

The two *.nix files have the minimum config needed for this (compare them with the generated ones in /mnt/etc/nixos/); The UUIDs should be filled-in by nixos-generate-config; the "usb_storage" addition is not needed for everyone, just like the keyfile options; the other important changes are the hostId, which is required by ZFS; and the boot.supportedFilesystems which I'm not even sure of if that's necessary

Raw
automated_install.sh
# you only need to set this to the disk to want to install to
# IT WILL BE WIPED
# but a backup of the partition table (not the contents) will be made
rootdisk="${rootdisk:-NONE}";
# use keyfile (optional)
keyfile="${keyfile:-NONE}";
keysize="${keysize:-4096}";
# set to "no" to not set a passphrase
# I highly recommed setting a passphrase and storing it in a safe location
use_passphrase="${use_passphrase:-yes}";
# You can set the passphrase here (so you can view it in plaintext,
# but do not forget to remove it.
# TODO: check if this file is on a tempfs just like /etc/nixos/configuration.nix
passphrase="${passphrase:-NONE}"
# Probably no need to change anything below or
# in the other scripts, there be dragons
# exit on error
set -e
# abort if no root disk is set
if [[ "${rootdisk}" != "NONE" ]]; echo "please set rootdisk with: \`rootdisk=/dev/disk/by-id/disk_id_for_root_device $0\`; exit 1; fi
if [[ "${use_passphrase}" == "no" ]]; echo "using a passprase is highly recommended, since keyfiles can get corrupt or lost."; fi
export rootdisk keyfile keysize use_passphrase;
# absolute location for this script (directory the files are in)
export scriptlocation=$(dirname $(readlink -f $0))
bash partition.sh
bash formatluks.sh
bash formatzfs.sh
nixos-generate-config --root /mnt
if [[ "$keyfile" != "NONE" ]];
then
# add lukskeyfile.nix
cp "${scriptlocation}/lukskeyfile.nix" /mnt/etc/nixos/
# replace `/path/to/device` with actual keyfile
echo "path to device"
blkdevice="$(basename "$(ls -l /dev/disk/by-partlabel/cryptroot | awk '{print $11}')")";
device="$(ls -l /dev/disk/by-partuuid/ | grep "$blkdevice" | awk '{print $9}')"
sed -i'' -e "s!/path/to/device!${device}!" /mnt/etc/nixos/lukskeyfile.nix;
# replace `/path/to/keyfile` with actual keyfile
sed -i '' -e "s!/path/to/keyfile!${keyfile}!" /mnt/etc/nixos/lukskeyfile.nix;
# replace placeholder keysize with actual keysize
sed -i '' -e "s!keyfile_size_here!${keysize}!" /mnt/etc/nixos/lukskeyfile.nix;
# add ./lukskeyfile.nix to the imports of configuration.nix
sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./lukskeyfile.nix!" /mnt/etc/nixos/configuration.nix
fi
# add the zfs.nix
cp "${scriptlocation}/zfs.nix" /mnt/etc/nixos/
# generate and insert a unique hostid
hostid="$(head -c4 /dev/urandom | od -A none -t x4)"
sed -i '' -e "s!cafebabe!${hostid}!" /mnt/etc/nixos/zfs.nix
# add ./zfs.nix to the imports of configuration.nix
sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./zfs.nix!" /mnt/etc/nixos/configuration.nix
echo "Done!"
echo "Please check if if everything looks allright in all the files in /mnt/etc/nixos/"
exit 0
Raw
formatluks.sh
#!/bin/bash
set -e
# temporary keyfile, will be removed (8k, ridiculously large)
dd if=/dev/urandom of=/tmp/keyfile bs=1k count=8
## now we encrypt the second partition
# pick a strong passphrase
echo "Creating the encrypted partition, follow the instructions and use a strong password!"
# formats the partition with luks and adds the temporary keyfile.
echo "YES" | cryptsetup luksFormat /dev/disk/by-partlabel/cryptroot --key-size 512 --hash sha512 --key-file /tmp/keyfile
if [[ $use_passphrase != "no" ]];
then
# sets the given passphrase or asks for one
if [[ "${passphrase}" != "NONE" ]];
then
echo "$passphrase" | cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
else
cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
fi
echo "added passphrase"
fi
if [[ "${keyfile}" != "NONE" ]];
then
cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot -d /tmp/keyfile --new-keyfile-size="${keysize}" "${keyfile}"
echo "added keyfile"
fi
# mount the cryptdisk at /dev/mapper/nixroot
cryptsetup luksOpen /dev/disk/by-partlabel/cryptroot nixroot -d /tmp/keyfile
# remove the temporary keyfile
cryptsetup luksRemoveKey /dev/disk/by-partlabel/cryptroot /tmp/keyfile
rm -f /tmp/keyfile
exit 0
Raw
formatzfs.sh
#!/bin/bash
set -e
# Install zfs in the live environment
# TODO: do this in the custom bootable iso
sed -i '' -e 's/^}/ boot.supportedFilesystems = ["zfs"];\n}/' /etc/nixos/configuration.nix;
nixos-rebuild switch
## the actual zpool create is below
#
# zpool create \
# -O atime=on \ #
# -O relatime=on \ # only write access time (requires atime, see man zfs)
# -O compression=lz4 \ # compress all the things! (man zfs)
# -O snapdir=visible \ # ever so sligthly easier snap management (man zfs)
# -O xattr=sa \ # selinux file permissions (man zfs)
# -o ashift=12 \ # 4k blocks (man zpool)
# -o altroot=/mnt \ # temp mount during install (man zpool)
# rpool \ # new name of the pool
# /dev/mapper/nixroot # devices used in the pool (in my case one, so no mirror or raid)
zpool create \
-O atime=on \
-O relatime=on \
-O compression=lz4 \
-O snapdir=visible \
-O xattr=sa \
-o ashift=12 \
-o altroot=/mnt \
rpool \
/dev/mapper/nixroot \
# dataset for / (root)
zfs create -o mountpoint=none rpool/root
echo created root dataset
zfs create -o mountpoint=legacy rpool/root/nixos
# dataset for home, make copies of all files against corruption
zfs create -o copies=2 -o mountpoint=legacy rpool/home
# dataset for swap
zfs create -o compression=off -V 8G rpool/swap
mkswap -L SWAP /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap
# mount the root dataset at /mnt
mount -t zfs rpool/root/nixos /mnt
# mount the home datset at future /home
mkdir -p /mnt/home
mount -t zfs rpool/home /mnt/home
# mount EFI partition at future /boot
mkdir -p /mnt/boot
mount /dev/disk/by-partlabel/efiboot /mnt/boot
# set boot filesystem
zpool set bootfs=rpool/root/nixos rpool
exit 0
Raw
lukskeyfile.nix
# /etc/nixos/lukskeyfile.nix (Don't forget to add it to configuration.nix)
# The minimal config required for a luksencrypted root with a keyfile
# You can skip this file if you use a passphrase
{ config, lib, pkgs, ... }:
{
# remove if you do not use an usb-stick with a keyfile
boot.kernelModules = [ "usb_storage" ];
boot.extraModulePackages = [ ];
boot.initrd.luks.devices."nixroot" = {
keyFile = "/path/to/keyfile";
keyFileSize = keyfile_size_here;
};
}
Raw
partition.sh
#!/bin/bash
set -e
# filename of partition backup
gptbackup="gpt-backup_$(basename ${rootdisk})_$(date +%s)"
# location to save the partition backup to
backuplocation="/var/tmp/${gptbackup}"
# First we make a backup of the current table
# at /tmp/gpt-backup_{id_of_device}_{current_timestamp}
# Note that this only backs-up the partition layout, not the data in it
#
# b: create backup
# $backloc: location to write to
# q: quit without saving
gdisk ${rootdisk} >/dev/null <<EOF
b
${backuplocation}
q
EOF
# copy the backup next to this script (partition.sh)
cp "${backuplocation}" "${scriptlocation}"
####################
### Partitioning ###
####################
# Now we have a backup, we can create
# a new GPT table
#
# o: create new GPT table
# y: confirm creation
#
# with the new partition table,
# we now create the EFI partition
#
# n: create new partion
# 1: partition number
# 2048: start position
# +300M: make it 300MB big
# ef00: set an EFI partition type
#
# With the EFI partition, we
# use the rest of the disk for LUKS
#
# n: create new partition
# 2: partition number
# <empty>: start partition right after first
# <empty>: use all remaining space
# 8300: set generic linux partition type
#
# We only need to set the partition labels
#
# c: change partition label
# 1: partition to label
# efiboot: name of the partition
# c: change partition label
# 2: partition to label
# cryptroot: name of the partition
#
gdisk ${rootdisk} >/dev/null <<EOF
o
y
n
1
2048
+300M
ef00
n
2
8300
c
1
efiboot
c
2
cryptroot
p
w
y
EOF
# check for the newly created partitions
# this *might* give unrelated errors;
# if so, change it to `partprobe || true`
partprobe "${rootdisk}" >/dev/null
# wait for label to show up
while [[ ! -e /dev/disk/by-partlabel/efiboot ]];
do
sleep 2;
done
# check if both labels exist
ls /dev/disk/by-partlabel/efiboot >/dev/null
ls /dev/disk/by-partlabel/cryptroot >/dev/null
## format the EFI partition
mkfs.vfat /dev/disk/by-partlabel/efiboot
exit 0
Raw
zfs.nix
# /etc/nixos/zfs.nix (Don't forget to add it to configuration.nix)
# These are the options ZFS requires, but a normal system has, of course,
# more options (like a bootloader, or installed software).
{ config, pkgs, ... }:
{
# remove this after 1st boot
# see https://nixos.org/nixos/options.html#boot.zfs.forceimportroot
boot.kernelParams = ["zfs_force=1"];
boot.zfs.forceImportRoot = false;
boot.zfs.forceImportAll = false;
boot.supportedFilesystems = [ "zfs" ];
# required by ZFS
# see https://nixos.org/nixos/options.html#networking.hostid
networking.hostId = "cafebabe";
# this enables the zfs-auto-snapshot
services.zfs.autoSnapshot = {
enable = true;
flags = "-k -p --utc";
};
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment