Kargo3.0 Bare Metal GitOps Hypervisor

WARNING: Microk8s is currently impacted by BUG #3085 please see bug workaround instructions to remediate until patch is released to stable channels!

01. Install OS

• Download Fedora 36 Server
• Create Fedora Install USB
• Plug into Desktop/Server Lab System
• Boot to Fedora USB Installer
• Erase HDD/SDD devices
• Install Fedora
• Reboot

02 Configure br0

03 Enable nested Virtualization && Disable selinux (not for production)

Warning: Disable selinux at your own risk!

sudo grubby --update-kernel=ALL --args 'selinux=0 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci pci=realloc'

04 Enable br_netfilter for Calico

echo "br_netfilter" | sudo tee -a /etc/modules
sudo modprobe br_netfilter

05 Update & Install Packages

sudo dnf update -y
sudo dnf install -y firewalld kernel-modules dnf-automatic kubernetes-client helm snapd dracut-squash squashfs-tools squashfuse fuse jq
sudo ln -s /var/lib/snapd/snap /snap

06 Install Binaries

curl --output /tmp/virtctl -L https://github.com/kubevirt/kubevirt/releases/download/$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | awk -F '[",]' '/tag_name/{print $4}')/virtctl-$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | awk -F '[",]' '/tag_name/{print $4}')-linux-amd64
sudo install -o root -g root -m 0755 /tmp/virtctl /usr/local/bin/virtctl

07 Disable Firewall

Warning: Disable Firewalld at your own risk!

sudo systemctl disable firewalld
sudo systemctl stop firewalld

08 Reboot

sudo shutdown -r now

09 Install Microk8s & Deploy Plugins

# Install Microk8s
# ! Currently installing latest/edge until bug #3085 is resolved
sudo snap install core
sudo snap install microk8s --channel=latest/edge --classic && sleep 15
sudo microk8s enable && sudo microk8s status -w && sleep 3
sudo microk8s start && sudo microk8s status -w && sleep 3
sudo usermod -aG microk8s $USER

# Enable Plugins
sudo microk8s enable dns && sudo microk8s status -w && sleep 3
sudo microk8s enable storage && sudo microk8s status -w && sleep 3
sudo microk8s enable community && sudo microk8s status -w
sudo microk8s enable multus && sudo microk8s status -w

# Setup KUBECONFIG
mkdir -p ~/.kube && sudo microk8s config > ~/.kube/config
sudo chown -f -R $USER ~/.kube && chmod 600 ~/.kube/config
kubectl get po -A

10 Install Containerized Data Importer

• Not required for ephemeral vm's like vyos

curl -sL https://github.com/kubevirt/containerized-data-importer/releases/download/$(curl -s https://api.github.com/repos/kubevirt/containerized-data-importer/releases/latest | awk -F '[",]' '/tag_name/{print $4}')/cdi-operator.yaml | kubectl apply -f -
curl -sL https://github.com/kubevirt/containerized-data-importer/releases/download/$(curl -s https://api.github.com/repos/kubevirt/containerized-data-importer/releases/latest | awk -F '[",]' '/tag_name/{print $4}')/cdi-cr.yaml | kubectl apply -f -

11 Cert Manager

helm repo add jetstack https://charts.jetstack.io; helm repo update
helm upgrade --install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true

12 Install Kubevirt

• Install Kubevirt

curl -sL https://github.com/kubevirt/kubevirt/releases/download/$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | awk -F '[",]' '/tag_name/{print $4}')/kubevirt-operator.yaml | kubectl apply -f -
curl -sL https://github.com/kubevirt/kubevirt/releases/download/$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | awk -F '[",]' '/tag_name/{print $4}')/kubevirt-cr.yaml | kubectl apply -f -
kubectl -n kubevirt wait kv kubevirt --for condition=Available

• (Optional) Notable FeatureGates

cat <<EOF | kubectl apply -f -
---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    developerConfiguration: 
      featureGates:
        - LiveMigration
        - DataVolumes
        - ExpandDisks
        - ExperimentalIgnitionSupport
        - Sidecar
        - HostDevices
        - Snapshot
        - HotplugVolumes
        - ExperimentalVirtiofsSupport
        - GPU
EOF

13 Create Kubevirt Resources

• Create SSH Key Secret

# Create SSH Key Secret
ls ~/.ssh/id_rsa.pub >/dev/null || ssh-keygen
kubectl create secret generic kubevirt-sshpubkey-kc2user \
    --from-file=key1=$HOME/.ssh/id_rsa.pub \
    --dry-run=client -oyaml \
  | kubectl apply -f -

kubectl get secret -oyaml kubevirt-sshpubkey-kc2user | awk '/key1:/{print $2}' | base64 -d

• Create VM Network Attachment Definition

cat <<EOF | kubectl apply -f -
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: nadbr0
spec:
  config: '{"cniVersion":"0.3.1","name":"br0","plugins":[{"type":"bridge","bridge":"br0","ipam":{}},{"type":"tuning"}]}'
EOF

14 Create VMs

*Ubuntu 22.04 Jammy Minimal

kubectl apply -f https://gist.github.com/usrbinkat/c8b56fb703328147c796bc4356b029b5/raw/86747680e7f8b3cb641c5464d9d4cd083bb29596/ubuntu-jammy-minimal.yaml

*Ubuntu 22.04 Jammy with xRDP Ubuntu Desktop

kubectl apply -f https://gist.github.com/usrbinkat/c8b56fb703328147c796bc4356b029b5/raw/2cbb9883867c6ad02dd72fdbb1b10008cec1a21f/ubuntu-jammy.yaml

15 Wait for image download and cdi import

16 Execute Cmds to find IP and access serial console ttyS0

kubectl get vmi
virtctl console ubuntu-rdp

17 SSH to VM at it's IP address with ssh key used to create secret

18 Connect to VM's RDP session at it's IP address with credentials:

kc2user:kc2user