Skip to content

Instantly share code, notes, and snippets.

@xahare
Last active July 11, 2023 15:43
Show Gist options
  • Save xahare/0f2078fc8c52e7ddece1e5ba70c6d5fc to your computer and use it in GitHub Desktop.
Save xahare/0f2078fc8c52e7ddece1e5ba70c6d5fc to your computer and use it in GitHub Desktop.

Revisions

  1. xahare revised this gist Oct 20, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -17,7 +17,7 @@ and move it to your template vm. You can use gdebi to install it on the command

    Do the following steps as root.

    In the template vm, we need all the packages for qemu, libvirt, and vagrant-libvirt.
    In the template vm, we need all the packages for qemu, libvirt, and vagrant-libvirt. Dont forget to remove /home/user/QubesIncoming from the template after installing vagrant.

    apt install qemu-kvm libvirt-clients libvirt-daemon-system \
    bash-completion debhelper gem2deb libvirt-dev pkg-config \
  2. xahare revised this gist Oct 20, 2017. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -51,6 +51,7 @@ Do the following as user. This installs and sets the libvirt plugin to use qemu
    libvirt.cpu_model = "qemu64"
    end
    end
    EOF

    Finally, add these two lines to users ~/.bashrc

    @@ -64,6 +65,8 @@ Appvm Setup

    Set the private storage max size higher, at least 20gigs. you'll want more memory too.

    If you want to use an existing appvm, you'll need to enable persistent storage of libvirt and make all the user changes above.

    A brief diversion to virt-manager
    ---------------------------------

  3. xahare revised this gist Oct 20, 2017. 1 changed file with 31 additions and 35 deletions.
    66 changes: 31 additions & 35 deletions vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -15,6 +15,8 @@ Currently, this only works with a (debian-9)[https://www.qubes-os.org/doc/templa
    From an appvm, download vagrant from https://vagrantup.com , check the sha256sum and gpg signature,
    and move it to your template vm. You can use gdebi to install it on the command line and pull in any dependencies.

    Do the following steps as root.

    In the template vm, we need all the packages for qemu, libvirt, and vagrant-libvirt.

    apt install qemu-kvm libvirt-clients libvirt-daemon-system \
    @@ -28,28 +30,39 @@ Add user to the libvirt and qemu groups,

    adduser user libvirt
    adduser user libvirt-qemu

    Add libvirt to persistent storage. see https://www.qubes-os.org/doc/bind-dirs/ for more on that.

    Thats all you need in the template vm. Shutdown so appvms can inherit the changes.
    mkdir -p /rw/config/qubes-bind-dirs.d
    cat << EOF >> /rw/config/qubes-bind-dirs.d/50_user.conf
    binds+=( '/etc/libvirt' )
    binds+=( '/var/lib/libvirt' )
    EOF

    Appvm Setup
    -----------
    Do the following as user. This installs and sets the libvirt plugin to use qemu in emulation instead of virtualization mode.

    Set the private storage max size higher, at least 20gigs. you'll want more memory too.
    vagrant plugin install vagrant-libvirt

    cat << EOF > ~/.vagrant.d/Vagrantfile
    Vagrant.configure("2") do |config|
    config.vm.provider "libvirt" do |libvirt|
    libvirt.driver = "qemu"
    libvirt.cpu_mode = "custom"
    libvirt.cpu_model = "qemu64"
    end
    end

    You'll need these two lines in your ~/.bashrc.
    Finally, add these two lines to users ~/.bashrc

    export LIBVIRT_DEFAULT_URI="qemu:///system"
    export VAGRANT_DEFAULT_PROVIDER=libvirt
    export LIBVIRT_DEFAULT_URI="qemu:///system"
    export VAGRANT_DEFAULT_PROVIDER=libvirt

    you'll also need to make the libvirt folders persistent using bind-dirs. Bind-dirs has a side effect of using appvms private storage since its mounted from /rw, so you can have more space for disk images.
    Thats all you need in the template vm. Shutdown so appvms can inherit the changes.

    sudo mkdir -p /rw/config/qubes-bind-dirs.d
    sudo cat << EOF > /rw/config/qubes-bind-dirs.d/50_user.conf
    binds+=( '/etc/libvirt' )
    binds+=( '/var/lib/libvirt' )
    EOF
    Appvm Setup
    -----------

    To enable all this, reboot the appvm.
    Set the private storage max size higher, at least 20gigs. you'll want more memory too.

    A brief diversion to virt-manager
    ---------------------------------
    @@ -61,29 +74,10 @@ Virt-manager is not strictly needed by vagrant, but it makes it easier to see wh
    Back to Vagrant
    ---------------

    All thats left is the libvirt plugin, as user (not root)

    vagrant plugin install vagrant-libvirt

    Another useful plugin is vagrant-mutate to convert virtualbox vagrant boxes to libvirt.

    You'll also need to make a global vagrant file setting guests to run in emulation instead
    of virtualization. put this in ~/.vagrant.d/Vagrantfile

    Vagrant.configure("2") do |config|
    config.vm.provider "libvirt" do |libvirt|
    libvirt.driver = "qemu"
    libvirt.cpu_mode = "custom"
    libvirt.cpu_model = "qemu64"
    end
    end

    Heres a sample Vagrantfile you can try. its an updated one from the vagrant-libvirt readme. make a folder and a file called Vagrantfile with this in it.

    Vagrant.configure("2") do |config|
    config.vm.define :test_vm do |test_vm|
    test_vm.vm.box = "fedora/26-cloud-base"
    end
    config.vm.box = "fedora/26-cloud-base"
    end

    Then
    @@ -93,11 +87,13 @@ Then
    and in a few minutes, the prompt should return and you can "vagrant ssh".

    time for vagrant up:

    real 2m41.135s
    user 0m2.956s
    sys 0m0.345s

    time on a real linux box
    time on a real linux box:

    real 0m37.065s
    user 0m7.900s
    sys 0m2.226s
  4. xahare revised this gist Jul 28, 2017. 1 changed file with 6 additions and 6 deletions.
    12 changes: 6 additions & 6 deletions vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -93,14 +93,14 @@ Then
    and in a few minutes, the prompt should return and you can "vagrant ssh".

    time for vagrant up:
    real 2m41.135s
    user 0m2.956s
    sys 0m0.345s
    real 2m41.135s
    user 0m2.956s
    sys 0m0.345s

    time on a real linux box
    real 0m37.065s
    user 0m7.900s
    sys 0m2.226s
    real 0m37.065s
    user 0m7.900s
    sys 0m2.226s

    Other Approaches
    ----------------
  5. xahare revised this gist Jul 27, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -107,7 +107,7 @@ Other Approaches

    * Virtualbox in an HVM. Limited to 32 bit guests, also emulation instead of virtualization.
    * LXC provider. Should be the same as any other linux. Havent tried, as its limited to linux.
    * Ssh to a linux box running libvirt with kvm. This is what i actually do.
    * Ssh to a linux box running libvirt with kvm. This is what i actually do. Faster and you get the benefits of remote tmux.

    Links
    -----
  6. xahare revised this gist Jul 27, 2017. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -102,6 +102,13 @@ time on a real linux box
    user 0m7.900s
    sys 0m2.226s

    Other Approaches
    ----------------

    * Virtualbox in an HVM. Limited to 32 bit guests, also emulation instead of virtualization.
    * LXC provider. Should be the same as any other linux. Havent tried, as its limited to linux.
    * Ssh to a linux box running libvirt with kvm. This is what i actually do.

    Links
    -----

  7. xahare revised this gist Jul 27, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -5,7 +5,7 @@ Vagrant on Qubes-OS

    (Qubes-OS)[https://www.qubes-os.org] "A Reasonably Secure Operating System."

    This is a guide on to using vagrant on qubes-os with qemu using the libvirt provider. because qubes-os does not support nested virtualization, you'r stuck with emulation. If you want performance, use a system with a proper vagrant setup.
    This is a guide on to using vagrant on qubes-os with qemu using the libvirt provider. Because qubes-os does not support nested virtualization, you'r stuck with emulation. If you want performance, use a system with a proper vagrant setup.

    Template Setup
    --------------
  8. xahare created this gist Jul 27, 2017.
    113 changes: 113 additions & 0 deletions vagrant-qubes.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,113 @@
    Vagrant on Qubes-OS
    -------------------

    (Vagrant)[https://www.vagrantup.com] "Development Environments Made Easy"

    (Qubes-OS)[https://www.qubes-os.org] "A Reasonably Secure Operating System."

    This is a guide on to using vagrant on qubes-os with qemu using the libvirt provider. because qubes-os does not support nested virtualization, you'r stuck with emulation. If you want performance, use a system with a proper vagrant setup.

    Template Setup
    --------------

    Currently, this only works with a (debian-9)[https://www.qubes-os.org/doc/template/debian/upgrade-8-to-9/] templatevm. fedora25 and debian8 have conflicting xen libraries.

    From an appvm, download vagrant from https://vagrantup.com , check the sha256sum and gpg signature,
    and move it to your template vm. You can use gdebi to install it on the command line and pull in any dependencies.

    In the template vm, we need all the packages for qemu, libvirt, and vagrant-libvirt.

    apt install qemu-kvm libvirt-clients libvirt-daemon-system \
    bash-completion debhelper gem2deb libvirt-dev pkg-config \
    rake libvirt-daemon ebtables dnsmasq libxslt-dev \
    libxml2-dev libvirt-dev zlib1g-dev ruby-dev virt-manager gdebi

    gdebi vagrant-XXX.deb

    Add user to the libvirt and qemu groups,

    adduser user libvirt
    adduser user libvirt-qemu

    Thats all you need in the template vm. Shutdown so appvms can inherit the changes.

    Appvm Setup
    -----------

    Set the private storage max size higher, at least 20gigs. you'll want more memory too.

    You'll need these two lines in your ~/.bashrc.

    export LIBVIRT_DEFAULT_URI="qemu:///system"
    export VAGRANT_DEFAULT_PROVIDER=libvirt

    you'll also need to make the libvirt folders persistent using bind-dirs. Bind-dirs has a side effect of using appvms private storage since its mounted from /rw, so you can have more space for disk images.

    sudo mkdir -p /rw/config/qubes-bind-dirs.d
    sudo cat << EOF > /rw/config/qubes-bind-dirs.d/50_user.conf
    binds+=( '/etc/libvirt' )
    binds+=( '/var/lib/libvirt' )
    EOF

    To enable all this, reboot the appvm.

    A brief diversion to virt-manager
    ---------------------------------

    Once rebooted try virt-manager. when you first start it, it will complain about not being able to connect to xen. Delete that connection by right clicking on "xen" under "Name" and delete. Then make a new one with File/"Add connection" and qemu/kvm. The generated uri should be "qemu:///system"

    Virt-manager is not strictly needed by vagrant, but it makes it easier to see whats going on, and to use the console of desktop vagrant boxes. You can also use virt-manager for remote sessions, to hopefully faster libvirt hosts.

    Back to Vagrant
    ---------------

    All thats left is the libvirt plugin, as user (not root)

    vagrant plugin install vagrant-libvirt

    Another useful plugin is vagrant-mutate to convert virtualbox vagrant boxes to libvirt.

    You'll also need to make a global vagrant file setting guests to run in emulation instead
    of virtualization. put this in ~/.vagrant.d/Vagrantfile

    Vagrant.configure("2") do |config|
    config.vm.provider "libvirt" do |libvirt|
    libvirt.driver = "qemu"
    libvirt.cpu_mode = "custom"
    libvirt.cpu_model = "qemu64"
    end
    end

    Heres a sample Vagrantfile you can try. its an updated one from the vagrant-libvirt readme. make a folder and a file called Vagrantfile with this in it.

    Vagrant.configure("2") do |config|
    config.vm.define :test_vm do |test_vm|
    test_vm.vm.box = "fedora/26-cloud-base"
    end
    end

    Then

    vagrant up

    and in a few minutes, the prompt should return and you can "vagrant ssh".

    time for vagrant up:
    real 2m41.135s
    user 0m2.956s
    sys 0m0.345s

    time on a real linux box
    real 0m37.065s
    user 0m7.900s
    sys 0m2.226s

    Links
    -----

    * https://www.vagrantup.com
    * https://wiki.debian.org/KVM
    * https://github.com/vagrant-libvirt/vagrant-libvirt
    * https://packages.debian.org/source/stretch/vagrant
    * https://packages.debian.org/source/stretch/ruby-libvirt
    * https://www.qubes-os.org/doc/bind-dirs/