Skip to content

Instantly share code, notes, and snippets.

@jkglasbrenner
Last active January 28, 2019 19:20
Show Gist options
  • Save jkglasbrenner/a51b4de9a1088c7d8f6b5c5983738851 to your computer and use it in GitHub Desktop.
Save jkglasbrenner/a51b4de9a1088c7d8f6b5c5983738851 to your computer and use it in GitHub Desktop.
CSI 702 virtual machine
# config file for ansible -- https://ansible.com/
# ===============================================
[defaults]
roles_path = /home/vagrant/.ansible/roles

HOWTO - CSI 702 - Setting up your virtual machine

Installing Virtualbox and Vagrant

Windows

Instructions below adapted from Ref. [1].

  1. Install the latest versions of the following software:

    After installing Vagrant, you will need to reboot your computer.

  2. Open Git bash by right-clicking your desktop or the white space inside a folder (Windows file explorer) and clicking Git Bash Here. If this is your first time using git on your Windows machine, you should set your user information by running these two commands:

    git config --global user.name "FirstName LastName"
    git config --global user.email "[email protected]"

    Replace FirstName LastName and [email protected] with your own information.

macOS

Open a terminal window and follow the instructions below. Instructions adapted from Ref. [2].

  1. Xcode must be installed before you can do anything else. You can install Xcode by either downloading it from Apple's website, https://developer.apple.com/xcode/, or by running the following command in the terminal:

    xcode-select --install
  2. (Possibly optional) Install XQuartz by downloading the dmg file from the project's website: https://www.xquartz.org/

  3. Install Homebrew if it is not currently installed on your computer:

    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

    Check your $HOME/.bash_profile file and see if /usr/local/bin is set in your $PATH environment variable. If it is not, run echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile to append a command that will set it. Close and reopen your terminal window after doing this.

    To check that everything is working, run brew doctor. For basic information on what you can do with Homebrew, see this page: http://sourabhbajaj.com/mac-setup/Homebrew/Usage.html.

  4. Install Homebrew-Cask if it is not already installed:

    brew tap caskroom/cask
  5. Install git using Homebrew:

    brew install git
  6. If this is your first time using git, you should set your user information by running these two commands:

    git config --global user.name "FirstName LastName"
    git config --global user.email "[email protected]"

    Replace FirstName LastName and [email protected] with your own information.

  7. Install Virtualbox using Homebrew-Cask:

    brew cask install virtualbox
  8. Install Vagrant using Homebrew-Cask:

    brew cask install vagrant
    brew cask install vagrant-manager

Ubuntu

  1. Install git and openssh-client if you haven't already:

    sudo apt-get update
    sudo apt-get install git openssh-client
  2. If this is your first time using git, you should set your user information by running these two commands:

    git config --global user.name "FirstName LastName"
    git config --global user.email "[email protected]"

    Replace FirstName LastName and [email protected] with your own information.

  3. Install the latest version of Virtualbox:

    wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -
    wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -
    sudo add-apt-repository "deb [arch=amd64] http://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib"
    sudo apt-get update
    sudo apt-get install virtualbox-6.0
  4. Install the latest version of Vagrant:

    # Install gdebi if you haven't done so already
    sudo apt-get install gdebi
    # Download Vagrant
    wget https://releases.hashicorp.com/vagrant/2.2.3/vagrant_2.2.3_x86_64.deb
    # Install Vagrant with gdebi
    sudo gdebi vagrant_2.2.3_x86_64.deb

Provisioning the virtual machine

  1. Create a folder for the class, for example:

    mkdir -p ~/Documents/class/csi702

    This will be the root folder for the virtual machine, so all files and directories related to the class will go under this folder.

  2. Navigate to this GitHub Gist, https://gist.github.com/jkglasbrenner/a51b4de9a1088c7d8f6b5c5983738851, click Download ZIP and save the file to your class folder. Extract the file contents directly into the folder. You should now see the following files in your class folder:

    /path/to/class/folder
    ├── ansible.cfg
    ├── CSI702_provision_virtual_machine.md
    ├── DESCRIPTION
    ├── environment.yml
    ├── Makefile
    ├── playbook.yml
    ├── requirements.yml
    └── Vagrantfile  
    

    If the files were unzipped into a new directory, move them into the class folder. Do not delete these files after setup.

  3. Download the official base image for Ubuntu 18.04:

    vagrant box add ubuntu/bionic64
  4. Provision your virtual machine by running

    vagrant up

    This will take a while. Please note that this step needs to be done from within the class folder where you stored all the unzipped files.

  5. After the provisioning is complete, you can connect to the virtual machine by running

    vagrant ssh

Install conda environment

  1. After connecting to your virtInstall conda environmentual machine with vagrant ssh, run the following commands to set up your conda environment:

    cd /vagrant
    make all

    This will take several minutes to complete.

  2. After make all is done, run:

    make clean

    This will remove anaconda's temporary files and free up some of your virtual disk space To disconnect from the virtual machine, run

Basic usage

After you install the conda environment, you are ready to start using your virtual machine in CSI 702. Below are a few tips and reminders that are specific to using a Vagrant-provisioned virtual machine. Otherwise, you are expected to be familiar with the basics of using a full Linux environment and navigating via the command line.

  • The steps in the Install conda environment section created a conda virtual environment named csi702. To activate it, run:

    conda activate csi702
  • The /vagrant directory is a shared folder. If you followed the above directions, you should see any files and folders inside the class directory you created earlier.

  • You can serve a Jupyter Lab instance from within the virtual machine and access it using your host OS's web browser. The easy way to do this is:

    cd /vagrant
    make lab

    Open your web browser (Google Chrome or Firefox are recommended) and type localhost:8888 in your address bar and press Enter. To stop the Jupyter Lab instance, just press Ctrl+C and answer y to the disconnect prompt.

  • To disconnect from your virtual machine, run:

    logout
  • The virtual machine will run in the background of your computer until you shut it down. To shut down the virtual machine, from the class folder, run

    vagrant halt

    This can only be done after you've disconnected with logout.

  • To restart the virtual machine after running vagrant halt, from the class folder, run:

    vagrant up
  • To remove the virtual machine from your computer, from the class folder, run

    vagrant destroy

References

[1] http://ifcuriousthenlearn.com/blog/2015/08/18/vagrant-linux/
[2] http://sourabhbajaj.com/mac-setup/

Title: CSI 702 virtual machine
Depends:
R (>= 3.5)
Imports:
knitr (>= 1.21),
readr (>= 1.3.1),
rmarkdown (>= 1.11)
---
name: csi702
channels:
- defaults
- conda-forge
dependencies:
- autopep8
- cython
- ipywidgets
- jedi
- jupyter
- jupyterlab
- jupyter_contrib_nbextensions
- jupyter_nbextensions_configurator
- matplotlib
- nodejs=10.13.0=he6710b0_0
- numpy
- pandas
- pandoc=2.5=1
- python=3.7.2
- rise
- scipy
- seaborn
- xeus-cling
- pip:
- black
- cite2c
- flake8
- grip
- importmagic
- ipympl
- jupyterlab_code_formatter
- jupyterlab_git
- jupyterlab_github
- jupyterlab_templates
- nb_pdf_template
- nbdime
- pip==19.0.1
- pydocstyle
- PyHamcrest
- rope
- rpy2
...
RM = rm
FIND = find
COPY = cp
LN = ln
CONDA = conda
CONDA_ENV_FILE = environment.yml
CONDA_ENV_NAME = csi702
PY = python
PY_SETUP = setup.py
JUPYTER = jupyter
JUPYTERLAB_EXTENSIONS = jupyterlab_vim \
jupyterlab_templates \
@jupyterlab/git \
@jupyterlab/github \
@mflevine/jupyterlab_html \
@jupyter-widgets/jupyterlab-manager \
jupyter-matplotlib \
@ryantam626/jupyterlab_code_formatter \
jupyterlab-python-file
JUPYTERLAB_SERVEREXTENSION = jupyterlab_templates \
jupyterlab_git \
jupyterlab_code_formatter
R = Rscript
define makefile_help
@echo 'Makefile for CSI 702 virtual machine '
@echo ' '
@echo 'Usage: '
@echo ' make all Install all dependencies '
@echo ' make clean Clean up conda packages '
@echo ' make help display this message '
@echo ' make env create conda venv and install deps '
@echo ' '
endef
define conda_cleanup
bash -lc "$(CONDA) clean -y -a"
sudo bash -lc "conda clean -y --packages"
endef
define install_python_deps
bash -lc " \
$(CONDA) activate $(CONDA_ENV_NAME) && \
$(PY) -m nb_pdf_template.install --minted \
"
endef
define install_r_deps
bash -lc "$(CONDA) activate $(CONDA_ENV_NAME) && \
$(R) -e \"install.packages('remotes', repos = 'https://cran.rstudio.com')\" \
-e \"remotes::install_deps()\""
endef
define update_conda_env
bash -lc "$(CONDA) env update --file $(CONDA_ENV_FILE)"
endef
define launch_jupyter_lab
bash -lc " \
$(CONDA) activate $(CONDA_ENV_NAME) && \
$(JUPYTER) lab --ip='0.0.0.0' \
"
endef
define install_jupyterlab_extensions
bash -lc " \
$(CONDA) activate $(CONDA_ENV_NAME) && \
$(JUPYTER) labextension install $(1) \
"
endef
define install_serverextension
bash -lc " \
$(CONDA) activate $(CONDA_ENV_NAME) && \
$(JUPYTER) serverextension enable --py $(1) \
"
endef
all : env pydeps rdeps labextensions serverextensions
help :
$(call makefile_help)
clean :
$(call conda_cleanup)
env :
$(call update_conda_env)
lab :
$(call launch_jupyter_lab)
labextensions :
$(foreach extension,$(JUPYTERLAB_EXTENSIONS),$(call install_jupyterlab_extensions,--no-build $(extension));)
bash -lc "$(CONDA) activate $(CONDA_ENV_NAME) && $(JUPYTER) lab build"
pydeps :
$(call install_python_deps)
rdeps :
$(call install_r_deps)
serverextensions :
$(foreach serverextension,$(JUPYTERLAB_SERVEREXTENSION),$(call install_serverextension,$(serverextension));)
.PHONY : all clean env help lab labextensions pydeps rdeps serverextensions $(DIRS)
#%Module
set _module_name [module-info name]
set pkg_name Intel_Math_Kernel_Library
set pkg_version {{ mkl_version }}
proc ModulesHelp { } {
global _module_name pkg_name pkg_version pkg_path
puts stderr "The $_module_name modulefile defines the default system paths"
puts stderr "and other environment variables needed to use all components of "
puts stderr "$pkg_name version $pkg_version, namely:"
puts stderr "Intel Math Kernel Library or MKL (version {{ mkl_version }})"
}
prepend-path LIBRARY_PATH {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/mkl/lib/intel64_lin};
prepend-path INTEL_LICENSE_FILE {/opt/intel/licenses};
prepend-path INTEL_LICENSE_FILE {/opt/intel/licenses};
append-path INTEL_LICENSE_FILE {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/licenses};
append-path INTEL_LICENSE_FILE {/opt/intel/licenses};
prepend-path CPATH {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/mkl/include};
prepend-path LD_LIBRARY_PATH {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/mkl/lib/intel64_lin};
setenv MKLROOT {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/mkl};
prepend-path NLSPATH {/opt/intel/compilers_and_libraries_{{ mkl_version }}/linux/mkl/lib/intel64_lin/locale/%l_%t/%N};
---
- hosts: all
become: yes
pre_tasks:
# See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
- name: Ensure 'man' directory exists.
become: yes
file:
path: /usr/share/man/man1
state: directory
recurse: yes
when:
- ansible_distribution == 'Ubuntu'
- ansible_distribution_version == '18.04'
roles:
- jkglasbrenner.miniconda
- jkglasbrenner.environment_modules
tasks:
- name: Add key for CRAN repository
apt_key:
id: "{{ apt_keys.cran_key.id }}"
keyserver: "{{ apt_keys.cran_key.keyserver }}"
- name: Add key for Intel repository
apt_key:
id: "{{ apt_keys.intel_key.id }}"
url: "{{ apt_keys.intel_key.url }}"
- name: Add CRAN and Intel repositories
apt_repository:
repo: "{{ item }}"
loop: "{{ apt_repositories | flatten(levels=1) }}"
- name: Install packages
apt:
name:
- aptitude
- automake
- bash
- binutils-dev
- build-essential
- bzip2
- clang
- clang-format
- clang-tidy
- clang-tools
- cmake
- dos2unix
- emacs25-nox
- ffmpeg
- fonts-liberation
- g++
- gcc
- gfortran
- git
- git-lfs
- gnuplot-nox
- imagemagick
- intel-mkl-64bit-2019.1-053
- libboost-all-dev
- libbz2-dev
- libclang-dev
- libgif-dev
- libjpeg-dev
- libperl-dev
- libpng-dev
- libssl-dev
- libtool
- lmodern
- mpi-default-dev
- openmpi-bin
- openssh-client
- pigz
- pkg-config
- python-dev
- python3-dev
- r-base
- r-base-dev
- rename
- texinfo
- texlive
- texlive-fonts-extra
- texlive-fonts-recommended
- texlive-formats-extra
- texlive-latex-extra
- texlive-latex-recommended
- texlive-luatex
- texlive-publishers
- texlive-science
- texlive-xetex
- ttf-mscorefonts-installer
- vim
- xzdec
update_cache: yes
state: present
install_recommends: no
- name: Create directory for storing MKL modulefiles
file:
path: "{{ modules_dir }}/modulefiles/mkl"
state: directory
recurse: yes
mode: 0755
owner: root
group: root
- name: Install modulefile for loading the MKL library
template:
src: "mklmodule.j2"
dest: "{{ modules_dir }}/modulefiles/mkl/{{ mkl_version }}"
mode: 0644
owner: root
group: root
- name: Install direnv
block:
- name: Check for direnv installation
stat:
path: "{{ direnv.dest }}"
register: direnv_binary
- name: Download and install direnv
get_url:
url: "{{ direnv.base_url }}/v{{ direnv.version }}/{{ direnv.arch }}"
checksum: "{{ direnv.checksum }}"
dest: "{{ direnv.dest }}"
mode: "0755"
owner: root
group: root
when: not direnv_binary.stat.exists
- name: Check for UPC++ installation
stat:
path: "{{ upcxx.dest }}/{{ upcxx.version }}/bin/upcxx"
register: upcxx_binary
- name: Install UPC++ library
when: not upcxx_binary.stat.exists
block:
- name: Download and extract UPC++ source tarfile
unarchive:
src: "{{ upcxx.base_url }}/upcxx-{{ upcxx.version }}.tar.gz"
dest: "/tmp/"
remote_src: yes
creates: "{{ upcxx.dest }}/{{ upcxx.version }}/bin/upcxx"
- name: Compile and install UPC++ library
command: "./install {{ upcxx.dest }}/{{ upcxx.version }}"
args:
chdir: "/tmp/upcxx-{{ upcxx.version }}"
- name: Remove UPC++ source directory
file:
path: "/tmp/upcxx-{{ upcxx.version }}"
state: absent
- name: Create directory for installing R libraries
file:
path: "{{ lookup('env', 'HOME') }}/R/x86_64-pc-linux-gnu-library/3.5/"
state: directory
recurse: yes
owner: vagrant
group: vagrant
vars:
apt_keys:
intel_key:
url: https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
id: BF4385F91CA5FC005AB39E1C1A8497B11911E097
cran_key:
keyserver: keyserver.ubuntu.com
id: E298A3A825C0D65DFD57CBB651716619E084DAB9
apt_repositories:
- "deb https://cloud.r-project.org/bin/linux/ubuntu {{ ansible_distribution_release }}-cran35/"
- "deb https://apt.repos.intel.com/mkl all main"
direnv:
version: 2.19.0
checksum: "sha256:99e6953a297ea442ffb5aa8e3edc3590e1870b742a7a0a6a9cdd94847182f8af"
arch: direnv.linux-amd64
base_url: https://github.com/direnv/direnv/releases/download
dest: /usr/local/bin/direnv
miniconda_dir: "/opt/miniconda"
mkl_version: 2019.1.144
modules_example_modulefiles_to_remove:
- dot
- module-git
- module-info
- "{{ 'null' }}"
modules_parent_dir: /usr/local
modules_dir: "{{ modules_parent_dir }}/Modules"
upcxx:
version: 2018.9.0
base_url: https://bitbucket.org/berkeleylab/upcxx/downloads
dest: "/opt/upcxx/gnu"
...
---
- src: jkglasbrenner.miniconda
- src: jkglasbrenner.environment_modules
...
# -*- mode: ruby -*-
# vi: set ft=ruby :
$install_ansible = <<-SCRIPT
sudo apt-get update
sudo apt-get install -y software-properties-common
sudo apt-add-repository -y ppa:ansible/ansible
sudo apt-get update
sudo apt-get install -y ansible
SCRIPT
unless Vagrant.has_plugin?("vagrant-vbguest")
system("vagrant plugin install vagrant-vbguest")
puts "Dependencies installed, please try the 'vagrant up' command again."
exit
end
unless Vagrant.has_plugin?("vagrant-disksize")
system("vagrant plugin install vagrant-disksize")
puts "Dependencies installed, please try the 'vagrant up' command again."
exit
end
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.disksize.size = '20GB'
config.vm.network "forwarded_port", guest: 8888, host: 8888, auto_correct: true
config.vm.provider "virtualbox" do |vb|
vb.name = "csi702"
vb.gui = false
# Source: https://github.com/rdsubhas/vagrant-faster
host = RbConfig::CONFIG['host_os']
if host =~ /darwin/
cpus = `sysctl -n hw.ncpu`.to_i
mem = `sysctl -n hw.memsize`.to_i / 1024 / 1024
elsif host =~ /linux/
cpus = `nproc`.to_i
mem = `grep 'MemTotal' /proc/meminfo | sed -e 's/MemTotal://' -e 's/ kB//'`.to_i / 1024
elsif host =~ /mswin|mingw|cygwin/
cpus = `wmic cpu Get NumberOfCores`.split[1].to_i
mem = `wmic computersystem Get TotalPhysicalMemory`.split[1].to_i / 1024 / 1024
end
mem = mem / 4
vb.memory = mem
vb.cpus = cpus
vb.customize ["modifyvm", :id, "--cpuexecutioncap", "100"]
end
config.vm.provision :shell do |shell|
shell.name = "Install dependencies with apt package manager"
shell.inline = $install_ansible
shell.privileged = true
end
config.vm.provision :ansible_local do |ansible|
ansible.playbook = "playbook.yml"
ansible.galaxy_role_file = "requirements.yml"
ansible.galaxy_roles_path = "/home/vagrant/.ansible/roles"
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment