Skip to content

Instantly share code, notes, and snippets.

@TimoRoth

TimoRoth/tests_on_cluster.bash

Created November 11, 2018 15:35
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save TimoRoth/f6cf8f0800e56a9120aab73b22b8d55d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save TimoRoth/f6cf8f0800e56a9120aab73b22b8d55d to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. TimoRoth created this gist Nov 11, 2018.
    52 changes: 52 additions & 0 deletions tests_on_cluster.bash
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,52 @@
    #!/bin/bash
    #
    #SBATCH --job-name=oggm_tests
    #SBATCH --ntasks=1
    #SBATCH --cpus-per-task=4

    # Abort whenever a single step fails. Without this, bash will just continue on errors.
    set -e

    # We create a sub-directory for this job to store its runtime data at.
    WORKDIR="/work/$USER/oggm_tests_$SLURM_JOB_ID"
    mkdir -p "$WORKDIR"
    cd "$WORKDIR"
    echo "Workdir for this run: $WORKDIR"

    # Export the WORKDIR as environment variable so our benchmark script can use it to find its working directory.
    export WORKDIR

    # Export the global read-only download-cache-dir
    export OGGM_DOWNLOAD_CACHE=/home/data/download
    export OGGM_DOWNLOAD_CACHE_RO=1

    # All commands in the EOF block run inside of the container
    # Adjust container version to your needs, they are guaranteed to never change after their respective day has passed.
    srun -n 1 -c "${SLURM_JOB_CPUS_PER_NODE}" singularity exec docker://oggm/oggm:20181110 bash -s <<EOF
    set -e
    # Setup a fake home dir inside of our workdir, so we don't clutter the actual shared homedir with potentially incompatible stuff.
    export HOME="$WORKDIR/fake_home"
    mkdir "\$HOME"
    # Create a venv that _does_ use system-site-packages, since everything is already installed on the container.
    # We cannot work on the container itself, as the base system is immutable.
    python3 -m venv --system-site-packages "$WORKDIR/oggm_env"
    source "$WORKDIR/oggm_env/bin/activate"
    # Make sure latest pip is installed
    pip install --upgrade pip setuptools
    # Install our test OGGM, using a fixed version here, to show how this will always produce identical results.
    pip install --upgrade "git+https://github.com/OGGM/oggm.git@3809cc80bc3d82ecd1dfa5e948c66f0ea864bf4c"
    # Run the actual tests.
    pytest --run-slow --run-download --mpl-oggm --pyargs oggm
    EOF

    # Clean up after ourselves
    rm -rf "$WORKDIR"

    # Print a final message so you can actually see it being done in the output log.
    echo "DONE"