Geoyi/python_environment_setup.md

When you're working on multiple coding projects, you might want a couple different version of python and/or modules installed. That way you can keep each project in its own sandbox instead of trying to juggle multiple projects (each with different needs) on your default computer system. Concrete examples of this situation:

1. Working on 2+ projects that each have their own dependencies; e.g., a Python 2 project and a Python 3 project, or building a module that needs to work across multiple versions of python. It's not reasonable to uninstall/reinstall modules every time you want to switch environments.
2. If you want to execute code on the cloud, you can set up a python environment that mirrors the relevant cloud instance. For example, one of Amazon's main EC2 deep learning instances runs Python 3.4, and you may hit obstacles if you use code developed on your machine with Python 3.6.
3. You might have some working python code and want to make sure everything stays frozen so that it'll still work in the future. Without virtual environments, upgrading python modules could unintentionally break that year-old project. Having to go back and determine the correct version for each dependency would be a huge pain.

If some of your required libraries don't work with Anaconda, pyenv and virtualenv (along with virtualenvwrapper) are a good approach to gain lower-level control without too much technical complication. Note, however, that there are a lot more ways to make your workflow fancier and more efficient. This guide covers what pyenv, virtualenv, and virtualenvwrapper are, how they're installed/used, and how they fit together.

1. pyenv: Short for "python environment." This manages which version of python is visible to your computer (and temporarily hides any others). With pyenv, you can install multiple versions of python and quickly switch between the "activated" version (i.e., the version your computer will use to execute code). Here, we'll switch the version of python we're using with virtualenvwrapper's workon command (as described later).

   Installation/use: brew install pyenv on Mac. See the docs for installation via git clone on other other systems. Then you can install new python versions, list installed versions, and set the python version like:

   pyenv install 3.6.3  # Install python version
   pyenv install 3.4.0
   pyenv versions       # List python versions
   pyenv global 3.6.3   # Set your system's python version
   

   Technical details: Every time you execute a python script or use pip, pyenv works in the background to intercept that command and send it to the python environment that is currently activated. It does this using shims on the PATH environment variable, which allow python-related commands to be dynamically rerouted.

2. virtualenv: Short for "virtual environment." This manages separate directories for the modules you install (e.g., with pip). That way, each virtual environment can have it's own set of installed modules that is walled off from every other virtual environment so they don't interfere with each other. Like with pyenv, we'll switch virtual environments with virtualenvwrapper's workon command (as described later).

   Installation: pip install virtualenv

   Technical details: virtualenv keeps each environment (and its installed modules) in separate folders; therefore, each is like a silo that doesn't interact with any other virtual environment. Usually, the exact file location is defined by the user, but we can use virtualenvwrapper to instead handle this for us.

3. virtualenvwrapper[https://virtualenvwrapper.readthedocs.io/en/latest/]. This helps pyenv and virtualenv gel like PB&J. With it, you can effectively switch between a single environment that has both the python version and virtual environment wrapped in one bundle. Make sure pyenv and virtualenv are installed before you install this wrapper.

   Installation: pip install virtualenvwrapper and then follow some one-time installation instructions

   Use First, we'll set the python environment and then make a couple virtual environments. Then you can use the workon command to switch between them.

   pyenv global 3.6.3           # Set your system's python version with pyenv
   mkvirtualenv my_project_py3  # Create a new virtual environment using virtualenvwrapper; it'll be tied to Python 3.6.3
   pip install numpy scipy      # Install the packages you want in this environment
   
   pyenv global 2.7.0       # Set your system's python version with pyenv
   mkvirtualenv webdev_py2  # Create/switch to a new virtual environment; it'll use Python 2.7.0
   pip install flask boto
   
   workon                 # List the environments available
   workon my_project_py3  # Use virtualenvwrapper to switch back to my_project_py3
   

This is my own personal preference, but when setting up my python environment, I also tend to store modules I'm developing in a Builds directory (i.e., /Users/wronk/Builds). Similarly, I put data in /Users/wronk/Data. Then, I'll define an environment variable (e.g., BUILDS_DIR and DATA_DIR) so that writing scripts/python code is more agnostic of the exact machine I'm using.

For example, any shell scripts can traverse directories from the BUILDS_DIR environment variable, and I'll use something like my_data_dir = os.environ['DATA_DIR'] in python code. That tends to be easier for getting the same code running locally and on the cloud (or another computer).