Skip to content

Instantly share code, notes, and snippets.

@kduy

kduy/caffe_ec2.md

Last active November 22, 2017 11:43
Show Gist options
  • Save kduy/46f5a3e70997abb53d174df9535b550c to your computer and use it in GitHub Desktop.
Save kduy/46f5a3e70997abb53d174df9535b550c to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. kduy revised this gist Nov 22, 2017. No changes.
  2. kduy revised this gist Apr 21, 2017. 1 changed file with 14 additions and 0 deletions.
    14 changes: 14 additions & 0 deletions caffe_ec2.md
    View file
    Original file line number Diff line number Diff line change
    @@ -35,6 +35,13 @@ Verify
    nvidia-smi
    ```

    Add to `~/.bashrc`
    ```
    export PATH=$PATH:/usr/local/cuda-7.0/bin
    export LD_LIBRARY_PATH=:/usr/local/cuda-7.0/lib64
    ```


    ## Install cdnn
    Download `cudnn` then
    ```
    @@ -68,6 +75,13 @@ Now we can download Caffe. Navigate to the directory of your choice for the clon
    cd ~
    git clone https://github.com/BVLC/caffe.git
    ```

    Add to `~/.bashrc`
    ```
    export CAFFE_ROOT=/home/username/caffe
    ```


    We now install more dependencies. Warning: This takes 10-30 minutes.
    ```
    cd caffe
  3. kduy revised this gist Apr 21, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions caffe_ec2.md
    View file
    Original file line number Diff line number Diff line change
    @@ -122,3 +122,5 @@ https://iqbalnaved.wordpress.com/2016/05/08/installing-caffe-and-pycaffe-on-ubun
    https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN-3)

    https://github.com/BVLC/caffe/wiki/Caffe-on-EC2-Ubuntu-14.04-Cuda-7

    http://tleyden.github.io/blog/2015/11/22/running-neural-style-on-an-aws-gpu-instance/
  4. kduy created this gist Apr 21, 2017.
    124 changes: 124 additions & 0 deletions caffe_ec2.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,124 @@
    ## Machine
    * Launch `ami-d05e75b8` from [here](https://cloud-images.ubuntu.com/releases/14.04/release-20150325/)
    * Choose a GPU instance type: `g2.2xlarge` or `g2.8xlarge`
    * Increase the size of the storage (this depends on what else you plan to install, I’d suggest at least 20 GB)

    ## Install Cuda 7.0
    ```
    wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/rpmdeb/cuda-repo-ubuntu1404-7-0-local_7.0-28_amd64.deb
    sudo dpkg -i cuda-repo-ubuntu1404-7-0-local_7.0-28_amd64.deb
    ```
    ```
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get install -y opencl-headers build-essential protobuf-compiler libprotoc-dev libboost-all-dev libleveldb-dev hdf5-tools libhdf5-serial-dev libopencv-core-dev libopencv-highgui-dev libsnappy-dev libsnappy1 libatlas-base-dev cmake libstdc++6-4.8-dbg libgoogle-glog0 libgoogle-glog-dev libgflags-dev liblmdb-dev git python-pip gfortran
    sudo apt-get clean
    ```
    You will get a dialog regarding the `menu.lst` file, just choose the default option it gives you.

    For an explanation of why this is needed, see [Caffe on EC2 Ubuntu 14.04 Cuda 7](https://github.com/BVLC/caffe/wiki/Caffe-on-EC2-Ubuntu-14.04-Cuda-7) and search for this command.
    ```
    sudo apt-get install -y linux-image-extra-`uname -r` linux-headers-`uname -r` linux-image-`uname -r`
    ```

    ```
    sudo apt-get install -y cuda
    sudo apt-get clean
    ```

    Verify
    ```
    nvidia-smi
    ```

    ## Install cdnn
    Download `cudnn` then
    ```
    tar -zxf cudnn-7.0-linux-x64-v3.0-prod.tgz
    cd cuda
    sudo cp lib64/* /usr/local/cuda/lib64/
    sudo cp include/cudnn.h /usr/local/cuda/include/
    sudo ldconfig /usr/local/cuda/lib64/
    ```


    ## Installing Caffe

    Install the dependencies:

    ```
    sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev protobuf-compiler gfortran libjpeg62 libfreeimage-dev libatlas-base-dev git python-dev python-pip libgoogle-glog-dev libbz2-dev libxml2-dev libxslt-dev libffi-dev libssl-dev libgflags-dev liblmdb-dev python-yaml python-numpy
    ```
    Then run:
    ```
    sudo easy_install pillow
    ```
    You could have "TypeError: 'NoneType' object is not callable" error when installing pillow, then try:
    ```
    sudo apt-get install pypy-dev
    ```

    Now we can download Caffe. Navigate to the directory of your choice for the cloning.
    ```
    cd ~
    git clone https://github.com/BVLC/caffe.git
    ```
    We now install more dependencies. Warning: This takes 10-30 minutes.
    ```
    cd caffe
    cat python/requirements.txt | xargs -L 1 sudo pip install
    ```
    Now we update the Makefile:
    ```
    cp Makefile.config.example Makefile.config
    vi Makefile.config
    ```
    Uncomment the line: `USE_CUDNN := 1`
    Make sure the CUDA_DIR correctly points to our CUDA installation.
    If you want the Matlab wrapper, uncomment the appropriate `MATLAB_DIR` line.
    If you add a layer in python, uncomment the line `WITH_PYTHON_LAYER=true`

    Now we build Caffe. Set X to the number of CPU threads (or cores) on your machine. Use the command htop to check how many CPU threads you have.
    ```
    make all -jX
    make pycaffe -jX
    make test -jX
    ```

    Add this line to `~/.bashrc` to use python from `caffe`
    ```
    export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH
    ```
    then make `~/.bashrc` valid.
    ```
    source ~/.bashrc
    ```

    Verify
    ```
    python -c 'import caffe'
    ```

    If something goes wrong with `matplotlib`, reinstall it
    ```
    sudo pip uninstall matplotlib
    sudo apt-get install python-matplotlib
    ```




    References

    http://tleyden.github.io/blog/2015/11/22/cuda-7-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/

    https://iqbalnaved.wordpress.com/2016/05/08/installing-caffe-and-pycaffe-on-ubuntu-14-04/

    https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN-3)

    https://github.com/BVLC/caffe/wiki/Caffe-on-EC2-Ubuntu-14.04-Cuda-7