-
Star
(188)
You must be signed in to star a gist -
Fork
(69)
You must be signed in to fork a gist
-
-
Save erikbern/78ba519b97b440e10640 to your computer and use it in GitHub Desktop.
| # This is mostly based on this excellent blog post: | |
| # http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/ | |
| # Install various packages | |
| sudo apt-get update | |
| sudo apt-get upgrade -y # choose “install package maintainers version” | |
| sudo apt-get install -y build-essential python-pip python-dev git python-numpy swig python-dev default-java-sdk zip zlib1g-dev | |
| # Blacklist Noveau which has some kind of conflict with the nvidia driver | |
| echo -e "blacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf | |
| echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf | |
| sudo update-initramfs -u | |
| sudo reboot # Reboot (annoying you have to do this in 2015!) | |
| # Some other annoying thing we have to do | |
| sudo apt-get install -y linux-image-extra-virtual | |
| sudo reboot # Not sure why this is needed | |
| # Install latest Linux headers | |
| sudo apt-get install -y linux-source linux-headers-`uname -r` | |
| # Install CUDA 7.0 | |
| wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/cuda_7.0.28_linux.run | |
| chmod +x cuda_7.0.28_linux.run | |
| ./cuda_7.0.28_linux.run -extract=`pwd`/nvidia_installers | |
| cd nvidia_installers | |
| sudo ./NVIDIA-Linux-x86_64-346.46.run | |
| sudo modprobe nvidia | |
| sudo ./cuda-linux64-rel-7.0.28-19326674.run | |
| cd | |
| # Install tensorflow | |
| sudo pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl | |
| export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64" | |
| export CUDA_HOME=/usr/local/cuda | |
| # Install Bazel | |
| git clone https://github.com/bazelbuild/bazel.git | |
| cd bazel | |
| git checkout tags/0.1.0 | |
| ./compile.sh | |
| sudo cp output/bazel /usr/bin | |
| cd | |
| # Install TensorFlow | |
| git clone https://github.com/tensorflow/tensorflow | |
| cd tensorflow/tensorflow/models/image/mnist | |
| python convolutional.py | |
| # At this point, it breaks down. It works, but doesn't use the GPU. On g2.2xlarge: | |
| # I tensorflow/core/common_runtime/gpu/gpu_device.cc:611] Ignoring gpu device (device: 0, name: GRID K520, pci bus id: 0000:00:03.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5. | |
| # On g2.8xlarge: | |
| # I tensorflow/core/common_runtime/gpu/gpu_device.cc:611] Ignoring gpu device (device: 0, name: GRID K520, pci bus id: 0000:00:03.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5. | |
| # I tensorflow/core/common_runtime/gpu/gpu_device.cc:611] Ignoring gpu device (device: 1, name: GRID K520, pci bus id: 0000:00:04.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5. | |
| # I tensorflow/core/common_runtime/gpu/gpu_device.cc:611] Ignoring gpu device (device: 2, name: GRID K520, pci bus id: 0000:00:05.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5. | |
| # I tensorflow/core/common_runtime/gpu/gpu_device.cc:611] Ignoring gpu device (device: 3, name: GRID K520, pci bus id: 0000:00:06.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5. | |
| # TODO: Seems like there's a discussion on GitHub about CUDA 3.0 support | |
| # https://github.com/tensorflow/tensorflow/issues/25 |
Yes, @hammer is right, python-dev is needed or the new python package will fail with a could not find <Python.h> error.
Hi,
Thanks for the great work.
I am trying to compare CPU / GPU and different hw and I am getting this:
Macbook Pro i5 2,6Ghz (cifar10_train.py)
2015-12-24 23:33:13.533470: step 50, loss = 4.58 (173.1 examples/sec; 0.739 sec/batch)
AWS g2.2xlarge GPU $2/hour - cifar10_multi_gpu_train.py
(Creating TensorFlow device (/gpu:0) -> (device: 0, name: GRID K520...)
2015-12-24 22:15:16.011190: step 50, loss = 4.59 (322.1 examples/sec; 0.397 sec/batch)
AWS g2.2xlarge NO GPU $2/hour - cifar10_multi_gpu_train.py
(Ignoring gpu device (device: 0, name: GRID K520... )
2015-12-24 22:00:05.110064: step 50, loss = 4.59 (254.6 examples/sec; 0.503 sec/batch)
Is it normal that my Mac is less than 2x slower than a g2.2xlarge that uses a GPU? I has expecting 10x...
These were very useful Erik - I finally got around to using tensor flow today
I'm seeing the same performance numbers as @marcotrombetti on g2.2xlarge instances, both on GPU and on CPU. This seems to be many times slower than Theano on the same hardware when running on GPU. Is this expected, or is this indicative of some misconfiguration on my side?
To correct Line 88 above, it CAN use all four cores
Performance Attributes
All measured at step = 50
| Instance Type | Num GPUs | Examples / Sec | Sec / Batch |
|---|---|---|---|
| g2.2xlarge | 1 | 216.0 | 0.593 |
| g2.2xlarge | 1 | 225.2 | 0.568 |
| g2.8xlarge | 4 | 675.2 | 0.190 |
Notes
To run with 4 cores call example with --num_gpus = 4
Very important you set Compute Capability = 3.0 (thanks @wfbradley)
If you pulled the latest tensorflow version 0.6 you need to change Line 80
sudo pip install /tmp/tensorflow_pkg/tensorflow-0.6.0-cp27-none-linux_x86_64.whl
However for me the led to segfaults in the example due to an issue with the Eigen Kernel. This has temporarily been resolved. Please see:
tensorflow/tensorflow#726
tensorflow/tensorflow#713
Why does Erik install tensorflow the way he does? Why not use pip?
Thanks @closedLoop for all the information, I get the same numbers. Still strange that @erikbern and @marcotrombetti report much higher speed. (32_x_ examples / sec instead of 22_x_ examples / sec)
To build for Python 3.4:
- During configure specify /usr/bin/python3
- Use bazel 0.1.1,
git checkout tags/0.1.1 - Use wheel 0.26
- Don't use the current tensorflow master. It is not Python 3 compatible. Checkout the 0.6.0 tag.
- Modify tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package. Add
--python-tag py34when building the wheel:
...
echo $(date) : "=== Building wheel"
python setup.py bdist_wheel --python-tag py34 >/dev/null
...
- Make sure to use the correct wheel name when installing:
sudo pip3 install /tmp/tensorflow_pkg/tensorflow-0.6.0-py34-none-any.whl
Just for reference. Geforce 970, i7 Local Machine, examples/sec 903.3 examples/sec .142 sec/batch
Just got to run. Thank you for the code. My timing - measured at step 50
| Instance Type | s/batch |
|---|---|
| g2.2xlarge | 0.26 |
| g2.8xlarge | 0.11 (with --num_gpus=4) |
| macbook-pro i7 quad end 2013 | 0.56 |
Quite unimpressive the usage of gpus, as @marcotrombetti says. I was also expecting an order of magnitude improvement.
I have done all the steps but the last pip install seems to be an issue
"Requirement '/tmp/tensorflow_pkg/tensorflow-0.5.0-cp27-none-linux_x86_64.whl' looks like a filename, but the file does not exist"
I used git "checkout tags/0.1.4" for basel instead.
*Edit ok silly me: Line 80 needs to be changed to 0.60 instead.
You can download CUDNN 6.5: http://developer.download.nvidia.com/compute/redist/cudnn/v2/cudnn-6.5-linux-x64-v2.tgz
Using keras mnist_cnn script to compare the performance of Theano to Tensorflow on g2.2xlarges, I see 8s / epoch with Theano and 97s / epoch with Tensorflow!
In case anyone's interested, we documented how we installed TensorFlow along with Python 3.4 and Jupyter on EC2 based on this gist and many of the comments here. Thank you everyone!
Didn't notice all the comments here – Github doesn't send notifications on gists I guess. Anyway you should check out @chrisconley's link instead!
Nice work! Thnx for guide ^^
Made another one with Python3, TensorFlow 0.7 and OpenCV 3.1:
https://gist.github.com/AlexJoz/1670baf0b32573ca7923
Public ami with my setup in N.Virginia: ami-9d0f3ff7
If you for some reason found / to fill up unexplainably when compiling with bazel, this is due to bazel putting cache files in ~/.cache/bazel by default. Set export TEST_TMPDIR=/tmp/.cache to avoid this.
There seems to be a new solution: https://aws.amazon.com/marketplace/pp/B01AOE205O
@AlexJoz's ami works great
Published new AMI in N. Virginia with 0.8.0 support: ami-1e19ee73
Thanks. I will look into my changing my bash file that installs the cpu version of tensorflow (with video) to a gpu version on cloud9 http://c9.io
When I run line 73, I get an error: Unrecognized option: --host_force_python=py2
Any idea why?
@shamak I'm getting the exact same error.
It seems that Cudnn can be downloaded with
curl -fvSL http://developer.download.nvidia.com/compute/redist/cudnn/v2/cudnn-6.5-linux-x64-v2.tgz -o cudnn-6.5-linux-x64-v2.tgz
Just for reference:
GTX 1070, i7 6700k, local machine, tensorflow inside docker container, using nvidia-docker (but I doubt it adds any overhead)
1744.3 examples/sec; 0.073 sec/batch
I've recently prepared a couple of convenience scripts for firing up your AWS instance with Jupyter Notebook on board that you may find useful:
For stat:
Zotac GTX 1080 AMP Extreme, 2560 CUDA cores, 1771 MHz core clock, 10000 MHz mem clock. i7 930 3.8 GHz boost clock.
step 100000, loss = 0.72 (1780.0 examples/sec; 0.072 sec/batch); time: 2h 5m.
Do you need
python-devtwice on line 12?