Skip to content

Instantly share code, notes, and snippets.

@jonaqp

jonaqp/install-py3-dataproc.sh

Forked from cerisier/install-py3-dataproc.sh
Created May 30, 2018 11:43
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save jonaqp/06f6c447b0c7e8c1b8d3f7b480c4885e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save jonaqp/06f6c447b0c7e8c1b8d3f7b480c4885e to your computer and use it in GitHub Desktop.
Download ZIP
Dataproc initialization action script for installing python3
Raw
install-py3-dataproc.sh
#!/bin/bash
# from https://gist.github.com/nehalecky/9258c01fb2077f51545a/raw/789f08141dc681cf1ad5da05455c2cd01d1649e8/install-py3-dataproc.sh
apt-get -y install python3
echo "export PYSPARK_PYTHON=python3" | tee -a /etc/profile.d/spark_config.sh /etc/*bashrc /usr/lib/spark/conf/spark-env.sh
echo "Adding PYTHONHASHSEED=0 to profiles and spark-defaults.conf..."
echo "export PYTHONHASHSEED=0" | tee -a /etc/profile.d/spark_config.sh /etc/*bashrc /usr/lib/spark/conf/spark-env.sh
echo "spark.executorEnv.PYTHONHASHSEED=0" >> /etc/spark/conf/spark-defaults.conf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment