Skip to content

Instantly share code, notes, and snippets.

@alexland

alexland/spark-custom-install.sh

Last active August 29, 2015 14:20
Show Gist options
  • Save alexland/cb2339d19e4515e813fe to your computer and use it in GitHub Desktop.
Save alexland/cb2339d19e4515e813fe to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. alexland revised this gist Apr 29, 2015. 1 changed file with 2 additions and 3 deletions.
    5 changes: 2 additions & 3 deletions spark-custom-install.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,3 @@

    #---------------- custom install Apache Spark -------------------#

    # instructions to build Apache Spark from source w/ current Scala & using sbt (vs maven)
    @@ -13,11 +12,11 @@ tar zxf spark-1.3.1.tgz
    # start sbt REPL
    sbt

    # set config to build w/ scala 2.11
    # set config to build w/ scala 2.11.6
    -Dscala-2.11=true

    # build the exxecutable jar file w/ all dependencies:
    ./sbt/sbt assembly
    assembly

    #---------------- build the documentation (html) -------------------#

  2. alexland created this gist Apr 29, 2015.
    39 changes: 39 additions & 0 deletions spark-custom-install.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,39 @@

    #---------------- custom install Apache Spark -------------------#

    # instructions to build Apache Spark from source w/ current Scala & using sbt (vs maven)


    # download Apache Spark src from the appropriate mirror


    # untar:
    tar zxf spark-1.3.1.tgz

    # start sbt REPL
    sbt

    # set config to build w/ scala 2.11
    -Dscala-2.11=true

    # build the exxecutable jar file w/ all dependencies:
    ./sbt/sbt assembly

    #---------------- build the documentation (html) -------------------#

    # install these dependencies:
    sudp pip3 install -U pygments --upgrade
    sudo pip3 install -U sphinx --upgrade

    # install ruby if not already (aside from your system ruby, which, you should leave alone)
    brew install ruby

    # and these ruby gems:
    sudo gem install jekyll
    sudo gem install jekyll-redirect-from


    # cd into /docs from top level spark dir
    cd /docs

    PRODUCTION=1 jekyll build