Skip to content

Instantly share code, notes, and snippets.

@jtzero

jtzero/Jupyter command find_spark_home.py not found

Last active February 11, 2025 16:50
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save jtzero/a0114a640524bceeee4a2c3e554a259f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save jtzero/a0114a640524bceeee4a2c3e554a259f to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. jtzero revised this gist May 28, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Jupyter command find_spark_home.py not found
    View file
    Original file line number Diff line number Diff line change
    @@ -18,4 +18,4 @@ workaround:
    set SPARK_HOME
    `SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark`

    finding spark home: `pip show pyspark | grep 'Location: ' | sed 's/Location: //g'`
    finding spark home: `"$(pip show pyspark | grep 'Location: ' | sed 's/Location: //g')/pyspark"`
  2. jtzero revised this gist May 27, 2020. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion Jupyter command find_spark_home.py not found
    View file
    Original file line number Diff line number Diff line change
    @@ -16,4 +16,6 @@ REASON: https://github.com/apache/spark/pull/28256

    workaround:
    set SPARK_HOME
    `SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark`
    `SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark`

    finding spark home: `pip show pyspark | grep 'Location: ' | sed 's/Location: //g'`
  3. jtzero revised this gist May 27, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Jupyter command find_spark_home.py not found
    View file
    Original file line number Diff line number Diff line change
    @@ -12,7 +12,7 @@ Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBk
    /home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
    ```

    REASON: (https://github.com/apache/spark/pull/28256)[https://github.com/apache/spark/pull/28256]
    REASON: https://github.com/apache/spark/pull/28256

    workaround:
    set SPARK_HOME
  4. jtzero revised this gist May 27, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Jupyter command find_spark_home.py not found
    View file
    Original file line number Diff line number Diff line change
    @@ -12,7 +12,7 @@ Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBk
    /home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
    ```

    https://github.com/apache/spark/pull/28256
    REASON: (https://github.com/apache/spark/pull/28256)[https://github.com/apache/spark/pull/28256]

    workaround:
    set SPARK_HOME
  5. jtzero revised this gist May 27, 2020. No changes.
  6. jtzero renamed this gist May 27, 2020. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → Jupyter command find_spark_home.py not found
    View file
    File renamed without changes.
  7. jtzero created this gist May 27, 2020.
    19 changes: 19 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,19 @@
    ```
    (email_ml) jtzero:~/Documents/ml$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark
    Traceback (most recent call last):
    File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/jupyter", line 8, in <module>
    sys.exit(main())
    File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 247, in main
    command = _jupyter_abspath(subcommand)
    File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath
    'Jupyter command `{}` not found.'.format(jupyter_subcommand)
    Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/find_spark_home.py` not found.
    /home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory
    /home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
    ```

    https://github.com/apache/spark/pull/28256

    workaround:
    set SPARK_HOME
    `SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark`