pyspark configuration with Juyter

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

pyspark configuration with Juyter

anudeep
Hello experts,

I install jupyter notebook thorugh anacoda, set the pyspark driver to use jupyter notebook.

I see the below issue when i try to open pyspark.

anudeepg@datanode2 spark-2.1.0]$ ./bin/pyspark
[I 07:29:53.184 NotebookApp] The port 8888 is already in use, trying another port.
[I 07:29:53.211 NotebookApp] JupyterLab alpha preview extension loaded from /home/anudeepg/anaconda2/lib/python2.7/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 07:29:53.212 NotebookApp] Running the core application with no additional extensions or settings
[I 07:29:53.214 NotebookApp] Serving notebooks from local directory: /opt/mapr/spark/spark-2.1.0
[I 07:29:53.214 NotebookApp] 0 active kernels
[I 07:29:53.214 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e9c70a7f2e8c9a7cf2e
[I 07:29:53.214 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 07:29:53.214 NotebookApp] No web browser found: could not locate runnable browser.
[C 07:29:53.214 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:


Can someone please help me here.

Thanks!
Anudeep

Reply | Threaded
Open this post in threaded view
|

Re: pyspark configuration with Juyter

zjffdu

You are setting PYSPARK_DRIVER to jupyter, please set it to python exec file


anudeep <[hidden email]>于2017年11月3日周五 下午7:31写道:
Hello experts,

I install jupyter notebook thorugh anacoda, set the pyspark driver to use jupyter notebook.

I see the below issue when i try to open pyspark.

anudeepg@datanode2 spark-2.1.0]$ ./bin/pyspark
[I 07:29:53.184 NotebookApp] The port 8888 is already in use, trying another port.
[I 07:29:53.211 NotebookApp] JupyterLab alpha preview extension loaded from /home/anudeepg/anaconda2/lib/python2.7/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 07:29:53.212 NotebookApp] Running the core application with no additional extensions or settings
[I 07:29:53.214 NotebookApp] Serving notebooks from local directory: /opt/mapr/spark/spark-2.1.0
[I 07:29:53.214 NotebookApp] 0 active kernels
[I 07:29:53.214 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e9c70a7f2e8c9a7cf2e
[I 07:29:53.214 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 07:29:53.214 NotebookApp] No web browser found: could not locate runnable browser.
[C 07:29:53.214 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:


Can someone please help me here.

Thanks!
Anudeep

Reply | Threaded
Open this post in threaded view
|

Re: pyspark configuration with Juyter

tokoma
I setup environment variables in my ~/.bashrc as follows:

export PYSPARK_PYTHON=/usr/local/oss/anaconda3/bin/python3.6
export PYTHONPATH=$(ls -a ${SPARK_HOME}/python/lib/py4j-*-src.zip):${SPARK_HOME}/python:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'


2017-11-03 20:56 GMT+09:00 Jeff Zhang <[hidden email]>:

You are setting PYSPARK_DRIVER to jupyter, please set it to python exec file


anudeep <[hidden email]>于2017年11月3日周五 下午7:31写道:
Hello experts,

I install jupyter notebook thorugh anacoda, set the pyspark driver to use jupyter notebook.

I see the below issue when i try to open pyspark.

anudeepg@datanode2 spark-2.1.0]$ ./bin/pyspark
[I 07:29:53.184 NotebookApp] The port 8888 is already in use, trying another port.
[I 07:29:53.211 NotebookApp] JupyterLab alpha preview extension loaded from /home/anudeepg/anaconda2/lib/python2.7/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 07:29:53.212 NotebookApp] Running the core application with no additional extensions or settings
[I 07:29:53.214 NotebookApp] Serving notebooks from local directory: /opt/mapr/spark/spark-2.1.0
[I 07:29:53.214 NotebookApp] 0 active kernels
[I 07:29:53.214 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e9c70a7f2e8c9a7cf2e
[I 07:29:53.214 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 07:29:53.214 NotebookApp] No web browser found: could not locate runnable browser.
[C 07:29:53.214 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:


Can someone please help me here.

Thanks!
Anudeep


Reply | Threaded
Open this post in threaded view
|

Re: pyspark configuration with Juyter

Marco Mistroni
Hi probably not what u r looking for but if u get stuck with conda jupyther and spark, if u get an account @ community.cloudera you will enjoy jupyther and spark out of the box
Gd luck and hth
Kr

On Nov 4, 2017 4:59 PM, "makoto" <[hidden email]> wrote:
I setup environment variables in my ~/.bashrc as follows:

export PYSPARK_PYTHON=/usr/local/oss/anaconda3/bin/python3.6
export PYTHONPATH=$(ls -a ${SPARK_HOME}/python/lib/py4j-*-src.zip):${SPARK_HOME}/python:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'


2017-11-03 20:56 GMT+09:00 Jeff Zhang <[hidden email]>:

You are setting PYSPARK_DRIVER to jupyter, please set it to python exec file


anudeep <[hidden email]>于2017年11月3日周五 下午7:31写道:
Hello experts,

I install jupyter notebook thorugh anacoda, set the pyspark driver to use jupyter notebook.

I see the below issue when i try to open pyspark.

anudeepg@datanode2 spark-2.1.0]$ ./bin/pyspark
[I 07:29:53.184 NotebookApp] The port 8888 is already in use, trying another port.
[I 07:29:53.211 NotebookApp] JupyterLab alpha preview extension loaded from /home/anudeepg/anaconda2/lib/python2.7/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 07:29:53.212 NotebookApp] Running the core application with no additional extensions or settings
[I 07:29:53.214 NotebookApp] Serving notebooks from local directory: /opt/mapr/spark/spark-2.1.0
[I 07:29:53.214 NotebookApp] 0 active kernels
[I 07:29:53.214 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e9c70a7f2e8c9a7cf2e
[I 07:29:53.214 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 07:29:53.214 NotebookApp] No web browser found: could not locate runnable browser.
[C 07:29:53.214 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:


Can someone please help me here.

Thanks!
Anudeep