Spark Python in Jupyter Notebook

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark Python in Jupyter Notebook

tmmet
This post has NOT been accepted by the mailing list yet.
Hi,
I've Spark in my Hadoop MAPR environment. Could anyone please clarify on the below?

1. Could anyone provide instructions to connect to Spark in Hadoop MAPR from Jupyter Notebook in my local machine?
2. I have installed Anaconda in my Hadoop MAPR. How can I connect to Spark in Hadoop MAPR from Jupyter Notebook in Hadoop MAPR?
3. Out of the two options, which one is highly recommended and why?

Any detailed instructions/ steps would be greatly helpful.
Thank You!

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Python in Jupyter Notebook

neil90
This post has NOT been accepted by the mailing list yet.
Assuming you don't have your environment variables setup in your .bash_profile you would do it like this -

import os
import sys

spark_home = '/usr/local/spark'
sys.path.insert(0, spark_home + "/python")
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.1-src.zip'))
#os.environ['PYSPARK_SUBMIT_ARGS'] = """--master spark://54.68.147.137:7077 pyspark-shell""" <---- where you can pass commands you would pass in launching pyspark directly from command line

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

conf = SparkConf()\
    .setMaster("local[8]")\
    .setAppName("Test")  

sc = SparkContext(conf=conf)

spark = SparkSession.builder\
    .config(conf=sc.getConf())\
    .enableHiveSupport()\
    .getOrCreate()

Mind you this is for spark 2.0 and above
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Python in Jupyter Notebook

tmmet
This post has NOT been accepted by the mailing list yet.
Thanks Much!
I'm using Spark 1.6.x. Will the same code work for Spark 1.6.x?

Thanks!
Loading...