Windows10 + pyspark + ipython + csv file loading with timestamps

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Windows10 + pyspark + ipython + csv file loading with timestamps

Esa Heikkinen

Hi

Does anyone have any hints or example (code) how to get combination: Windows10 + pyspark + ipython notebook + csv file loading with timestamps (timeseries data) to dataframe or RDD to work ?

I have already installed windows10 + pyspark + ipython notebook and they seem to work, but my python code in notebook does not, because "spark context" may not work ?

What commands should be put into the beginning of the notebook ? sc = SparkContext.getOrCreate() ? spark = SparkSession(sc) ?

I have installed: spark-2.2.1-bin-hadoop2.7  and ipython 6.1.0 to Windows10.

----

Eras

Reply | Threaded
Open this post in threaded view
|

Re: Windows10 + pyspark + ipython + csv file loading with timestamps

Szuromi Tamás

2017-12-16 11:04 GMT+01:00 Esa Heikkinen <[hidden email]>:

Hi

Does anyone have any hints or example (code) how to get combination: Windows10 + pyspark + ipython notebook + csv file loading with timestamps (timeseries data) to dataframe or RDD to work ?

I have already installed windows10 + pyspark + ipython notebook and they seem to work, but my python code in notebook does not, because "spark context" may not work ?

What commands should be put into the beginning of the notebook ? sc = SparkContext.getOrCreate() ? spark = SparkSession(sc) ?

I have installed: spark-2.2.1-bin-hadoop2.7  and ipython 6.1.0 to Windows10.

----

Eras