PyCharm IDE throws spark error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

PyCharm IDE throws spark error

Mich Talebzadeh
Hi,

This is basically a simple module 

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql.types import StringType, ArrayType
from pyspark.sql.functions import udf, col
import random
import string
import math
spark = SparkSession.builder.appName("sparkp"
).enableHiveSupport().getOrCreate()

and comes back with the following error


Traceback (most recent call last):

  File "C:/Users/whg220/PycharmProjects/sparkp/venv/Scripts/sparkp.py", line 43, in <module>

    spark = SparkSession.builder.appName("sparkp").enableHiveSupport().getOrCreate()

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\sql\session.py", line 173, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 363, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 129, in __init__

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 312, in _ensure_initialized

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 46, in launch_gateway

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 101, in _launch_gateway

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 707, in __init__

    restore_signals, start_new_session)

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 990, in _execute_child

    startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified


very frustrating. Any help will be appreciated.


Thanks



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 

Reply | Threaded
Open this post in threaded view
|

Re: PyCharm IDE throws spark error

Wim Van Leuven
No Java installed? Or process can but find it? Java-home not set?

On Fri, 13 Nov 2020 at 23:24, Mich Talebzadeh <[hidden email]> wrote:
Hi,

This is basically a simple module 

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql.types import StringType, ArrayType
from pyspark.sql.functions import udf, col
import random
import string
import math
spark = SparkSession.builder.appName("sparkp"
).enableHiveSupport().getOrCreate()

and comes back with the following error


Traceback (most recent call last):

  File "C:/Users/whg220/PycharmProjects/sparkp/venv/Scripts/sparkp.py", line 43, in <module>

    spark = SparkSession.builder.appName("sparkp").enableHiveSupport().getOrCreate()

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\sql\session.py", line 173, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 363, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 129, in __init__

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 312, in _ensure_initialized

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 46, in launch_gateway

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 101, in _launch_gateway

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 707, in __init__

    restore_signals, start_new_session)

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 990, in _execute_child

    startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified


very frustrating. Any help will be appreciated.


Thanks



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 

Reply | Threaded
Open this post in threaded view
|

Re: PyCharm IDE throws spark error

Mich Talebzadeh
Many thanks Wim. It turned up to be a spelling error in the code!




LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Fri, 13 Nov 2020 at 23:25, Wim Van Leuven <[hidden email]> wrote:
No Java installed? Or process can but find it? Java-home not set?

On Fri, 13 Nov 2020 at 23:24, Mich Talebzadeh <[hidden email]> wrote:
Hi,

This is basically a simple module 

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql.types import StringType, ArrayType
from pyspark.sql.functions import udf, col
import random
import string
import math
spark = SparkSession.builder.appName("sparkp"
).enableHiveSupport().getOrCreate()

and comes back with the following error


Traceback (most recent call last):

  File "C:/Users/whg220/PycharmProjects/sparkp/venv/Scripts/sparkp.py", line 43, in <module>

    spark = SparkSession.builder.appName("sparkp").enableHiveSupport().getOrCreate()

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\sql\session.py", line 173, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 363, in getOrCreate

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 129, in __init__

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 312, in _ensure_initialized

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 46, in launch_gateway

  File "C:\Users\whg220\spark\spark-2.3.4-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 101, in _launch_gateway

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 707, in __init__

    restore_signals, start_new_session)

  File "C:\Program Files\Anaconda3\lib\subprocess.py", line 990, in _execute_child

    startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified


very frustrating. Any help will be appreciated.


Thanks



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.