RV: Unintelligible warning arose out of the blue.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

RV: Unintelligible warning arose out of the blue.

Tomas Zubiri





De: Tomas Zubiri
Enviado: viernes, 04 de mayo de 2018 04:23 p.m.
Para: [hidden email]
Asunto: Unintelligible warning arose out of the blue.
 

My setup is as follows:
Windows 10
Python 3.6.5
Spark 2.3.0
The latest java jdk
winutils/hadoop installed from this github page https://github.com/steveloughran/winutils

I initialize spark from the pyspark shell as follows:
df = spark.read.csv('mvce.csv')


the mvce is a small file with 3 lines and 2 columns.


The warning I receive is:

2018-05-04 16:17:44 WARN  ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException     

What does this mean? I think it has something to do with YARN, but being an internal technology I have no clue about it. It doesn't seem to be causing any trouble, but I don't want to add the uncertainty that this might be causing an bug in future diagnosing of issues.

Thank you for your help!



Reply | Threaded
Open this post in threaded view
|

Re: RV: Unintelligible warning arose out of the blue.

Marco Mistroni
Hi
  i think it has to do with spark configuration,  dont think the standard configuration is geared up to be running in local mode on windows
your dataframe is ok, you can check out that you have read it successfully by printing out df.count() and you will see your code is reading the dataframe successfully

hth
m

On Fri, May 4, 2018 at 9:15 PM, Tomas Zubiri <[hidden email]> wrote:





De: Tomas Zubiri
Enviado: viernes, 04 de mayo de 2018 04:23 p.m.
Para: [hidden email]
Asunto: Unintelligible warning arose out of the blue.
 

My setup is as follows:
Windows 10
Python 3.6.5
Spark 2.3.0
The latest java jdk
winutils/hadoop installed from this github page https://github.com/steveloughran/winutils

I initialize spark from the pyspark shell as follows:
df = spark.read.csv('mvce.csv')


the mvce is a small file with 3 lines and 2 columns.


The warning I receive is:

2018-05-04 16:17:44 WARN  ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException     

What does this mean? I think it has something to do with YARN, but being an internal technology I have no clue about it. It doesn't seem to be causing any trouble, but I don't want to add the uncertainty that this might be causing an bug in future diagnosing of issues.

Thank you for your help!