[SPARK: org.apache.spark.util.TaskCompletionListenerException]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[SPARK: org.apache.spark.util.TaskCompletionListenerException]

Romeo Valencia

Hi,

I wonder if someone could help me in finding the solution to a rather vague exception that we are getting.   I am attaching the STDOUT & STDERR files when we execute spark-submit.   The exception message that we are getting is per below excerpt.

 

“org.apache.spark.util.TaskCompletionListenerException: org.codehaus.jackson.JsonGenerationException: Incomplete surrogate pair: first char 0xdf46, second 0x5b”

 

This normally happens and according to stack trace is from the code (excerpt).

..

..

GraphToTableLogger.warn("running collect on component")
val distinctComps = ss.sql("SELECT CAST(componentID AS VARCHAR) componentID FROM components_DF GROUP BY componentID")
// .repartition(repartition_size)
 
.collect()

..

..

 

 

What makes it interesting is that the same dataset when re-invoking the spark-submit again will complete.   

Appreciate the help in advance.

______________________________________________________________________

Thanks and best regards,

 

Romeo Valencia
Senior Data Engineer, IP MM-Product Management-USA |  Clarivate Analytics

Phone +1 415 278 8463  | markmonitor.comclarivate.com  

50 California St.  |  San Francisco, CA  94111  |  US

cid:image003.png@01D2B3B7.8C61C350

 



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

stdout (101K) Download Attachment
stderr (105K) Download Attachment