[SPARK: org.apache.spark.util.TaskCompletionListenerException]

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[SPARK: org.apache.spark.util.TaskCompletionListenerException]

Romeo Valencia


I wonder if someone could help me in finding the solution to a rather vague exception that we are getting.   I am attaching the STDOUT & STDERR files when we execute spark-submit.   The exception message that we are getting is per below excerpt.


“org.apache.spark.util.TaskCompletionListenerException: org.codehaus.jackson.JsonGenerationException: Incomplete surrogate pair: first char 0xdf46, second 0x5b”


This normally happens and according to stack trace is from the code (excerpt).



GraphToTableLogger.warn("running collect on component")
val distinctComps = ss.sql("SELECT CAST(componentID AS VARCHAR) componentID FROM components_DF GROUP BY componentID")
// .repartition(repartition_size)





What makes it interesting is that the same dataset when re-invoking the spark-submit again will complete.   

Appreciate the help in advance.


Thanks and best regards,


Romeo Valencia
Senior Data Engineer, IP MM-Product Management-USA |  Clarivate Analytics

Phone +1 415 278 8463  | markmonitor.comclarivate.com  

50 California St.  |  San Francisco, CA  94111  |  US



To unsubscribe e-mail: [hidden email]

stdout (101K) Download Attachment
stderr (105K) Download Attachment