Re: Spark job failing on jackson dependencies

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Re: Spark job failing on jackson dependencies

Fawze Abujaber
Hello All,

I have spark job that is failing after the upgrade of the cdh from 5.5.4 which had spark 1.5.0 to cdh 5.13.0 which has spark 1.6.0

The job is running with the new spark dependancies but i see strange behaviour for one spark job that:

1) sometimes it's oozie launcher marked as success and other as killed,

You can see below snapshots and the log for the 2 oozies attached.

2) also for the spark job itself i see that is failing on the jackson databind.

 2018-01-05 19:07:17,672 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.VerifyError: Bad type on operand stack
Exception Details:
    org/apache/spark/metrics/sink/MetricsServlet.<init>(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V @116: invokevirtual
    Type 'com/codahale/metrics/json/MetricsModule' (current frame, stack[2]) is not assignable to 'com/fasterxml/jackson/databind/Module'

attached the log for the 2 jobs one which is success the other failed.

Please help me

Inline image 1

Inline image 2

Inline image 3