I am trying to deploy a standalone cluster but running into
I have tried a whole myriad of different approaches varying from packaging all dependencies into a single JAR and using the
I’ve got a master node started, a slave node running on the same system, and am using spark submit to get the streaming job kicked off.
Here is the error I’m getting:
Here is the spark submit command I’m using:
I’ve tried all sorts of combinations of including different packages and driver-class-path jar files. As far as I can find, the serializer should be in the
Pom Dependencies are as follows:
If I remove
Thanks Everyone -
it seems that you're missing the kafka-clients jar (and probably some other dependencies as well)
how did you packaged you application jar? does it includes all the required dependencies (as an uber jar)?
if it's not an uber jar you need to pass via the driver-class-path and the executor-class-path all the files\dirs where your dependencies can be found (note that those must be accessible from each node in the cluster)
On Wed, Dec 27, 2017 at 1:08 AM, Geoff Von Allmen <[hidden email]> wrote:
I’ve tried it both ways.
Uber jar gives me gives me the following:
If I only do minimal packaging and add
I agree it seems it’s missing the
I can see the following two packages in the ClassPath entries on the history server (Though the source shows: ********(redacted) — not sure why?)
As as side note, i’m running both a master and worker on the same system just to test out running in cluster mode. Not sure if that would have anything to do with it. I would think it would make it easier since it's got access to all the same file system... but I'm pretty new to Spark.
I have also read through and followed those instructions as well as many others at this point.
On Wed, Dec 27, 2017 at 12:56 AM, Eyal Zituny <[hidden email]> wrote:
The cluster mode doesn't upload jars to the driver node. This is a known issue: https://issues.apache.org/jira/browse/SPARK-4160
On Wed, Dec 27, 2017 at 1:27 AM, Geoff Von Allmen <[hidden email]> wrote:
|Free forum by Nabble||Edit this page|