Prepare spark executor

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Prepare spark executor

yaoxin
Hi

Is there any way to prepare spark executor? Like what we do in MapReduce, we implements a setup and a clearup method.

For my case, I need this prepare method to init StaticParser base on the env(dev, production). Then, I can directly use this StaticParser on executor. like this

    object StaticParser {
        init(env)
        parse()
    }

    sc.setPrepareExecutorMethod(StaticParser.init(dev))

    rdd.map(StaticParser.parse)

Now I can find to ways:
    1. Prepare the executor before any action are taken. The drawback is if we add new executors, following job will fail.
            sc.parallelize(Seq(1 to 10000), 10000).foreach(setupExecutor)

    2. Change StaticParser to a class, and make an instance then ship it from driver to executors.
            val shippedParser = new StaticParser().init(dev)
            rdd.map(shippedParser.parse)