Prepare spark executor

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Prepare spark executor


Is there any way to prepare spark executor? Like what we do in MapReduce, we implements a setup and a clearup method.

For my case, I need this prepare method to init StaticParser base on the env(dev, production). Then, I can directly use this StaticParser on executor. like this

    object StaticParser {


Now I can find to ways:
    1. Prepare the executor before any action are taken. The drawback is if we add new executors, following job will fail.
            sc.parallelize(Seq(1 to 10000), 10000).foreach(setupExecutor)

    2. Change StaticParser to a class, and make an instance then ship it from driver to executors.
            val shippedParser = new StaticParser().init(dev)