Can I set the Alluxio WriteType in Spark applications?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Can I set the Alluxio WriteType in Spark applications?

Mark Zhao
Hi,

If Spark applications write data into alluxio, can WriteType be configured?

Thanks,
Mark

Reply | Threaded
Open this post in threaded view
|

Re: Can I set the Alluxio WriteType in Spark applications?

Bin Fan
Hi Mark,

You can follow the instructions here:

Something like this:

$ spark-submit \
--conf 'spark.driver.extraJavaOptions=-Dalluxio.user.file.writetype.default=CACHE_THROUGH' \
--conf 'spark.executor.extraJavaOptions=-Dalluxio.user.file.writetype.default=CACHE_THROUGH' \
...

Hope it helps

- Bin

On Tue, Sep 17, 2019 at 7:53 AM Mark Zhao <[hidden email]> wrote:
Hi,

If Spark applications write data into alluxio, can WriteType be configured?

Thanks,
Mark