Databricks Spark Parallelism and Shuffle Partitions

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Databricks Spark Parallelism and Shuffle Partitions

Erica Lin
Hello!

Is there a way to set spark.sql.shuffle.partitions and spark.default.parallelism in Databricks? I checked the event log and can't find those parameters in the log either. Is it something that Databricks sets automatically?

Thanks,
Erica
Reply | Threaded
Open this post in threaded view
|

Re: Databricks Spark Parallelism and Shuffle Partitions

Subhash Sriram
Hi Erica,

On your cluster details, you can click on "Advanced", and then set those parameters in the "Spark" tab. Hope that helps.

Thanks,
Subhash

On Thu, Feb 4, 2021 at 5:27 PM Erica Lin <[hidden email]> wrote:
Hello!

Is there a way to set spark.sql.shuffle.partitions and spark.default.parallelism in Databricks? I checked the event log and can't find those parameters in the log either. Is it something that Databricks sets automatically?

Thanks,
Erica