Databricks Spark Parallelism and Shuffle Partitions

[Erica Lin]

Hello!

Is there a way to set spark.sql.shuffle.partitions and spark.default.parallelism in Databricks? I checked the event log and can't find those parameters in the log either. Is it something that Databricks sets automatically?

Thanks,

Erica

[Subhash Sriram]

Hi Erica,

On your cluster details, you can click on "Advanced", and then set those parameters in the "Spark" tab. Hope that helps.

Thanks,

Subhash

On Thu, Feb 4, 2021 at 5:27 PM Erica Lin <[hidden email]> wrote:

Hello!

Is there a way to set spark.sql.shuffle.partitions and spark.default.parallelism in Databricks? I checked the event log and can't find those parameters in the log either. Is it something that Databricks sets automatically?

Thanks,

Erica