Bucketing 3.1.1

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Bucketing 3.1.1

German Schiavon Matteo
Hi all!

In the 3.1.1 release a new bucket property was added in this PR.

I'm trying to check this new behaviour but I'm not getting the same physical plan as the one given in the example.

I'm executing the same code snippet from the PR in a 3.1.1 spark-shell.

What am I missing? 

Best
Reply | Threaded
Open this post in threaded view
|

Re: Bucketing 3.1.1

Bartosz Konieczny
Hi German Schiavon,

The property is supported in shuffle hash join strategy too and it was renamed here https://github.com/apache/spark/pull/29079/files. Try with spark.sql.bucketing.coalesceBucketsInJoin.enabled instead of spark.sql.bucketing.coalesceBucketsInSortMergeJoin.enabled :)

Best,
Bartosz.

On Mon, Mar 22, 2021 at 8:52 AM German Schiavon <[hidden email]> wrote:
Hi all!

In the 3.1.1 release a new bucket property was added in this PR.

I'm trying to check this new behaviour but I'm not getting the same physical plan as the one given in the example.

I'm executing the same code snippet from the PR in a 3.1.1 spark-shell.

What am I missing? 

Best


--
Reply | Threaded
Open this post in threaded view
|

Re: Bucketing 3.1.1

German Schiavon Matteo
Ohh! That is why! I missed that rename 😅

Thanks a lot Bartosz! 

On Mon, 22 Mar 2021 at 09:55, Bartosz Konieczny <[hidden email]> wrote:
Hi German Schiavon,

The property is supported in shuffle hash join strategy too and it was renamed here https://github.com/apache/spark/pull/29079/files. Try with spark.sql.bucketing.coalesceBucketsInJoin.enabled instead of spark.sql.bucketing.coalesceBucketsInSortMergeJoin.enabled :)

Best,
Bartosz.

On Mon, Mar 22, 2021 at 8:52 AM German Schiavon <[hidden email]> wrote:
Hi all!

In the 3.1.1 release a new bucket property was added in this PR.

I'm trying to check this new behaviour but I'm not getting the same physical plan as the one given in the example.

I'm executing the same code snippet from the PR in a 3.1.1 spark-shell.

What am I missing? 

Best


--