org.apache.spark.sql.AnalysisException: Detected implicit cartesian product

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

org.apache.spark.sql.AnalysisException: Detected implicit cartesian product

kyunam
"left join" complains and tells me I need to turn on
"spark.sql.crossJoin.enabled=true".

But when I persist one dataframe, it runs fine.

Why do you have to "persist"?

org.apache.spark.sql.AnalysisException: Detected implicit cartesian product
for INNER join between logical plans

SELECT * FROM LHS left join RHS on LHS.R = RHS.R

The above happens in both Spark 2.3.3 and 2.4.4.

Thanks,
Kyunam



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]