listPartitionsByFilter Usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

listPartitionsByFilter Usage

navdeepniku
I'm trying to list partitions for a table with a filter using the below
function,

spark.sharedState.externalCatalog.listPartitionsByFilter()

Signature of listPartitionsByFilter in
org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener is as
below,

def listPartitionsByFilter(
      db: String,
      table: String,
      predicates: Seq[Expression],
      defaultTimeZoneId: String): Seq[CatalogTablePartition] = {

This is how I'm creating an expression to pass for the predicates parameter,
val exp = expr("`year`='2020' and `month`='03' and `day`='01'").expr
InterpretedPredicate.create(exp.expr).expression

spark.sharedState.externalCatalog.listPartitionsByFilter("prejoin",
"bookings", Seq(exp),"UTC")

This produces below error:

java.lang.UnsupportedOperationException: Cannot evaluate expression: 'year
  at
org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:255)

However with empty sequence for predicates parameter, it produces results

spark.sharedState.externalCatalog.listPartitionsByFilter("prejoin",
"bookings", Seq.empty,"UTC")

Can someone help with correct way of creating predicates for
listPartitionsByFilter function?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]