Spark DataSet class is not truly private[sql]

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Spark DataSet class is not truly private[sql]

Nirav Patel
I see Spark dataset is defined as:

class Dataset[T] private[sql](

    @transient val sparkSession: SparkSession,

    @DeveloperApi @InterfaceStability.Unstable @transient val queryExecution: QueryExecution,

    encoder: Encoder[T])

However it has public constructors which allows DataSet to be extended which I don't think is intended by Developer. 

  def this(sparkSession: SparkSession, logicalPlan: LogicalPlan, encoder: Encoder[T]) = {
    this(sparkSession, sparkSession.sessionState.executePlan(logicalPlan), encoder)

  def this(sqlContext: SQLContext, logicalPlan: LogicalPlan, encoder: Encoder[T]) = {
    this(sqlContext.sparkSession, logicalPlan, encoder)