I am testing on Spark 3.0 preview release

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

I am testing on Spark 3.0 preview release

Punya Maremalla
Hi All,

I download spark-3.0.0-preview-bin-hadoop2.7.tgz and trying to  access new api’s like from_csv, map_entries. Don’t know for some reason I am getting these error.

For some reason dont see the from_csv function at all.

scala> import org.apache.spark.sql.functions.from_
from_json   from_unixtime   from_utc_timestamp

I am working on finding on the below information. I don’t see any helpful documentation on this. Please advise

Currently, all our applications are in Spark 2.2. We are planning to update our applications to Spark 3.0.I am doing a feature analysis on Spark 3.0 and document suggestions on what can we use from Spark 3.

Following are the features that we heavily use from Spark 2.2

  • RDD
  • Dataframe SQL
  • YARN Scheduler
  • Streaming
    • updateStateByKey for Stateful Processing
    • Zookeeper based Kafka Integration
    • spark-avro package for Serialization/Deserialization and custom features built over it
  1. I would like to understand whether any of the above features are impacted significantly / or alternative efficient features available in Spark 3.0
  2. We are planning to migrate from YARN Scheduler to Kubernetes Scheduler. What are the new features available there?
  3. Any new significant SQL optimization available there?

Please let know thanks in advance.