I am working on finding on the below information. I don’t see any helpful documentation on this. Please advise
Currently, all our applications are in Spark 2.2. We are planning to update our applications to Spark 3.0.I am doing a feature analysis on Spark 3.0 and document suggestions on what can we use from Spark 3.
Following are the features that we heavily use from Spark 2.2
updateStateByKey for Stateful Processing
Zookeeper based Kafka Integration
spark-avro package for Serialization/Deserialization and custom features built over it
I would like to understand whether any of the above features are impacted significantly / or alternative efficient features available in Spark 3.0
We are planning to migrate from YARN Scheduler to Kubernetes Scheduler. What are the new features available there?
Any new significant SQL optimization available there?