Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

[Sachit Murarka]

Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 

Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

[Prashant Sharma]

 A lot of developers may have already moved to 3.0.x, FYI 3.1.0 is just around the corner hopefully(in a few days) and has a lot of improvements to spark on K8s, including it will be transitioning from experimental to GA in this release.

See: https://issues.apache.org/jira/browse/SPARK-33005

Thanks,

On Tue, Jan 5, 2021 at 12:41 AM Sachit Murarka <[hidden email]> wrote:

Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 

Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

Kind Regards,
Sachit Murarka

[Sachit Murarka]

Thanks for the link Prashant.

Regards

Sachit

On Tue, 5 Jan 2021, 15:08 Prashant Sharma, <[hidden email]> wrote:

 A lot of developers may have already moved to 3.0.x, FYI 3.1.0 is just around the corner hopefully(in a few days) and has a lot of improvements to spark on K8s, including it will be transitioning from experimental to GA in this release.

See: https://issues.apache.org/jira/browse/SPARK-33005

Thanks,

On Tue, Jan 5, 2021 at 12:41 AM Sachit Murarka <[hidden email]> wrote:

Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 

Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

Kind Regards,
Sachit Murarka