Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

Sachit Murarka
Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 
Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

Prashant Sharma
 A lot of developers may have already moved to 3.0.x, FYI 3.1.0 is just around the corner hopefully(in a few days) and has a lot of improvements to spark on K8s, including it will be transitioning from experimental to GA in this release.

On Tue, Jan 5, 2021 at 12:41 AM Sachit Murarka <[hidden email]> wrote:
Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 
Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

Sachit Murarka
Thanks for the link Prashant.

Regards
Sachit

On Tue, 5 Jan 2021, 15:08 Prashant Sharma, <[hidden email]> wrote:
 A lot of developers may have already moved to 3.0.x, FYI 3.1.0 is just around the corner hopefully(in a few days) and has a lot of improvements to spark on K8s, including it will be transitioning from experimental to GA in this release.

On Tue, Jan 5, 2021 at 12:41 AM Sachit Murarka <[hidden email]> wrote:
Hi Users,

Could you please tell which Spark version have you used in Production for Kubernetes. 
Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. 

Thanks !

Kind Regards,
Sachit Murarka