Re: How to upgrade kafka client in spark_streaming_kafka 2.2

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Gabor Somogyi
Kafka client upgrade is not a trivial change which may or may not work since new versions can contain incompatible API and/or behavior changes.
I've collected how Spark evolved in terms of Kafka client and there I've gathered the breaking changes to make our life easier.
Have a look and based on that you can make your choice: https://gist.github.com/gaborgsomogyi/3476c32d69ff2087ed5d7d031653c7a9

As a general suggestion it would be best to upgrade Spark as-is because we've added many fixes which one can face...

Hope this helps!

G


On Fri, Mar 12, 2021 at 9:45 AM Renu Yadav <[hidden email]> wrote:
Hi Team,
 I am using spark -2.2 and spark_streamin_kafka 2.2  , which is pointing to kafka-client 0.10 . How can I upgrade a kafka client to kafka 2.2.0 ?

Thanks & Regards,
Renu Yadav
Reply | Threaded
Open this post in threaded view
|

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Gabor Somogyi
Mainly bugfixes and no breaking AFAIK.

As a side note there were intentions to close DStreams and discontinue as-is.
It's not yet happened but it's on the road so I strongly recommend to migrate to Structured Streaming...
We simply can't support 2 streaming engines for huge amount of time.

G


On Fri, Mar 12, 2021 at 3:02 PM Renu Yadav <[hidden email]> wrote:
Hi Gabor,
 
It seems like it is better to upgrade my spark version . 

Are there major changes in terms of streaming from spark 2.2 to spark 2.4?

PS: I am using KafkaUtils api to create steam

Thanks & Regards,
Renu yadav

On Fri, Mar 12, 2021 at 7:25 PM Renu Yadav <[hidden email]> wrote:
Thanks Gabor,
This is  very useful.

Regards,
Renu Yadav

On Fri, Mar 12, 2021 at 5:36 PM Gabor Somogyi <[hidden email]> wrote:
Kafka client upgrade is not a trivial change which may or may not work since new versions can contain incompatible API and/or behavior changes.
I've collected how Spark evolved in terms of Kafka client and there I've gathered the breaking changes to make our life easier.
Have a look and based on that you can make your choice: https://gist.github.com/gaborgsomogyi/3476c32d69ff2087ed5d7d031653c7a9

As a general suggestion it would be best to upgrade Spark as-is because we've added many fixes which one can face...

Hope this helps!

G


On Fri, Mar 12, 2021 at 9:45 AM Renu Yadav <[hidden email]> wrote:
Hi Team,
 I am using spark -2.2 and spark_streamin_kafka 2.2  , which is pointing to kafka-client 0.10 . How can I upgrade a kafka client to kafka 2.2.0 ?

Thanks & Regards,
Renu Yadav