Which kafka client to use with spark streaming

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Which kafka client to use with spark streaming

Serkan TAS

Hi,

 

Working on spark 2.2.0 cluster and 1.0 kafka brokers.

 

I was using the library

"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"

 

and had lots of problems during streaming process then downgraded to

               "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"

 

And i know there is also another path which is using kafka-clients jars which has the latest version of 1.0.0

 

<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->

<dependency>

    <groupId>org.apache.kafka</groupId>

    <artifactId>kafka-clients</artifactId>

    <version>1.0.0</version>

</dependency>

 

I am confused which path  is the  right one

 

Thanks…

 

 




Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content.
Reply | Threaded
Open this post in threaded view
|

Re: Which kafka client to use with spark streaming

Diogo Munaro Vieira
Hey Serkan, it depends of your Kafka version... Is it 0.8.2?

Em 25 de dez de 2017 06:17, "Serkan TAS" <[hidden email]> escreveu:

Hi,

 

Working on spark 2.2.0 cluster and 1.0 kafka brokers.

 

I was using the library

"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"

 

and had lots of problems during streaming process then downgraded to

               "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"

 

And i know there is also another path which is using kafka-clients jars which has the latest version of 1.0.0

 

<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->

<dependency>

    <groupId>org.apache.kafka</groupId>

    <artifactId>kafka-clients</artifactId>

    <version>1.0.0</version>

</dependency>

 

I am confused which path  is the  right one

 

Thanks…

 

 




Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content.
Reply | Threaded
Open this post in threaded view
|

Re: Which kafka client to use with spark streaming

Cody Koeninger
Do not add a dependency on kafka-clients, the spark-streaming-kafka
library has appropriate transitive dependencies.

Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?



On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira
<[hidden email]> wrote:

> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <[hidden email]> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>>                "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" %
>> "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients jars
>> which has the latest version of 1.0.0
>>
>>
>>
>> <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
>>
>> <dependency>
>>
>>     <groupId>org.apache.kafka</groupId>
>>
>>     <artifactId>kafka-clients</artifactId>
>>
>>     <version>1.0.0</version>
>>
>> </dependency>
>>
>>
>>
>> I am confused which path  is the  right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler
>> içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak
>> ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal
>> gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından
>> silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi
>> sorumludur.
>>
>> This communication may contain information that is legally privileged,
>> confidential or exempt from disclosure. If you are not the intended
>> recipient, please note that any dissemination, distribution, or copying of
>> this communication is strictly prohibited. Anyone who receives this message
>> in error should notify the sender immediately by telephone or by return
>> communication and delete it from his or her computer. Only the person who
>> has sent this message is responsible for its content.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Which kafka client to use with spark streaming

Serkan TAS
Kafka Clients are blocking spark streaming jobs and after a time streaming job queue increases.

-----Original Message-----
From: Cody Koeninger [mailto:[hidden email]]
Sent: Tuesday, December 26, 2017 6:47 PM
To: Diogo Munaro Vieira <[hidden email]>
Cc: Serkan TAS <[hidden email]>; user <[hidden email]>
Subject: Re: Which kafka client to use with spark streaming

Do not add a dependency on kafka-clients, the spark-streaming-kafka library has appropriate transitive dependencies.

Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?



On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira <[hidden email]> wrote:

> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <[hidden email]> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>>                "org.apache.spark" % "spark-streaming-kafka-0-8_2.11"
>> % "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients
>> jars which has the latest version of 1.0.0
>>
>>
>>
>> <!--
>> https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
>>
>> <dependency>
>>
>>     <groupId>org.apache.kafka</groupId>
>>
>>     <artifactId>kafka-clients</artifactId>
>>
>>     <version>1.0.0</version>
>>
>> </dependency>
>>
>>
>>
>> I am confused which path  is the  right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken
>> bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu
>> iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan
>> kişi, bu durumu derhal gönderene telefonla ya da e-posta ile
>> bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden
>> yalnızca iletiyi gönderen kişi sorumludur.
>>
>> This communication may contain information that is legally
>> privileged, confidential or exempt from disclosure. If you are not
>> the intended recipient, please note that any dissemination,
>> distribution, or copying of this communication is strictly
>> prohibited. Anyone who receives this message in error should notify
>> the sender immediately by telephone or by return communication and
>> delete it from his or her computer. Only the person who has sent this message is responsible for its content.

________________________________

Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]