[Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Amit Joshi
Hi All,

I am trying to understand the effect of adding topics and partitions to a topic in kafka, which is being consumed by spark structured streaming applications.

Do we have to restart the spark structured streaming application to read from the newly added topic?
Do we have to restart the spark structured streaming application to read from the newly added partition to a topic?

Kafka consumers have a meta data refresh property that works without restarting.

Thanks advance.

Regards
Amit Joshi
Reply | Threaded
Open this post in threaded view
|

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Amit Joshi
Any pointers will be appreciated.

On Thursday, August 27, 2020, Amit Joshi <[hidden email]> wrote:
Hi All,

I am trying to understand the effect of adding topics and partitions to a topic in kafka, which is being consumed by spark structured streaming applications.

Do we have to restart the spark structured streaming application to read from the newly added topic?
Do we have to restart the spark structured streaming application to read from the newly added partition to a topic?

Kafka consumers have a meta data refresh property that works without restarting.

Thanks advance.

Regards
Amit Joshi
Reply | Threaded
Open this post in threaded view
|

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Jungtaek Lim-2
Hi Amit,

if I remember correctly, you don't need to restart the query to reflect the newly added topic and partition, if your subscription covers the topic (like subscribe pattern). Please try it out.

Hope this helps.

Thanks,
Jungtaek Lim (HeartSaVioR)

On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <[hidden email]> wrote:
Any pointers will be appreciated.

On Thursday, August 27, 2020, Amit Joshi <[hidden email]> wrote:
Hi All,

I am trying to understand the effect of adding topics and partitions to a topic in kafka, which is being consumed by spark structured streaming applications.

Do we have to restart the spark structured streaming application to read from the newly added topic?
Do we have to restart the spark structured streaming application to read from the newly added partition to a topic?

Kafka consumers have a meta data refresh property that works without restarting.

Thanks advance.

Regards
Amit Joshi
Reply | Threaded
Open this post in threaded view
|

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Gabor Somogyi
Hi Amit,

The answer is no.

G


On Fri, Aug 28, 2020 at 9:16 AM Jungtaek Lim <[hidden email]> wrote:
Hi Amit,

if I remember correctly, you don't need to restart the query to reflect the newly added topic and partition, if your subscription covers the topic (like subscribe pattern). Please try it out.

Hope this helps.

Thanks,
Jungtaek Lim (HeartSaVioR)

On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <[hidden email]> wrote:
Any pointers will be appreciated.

On Thursday, August 27, 2020, Amit Joshi <[hidden email]> wrote:
Hi All,

I am trying to understand the effect of adding topics and partitions to a topic in kafka, which is being consumed by spark structured streaming applications.

Do we have to restart the spark structured streaming application to read from the newly added topic?
Do we have to restart the spark structured streaming application to read from the newly added partition to a topic?

Kafka consumers have a meta data refresh property that works without restarting.

Thanks advance.

Regards
Amit Joshi
Reply | Threaded
Open this post in threaded view
|

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

Amit Joshi
In reply to this post by Jungtaek Lim-2
Hi Jungtaek,

Thanks for the input. I did tried and it worked.
I got confused earlier after reading some blogs.

Regards
Amit

On Friday, August 28, 2020, Jungtaek Lim <[hidden email]> wrote:
Hi Amit,

if I remember correctly, you don't need to restart the query to reflect the newly added topic and partition, if your subscription covers the topic (like subscribe pattern). Please try it out.

Hope this helps.

Thanks,
Jungtaek Lim (HeartSaVioR)

On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <[hidden email]> wrote:
Any pointers will be appreciated.

On Thursday, August 27, 2020, Amit Joshi <[hidden email]> wrote:
Hi All,

I am trying to understand the effect of adding topics and partitions to a topic in kafka, which is being consumed by spark structured streaming applications.

Do we have to restart the spark structured streaming application to read from the newly added topic?
Do we have to restart the spark structured streaming application to read from the newly added partition to a topic?

Kafka consumers have a meta data refresh property that works without restarting.

Thanks advance.

Regards
Amit Joshi