Using startingOffsets latest - no data from structured streaming kafka query

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using startingOffsets latest - no data from structured streaming kafka query

Ruijing Li
Hi all,

Apologies if this has been asked before, but I could not find the answer to this question. We have a structured streaming job, but for some reason, if we use startingOffsets = latest with foreachbatch mode, it doesn’t produce any data. 

Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting offset for partition to offset” over and over again.. 

However with startingOffsets=earliest, we don’t get this issue. I’m wondering then how we can use startingOffsets=latest as I wish to start from the latest offset available.
--
Cheers,
Ruijing Li
Reply | Threaded
Open this post in threaded view
|

Re: Using startingOffsets latest - no data from structured streaming kafka query

Jungtaek Lim-2
Did you provide more records to topic "after" you started the query? That's the only one I can imagine based on such information.

On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <[hidden email]> wrote:
Hi all,

Apologies if this has been asked before, but I could not find the answer to this question. We have a structured streaming job, but for some reason, if we use startingOffsets = latest with foreachbatch mode, it doesn’t produce any data. 

Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting offset for partition to offset” over and over again.. 

However with startingOffsets=earliest, we don’t get this issue. I’m wondering then how we can use startingOffsets=latest as I wish to start from the latest offset available.
--
Cheers,
Ruijing Li
Reply | Threaded
Open this post in threaded view
|

Re: Using startingOffsets latest - no data from structured streaming kafka query

Ruijing Li
Yes, we did. But for some reason latest does not show them. The count is always 0.

On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim <[hidden email]> wrote:
Did you provide more records to topic "after" you started the query? That's the only one I can imagine based on such information.

On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <[hidden email]> wrote:
Hi all,

Apologies if this has been asked before, but I could not find the answer to this question. We have a structured streaming job, but for some reason, if we use startingOffsets = latest with foreachbatch mode, it doesn’t produce any data. 

Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting offset for partition to offset” over and over again.. 

However with startingOffsets=earliest, we don’t get this issue. I’m wondering then how we can use startingOffsets=latest as I wish to start from the latest offset available.
--
Cheers,
Ruijing Li
--
Cheers,
Ruijing Li
Reply | Threaded
Open this post in threaded view
|

Re: Using startingOffsets latest - no data from structured streaming kafka query

Ruijing Li
For some reason, after restarting the app and trying again, latest now works as expected. Not sure why it didn’t work before.

On Tue, Apr 21, 2020 at 1:46 PM Ruijing Li <[hidden email]> wrote:
Yes, we did. But for some reason latest does not show them. The count is always 0.

On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim <[hidden email]> wrote:
Did you provide more records to topic "after" you started the query? That's the only one I can imagine based on such information.

On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <[hidden email]> wrote:
Hi all,

Apologies if this has been asked before, but I could not find the answer to this question. We have a structured streaming job, but for some reason, if we use startingOffsets = latest with foreachbatch mode, it doesn’t produce any data. 

Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting offset for partition to offset” over and over again.. 

However with startingOffsets=earliest, we don’t get this issue. I’m wondering then how we can use startingOffsets=latest as I wish to start from the latest offset available.
--
Cheers,
Ruijing Li
--
Cheers,
Ruijing Li
--
Cheers,
Ruijing Li