Elastic Search sink showing -1 for numOutputRows

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Elastic Search sink showing -1 for numOutputRows

jainshasha
Hi,

Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.

What could be the reason for showing -1 for ElasticSearch ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search sink showing -1 for numOutputRows

jainshasha
Hi,

Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.

What could be the reason for showing -1 for ElasticSearch ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search sink showing -1 for numOutputRows

Jungtaek Lim-2
I don't know about ES sink. The availability of "numOutputRows" depends on the API version the sink is implementing (DSv1 vs DSv2), so you may be better to ask a question to the author of ES sink and confirm the case.

On Tue, Sep 8, 2020 at 5:15 AM jainshasha <[hidden email]> wrote:
Hi,

Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.

What could be the reason for showing -1 for ElasticSearch ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search sink showing -1 for numOutputRows

jainshasha
Thanks Jungtaek Lim-2 for replying.

May i knw the reference of the API version for sink for both types (DSv1 and
DSv2) in code ?
Where could i see it ? Under what module of spark code ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]