[Structured Streaminig] multiple queries in one application

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Structured Streaminig] multiple queries in one application

lec ssmi
Hi:
   I run a lot of queries in one spark structured streaming application. I found that when one query fails, other queries can continue to run. But there is no abnormal information.So the  queries are getting less and less, and we can't find the reason. Is there any good solution for this situation?

Best
Lec Ssmi

Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaminig] multiple queries in one application

Abhisheks
I hope you are using the Query object that is returned by the Structured
streaming, right?
Returned object contains a lot of information about each query and tracking
state of the object should be helpful.

Hope this may help, if not can you please share more details with examples?

Best,
A



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaminig] multiple queries in one application

lec ssmi
For example, put the generated query into a list  and start every one, then use the method awaitTermination() on the last one .

Abhisheks <[hidden email]> 于2020年5月1日周五 上午10:32写道:
I hope you are using the Query object that is returned by the Structured
streaming, right?
Returned object contains a lot of information about each query and tracking
state of the object should be helpful.

Hope this may help, if not can you please share more details with examples?

Best,
A



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]