how to refresh the loaded non-streaming dataframe for each steaming batch ?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

how to refresh the loaded non-streaming dataframe for each steaming batch ?

Shyam P
Hi,

I am using spark-sql-2.4.1v to streaming in my PoC.

how to refresh the loaded dataframe from hdfs/cassandra table every time new batch of stream processed ? What is the practice followed in general to handle this kind of scenario?

Below is the SOF link for more details .

https://stackoverflow.com/questions/57815645/how-to-refresh-the-contents-of-non-streaming-dataframe

Thank you,
Shyam
 
Reply | Threaded
Open this post in threaded view
|

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

David Zhou
I have the same question with yours

On Thu, Sep 5, 2019 at 9:18 PM Shyam P <[hidden email]> wrote:
Hi,

I am using spark-sql-2.4.1v to streaming in my PoC.

how to refresh the loaded dataframe from hdfs/cassandra table every time new batch of stream processed ? What is the practice followed in general to handle this kind of scenario?

Below is the SOF link for more details .

https://stackoverflow.com/questions/57815645/how-to-refresh-the-contents-of-non-streaming-dataframe

Thank you,
Shyam
 
Reply | Threaded
Open this post in threaded view
|

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

Shyam P
cool ,but did you find a way or anyhelp or clue ?

On Fri, Sep 6, 2019 at 11:40 PM David Zhou <[hidden email]> wrote:
I have the same question with yours

On Thu, Sep 5, 2019 at 9:18 PM Shyam P <[hidden email]> wrote:
Hi,

I am using spark-sql-2.4.1v to streaming in my PoC.

how to refresh the loaded dataframe from hdfs/cassandra table every time new batch of stream processed ? What is the practice followed in general to handle this kind of scenario?

Below is the SOF link for more details .

https://stackoverflow.com/questions/57815645/how-to-refresh-the-contents-of-non-streaming-dataframe

Thank you,
Shyam
 
Reply | Threaded
Open this post in threaded view
|

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

David Zhou
Not yet. Learning spark

On Fri, Sep 6, 2019 at 2:17 PM Shyam P <[hidden email]> wrote:
cool ,but did you find a way or anyhelp or clue ?

On Fri, Sep 6, 2019 at 11:40 PM David Zhou <[hidden email]> wrote:
I have the same question with yours

On Thu, Sep 5, 2019 at 9:18 PM Shyam P <[hidden email]> wrote:
Hi,

I am using spark-sql-2.4.1v to streaming in my PoC.

how to refresh the loaded dataframe from hdfs/cassandra table every time new batch of stream processed ? What is the practice followed in general to handle this kind of scenario?

Below is the SOF link for more details .

https://stackoverflow.com/questions/57815645/how-to-refresh-the-contents-of-non-streaming-dataframe

Thank you,
Shyam
 
Reply | Threaded
Open this post in threaded view
|

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

Shyam P
Difficult things in spark is debugging and tuning.