Will this use-case can be handled with spark-sql streaming and cassandra?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Will this use-case can be handled with spark-sql streaming and cassandra?

Shyam P
Hi,

I need to do a PoC for a business use-case.

Use case : Need to update a record in Cassandra table if exists.

Will spark streaming support compare each record and update existing Cassandra record ?

For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record.

How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have.

Thank you,

Shyam

Reply | Threaded
Open this post in threaded view
|

Re: Will this use-case can be handled with spark-sql streaming and cassandra?

Aayush Ranaut
Cassandra is upsert, you should be able to do what you need with a single statement unless you’re looking to maintain counters. 

I’m not sure if there is a Cassandra connector library written for spark streaming because we wrote one ourselves when we wanted to do the same.

Regards
Prathmesh Ranaut

On Aug 29, 2019, at 7:21 AM, Shyam P <[hidden email]> wrote:

Hi,

I need to do a PoC for a business use-case.

Use case : Need to update a record in Cassandra table if exists.

Will spark streaming support compare each record and update existing Cassandra record ?

For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record.

How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have.

Thank you,

Shyam

Reply | Threaded
Open this post in threaded view
|

Re: Will this use-case can be handled with spark-sql streaming and cassandra?

Shyam P
thanks Aayush.
     For every record I need to get the data from cassandra table and update it ? Else it may not update the existing record.

    What is this datastax-spark-connector ? is that not a "Cassandra connector library written for spark"?
If not , how to write ourselves.   
Where and how to start ? Can you please guide me.

Thank you.
Shyam


On Thu, Aug 29, 2019 at 5:03 PM Aayush Ranaut <[hidden email]> wrote:
Cassandra is upsert, you should be able to do what you need with a single statement unless you’re looking to maintain counters. 

I’m not sure if there is a Cassandra connector library written for spark streaming because we wrote one ourselves when we wanted to do the same.

Regards
Prathmesh Ranaut

On Aug 29, 2019, at 7:21 AM, Shyam P <[hidden email]> wrote:

Hi,

I need to do a PoC for a business use-case.

Use case : Need to update a record in Cassandra table if exists.

Will spark streaming support compare each record and update existing Cassandra record ?

For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record.

How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have.

Thank you,

Shyam

Reply | Threaded
Open this post in threaded view
|

Re: Will this use-case can be handled with spark-sql streaming and cassandra?

Aayush Ranaut
What exactly is your requirement? 
Is the read before write mandatory?
Are you maintaining states in Cassandra?

Regards
On Aug 29, 2019, at 3:35 PM, Shyam P <[hidden email]> wrote:

thanks Aayush.
     For every record I need to get the data from cassandra table and update it ? Else it may not update the existing record.

    What is this datastax-spark-connector ? is that not a "Cassandra connector library written for spark"?
If not , how to write ourselves.   
Where and how to start ? Can you please guide me.

Thank you.
Shyam


On Thu, Aug 29, 2019 at 5:03 PM Aayush Ranaut <[hidden email]> wrote:
Cassandra is upsert, you should be able to do what you need with a single statement unless you’re looking to maintain counters. 

I’m not sure if there is a Cassandra connector library written for spark streaming because we wrote one ourselves when we wanted to do the same.

Regards
Prathmesh Ranaut

On Aug 29, 2019, at 7:21 AM, Shyam P <[hidden email]> wrote:

Hi,

I need to do a PoC for a business use-case.

Use case : Need to update a record in Cassandra table if exists.

Will spark streaming support compare each record and update existing Cassandra record ?

For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record.

How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have.

Thank you,

Shyam

Reply | Threaded
Open this post in threaded view
|

Re: Will this use-case can be handled with spark-sql streaming and cassandra?

Jörn Franke
In reply to this post by Shyam P
1) this is not a use case, but a technical solution. Hence nobody can tell you if it make sense or not
2) do an upsert in Cassandra. However keep in mind that the application submitting to the Kafka topic and the one consuming from the Kafka topic need to ensure that they process messages in the right order. This may not be always guaranteed, eg in case of errors, and they need to avoid overwriting new data with old data. This is also not a Kafka setting that has to be dealt with at producer and consumer level

Am 29.08.2019 um 13:21 schrieb Shyam P <[hidden email]>:

Hi,

I need to do a PoC for a business use-case.

Use case : Need to update a record in Cassandra table if exists.

Will spark streaming support compare each record and update existing Cassandra record ?

For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record.

How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have.

Thank you,

Shyam