spark cassandra questiom

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

spark cassandra questiom

adfel70
I an very very new to both spark and spark structured streaming. I have to
write an application that receives a very very large csv files in hdfs
folder. the app must take the file and on each row it must read from
Cassandra data base some rows (not many rows will be returned for each row
in csv). On each row it red it must preform some simple calculations and
update the rows it red with the results and save the updated rows to
Cassandra.

I have spark version 2.4 and must use python.

Is this a suitable scenario for spark structured streaming?

thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark cassandra questiom

Sonal Goyal
Yes, it should be good to use Spark for this use case in my opinion. You can look into using the Cassandra Spark connector for persisting your updated data into Cassandra.

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 10, 2020 at 6:39 PM adfel70 <[hidden email]> wrote:
I an very very new to both spark and spark structured streaming. I have to
write an application that receives a very very large csv files in hdfs
folder. the app must take the file and on each row it must read from
Cassandra data base some rows (not many rows will be returned for each row
in csv). On each row it red it must preform some simple calculations and
update the rows it red with the results and save the updated rows to
Cassandra.

I have spark version 2.4 and must use python.

Is this a suitable scenario for spark structured streaming?

thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]