how to copy from one cassandra cluster to another

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

how to copy from one cassandra cluster to another

Amit Sharma
Hi, I have table A in the cassandra cluster  cluster -1  in one data center. I have table B in cluster -2 in another data center. I want to copy the data from one cluster to another using spark. I faced the problem that I can not create two spark sessions as we need spark sessions per cluster. Please let me know if there is any way to use spark batch job to copy data among two cassandra clusters.


Thanks
Amit
Reply | Threaded
Open this post in threaded view
|

Re: how to copy from one cassandra cluster to another

Russell Spitzer
You do not need one spark session per cluster.

Spark SQL with Datasource v1

DatasourceV2

Would require making two catalog references then copying between them





On Tue, Jul 28, 2020, 6:26 AM Amit Sharma <[hidden email]> wrote:
Hi, I have table A in the cassandra cluster  cluster -1  in one data center. I have table B in cluster -2 in another data center. I want to copy the data from one cluster to another using spark. I faced the problem that I can not create two spark sessions as we need spark sessions per cluster. Please let me know if there is any way to use spark batch job to copy data among two cassandra clusters.


Thanks
Amit