keep getting empty table while using saveAsTable() to save DataFrame as table
I am using Spark SQL in a cluster, and try to use the CBO supported in
Spark SQL. The dataset that I am using is TPC-DS. In order to collect the
statistic of these data, I first load the data from HDFS to create
dataframes. Then use saveAsTable() to save dataframes as tables. All the
process seems to be fine, and I didn't get any exceptions. However, when I
was trying to check those created tables, all of them is empty.
Question: Did anyone have the same problem when you are using the CBO
provided in Spark SQL? Please help me with this issue. Thank you very much.