Writing to vertica from spark

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing to vertica from spark

Nikhil Goyal
Hi guys,

My code is failing with this error

java.lang.Exception: S2V: FATAL ERROR for job S2V_job9197956021769393773. Job status information is available in the Vertica table test.S2V_JOB_STATUS_USER_NGOYAL.  Unable to save intermediate orc files to HDFS path:hdfs://hadoop-dw2-nn.smf1.com/tmp/S2V_job9197956021769393773Error message:org.apache.spark.sql.AnalysisException: The ORC data source must be used with Hive support enabled;

This is how I am writing it. I followed steps from this link.
dataFrame
.write
.format("com.vertica.spark.datasource.DefaultSource")
.options(connectionProperties)
.mode(SaveMode.Append)
.save()
Does anybody have any idea how to fix this?

Thanks
Nikhil
Reply | Threaded
Open this post in threaded view
|

Re: Writing to vertica from spark

Nikhil Goyal
Fixed this by setting fileformat -> "parquet"

On Mon, Oct 22, 2018 at 11:48 AM Nikhil Goyal <[hidden email]> wrote:
Hi guys,

My code is failing with this error

java.lang.Exception: S2V: FATAL ERROR for job S2V_job9197956021769393773. Job status information is available in the Vertica table test.S2V_JOB_STATUS_USER_NGOYAL.  Unable to save intermediate orc files to HDFS path:hdfs://hadoop-dw2-nn.smf1.com/tmp/S2V_job9197956021769393773Error message:org.apache.spark.sql.AnalysisException: The ORC data source must be used with Hive support enabled;

This is how I am writing it. I followed steps from this link.
dataFrame
.write
.format("com.vertica.spark.datasource.DefaultSource")
.options(connectionProperties)
.mode(SaveMode.Append)
.save()
Does anybody have any idea how to fix this?

Thanks
Nikhil