Exception during writing a spark Dataframe to Redshift

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Exception during writing a spark Dataframe to Redshift

Sandeep Patra
This is where the exception occurs:

myAppDes.coalesce(1)
        .write
        .format("com.databricks.spark.redshift")
        .option("url", redshiftURL)
        .option("dbtable", redshiftTableName)
        .option("forward_spark_s3_credentials", "true")
        .option("tempdir", "s3a://zest-hevo-datalake/temp/data")
        .mode(SaveMode.Append)
        .save()

I have attached the stack trace with the email.

My build.sbt looks like:
version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "2.3.0"
val hadoopVersion = "3.1.2"

resolvers ++= Seq(
  "apache-snapshots" at "http://repository.apache.org/snapshots/",
  "redshift" at "https://s3.amazonaws.com/redshift-maven-repository/release",
  "redshift" at "http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release",
  "jitpack" at "https://jitpack.io"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.hadoop" % "hadoop-aws" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-common" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion,
  "com.amazon.redshift" % "redshift-jdbc42-no-awssdk" % "1.2.15.1025",
  "com.databricks" %% "spark-redshift" % "3.0.0-preview1",
  "com.amazon.redshift" % "redshift-jdbc42" % "1.2.1.1001"
)

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.8"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.8"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.8"

I have checked and I can read data from the same Redshift Instance. I also have created table in redshift. (I also tried it without creating table but got the same error)


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

exception (30K) Download Attachment