Spark 3.0 upgrade non compatible table error

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark 3.0 upgrade non compatible table error

lsn24
Hi,

 The spark job where we save a dataset to a give table using
exampledataset.saveAsTable() fails with the following error after upgrading
from spark 2.4 to spark 3.0

Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot
write incompatible data to table '<TABLE_NAME>':
- Cannot write nullable values to non-null column '<COLUMN_NAME>'
- Cannot write nullable values to non-null column '<COLUMN_NAME>';
        at
org.apache.spark.sql.catalyst.analysis.TableOutputResolver$.resolveOutputColumns(TableOutputResolver.scala:72)

We use glue as hive metastore. Checked the table definition, it is set as
nullable. In dataset too it  is nullable. Wondering from where is it getting
the non-null constraint.

Any help would be appreciated.

Below is the spark source code snippet:

 case StoreAssignmentPolicy.STRICT | StoreAssignmentPolicy.ANSI =>
        // run the type check first to ensure type errors are present
        val canWrite = DataType.canWrite(
          queryExpr.dataType, tableAttr.dataType, byName, conf.resolver,
tableAttr.name,
          storeAssignmentPolicy, addError)
        if (queryExpr.nullable && !tableAttr.nullable) {
          addError(s"Cannot write nullable values to non-null column
'${tableAttr.name}'")
          None

        }



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]