Hi,
The spark job where we save a dataset to a give table using
exampledataset.saveAsTable() fails with the following error after upgrading
from spark 2.4 to spark 3.0
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot
write incompatible data to table '<TABLE_NAME>':
- Cannot write nullable values to non-null column '<COLUMN_NAME>'
- Cannot write nullable values to non-null column '<COLUMN_NAME>';
at
org.apache.spark.sql.catalyst.analysis.TableOutputResolver$.resolveOutputColumns(TableOutputResolver.scala:72)
We use glue as hive metastore. Checked the table definition, it is set as
nullable. In dataset too it is nullable. Wondering from where is it getting
the non-null constraint.
Any help would be appreciated.
Below is the spark source code snippet:
case StoreAssignmentPolicy.STRICT | StoreAssignmentPolicy.ANSI =>
// run the type check first to ensure type errors are present
val canWrite = DataType.canWrite(
queryExpr.dataType, tableAttr.dataType, byName, conf.resolver,
tableAttr.name,
storeAssignmentPolicy, addError)
if (queryExpr.nullable && !tableAttr.nullable) {
addError(s"Cannot write nullable values to non-null column
'${tableAttr.name}'")
None
}
--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/---------------------------------------------------------------------
To unsubscribe e-mail:
[hidden email]
Locked
