Datafarme save as table operation is failing when the child columns name contains special characters

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Datafarme save as table operation is failing when the child columns name contains special characters

abhijeet bedagkar

I am using SPARK to read the XML / JSON files to create a dataframe and save it as a hive table

Sample XML file:

Note field 'validation-timeout' under testexecutioncontroller.

Below is the schema populated by DF after reading the XML file

|-- id: long (nullable = true)
|-- testexecutioncontroller: struct (nullable = true)
|    |-- execution-timeout: long (nullable = true)
|    |-- execution-method: string (nullable = true)

While saving this dataframe to hive table I am getting below exception

Caused by: java.lang.IllegalArgumentException: Error: : expected at the position 24 of 'bigint:struct<execution-timeout:bigint,execution-method:string>' but '-' is found.        at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(        at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(        at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(        at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(        at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(        at        at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(        at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(        at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(        at org.apache

It looks like the issue is happening due to special character '-' in the field. As after removing the special character it iw working properly.

Could you please let me know if there is way to replaces all child column names so that it can be saved as table without any issue.

Creating the STRUCT FIELD from df.schema and recursively creating another STRUCTFIELD with renamed column is one solution I am aware of. But still wanted to check if there is easy way to do this.