I process a lot of JSON files of different sizes. All files share the same overall structure. I have no issue with files of sizes around 150-300MB.
Another file of around 530MB now causes errors when I apply selectExpr on the resulting DF after reading the file.
cannot resolve '`entity`' given input columns: [_corrupt_record]; line 1 pos 6;
'Project ['LOWER('entity) AS entity#668391, 'extraction_model, 'part, 'pipeline_run_id, 'timestamp]
+- Relation[_corrupt_record#668389] json
The schema of the read DF looks like:
|-- _corrupt_record: string (nullable = true)
I analyzed the file and it contains all requested columns. Actually I can fully parse it using ijson, as I thought the issue might relate syntax errors.
Any hints on the _corrupt_record?