Strange behavior with "bigger" JSON file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Strange behavior with "bigger" JSON file

Bode, Meikel, NMA-CFD

Hi all,


I process a lot of JSON files of different sizes. All files share the same overall structure. I have no issue with files of sizes around 150-300MB.

Another file of around 530MB now causes errors when I apply selectExpr on the resulting DF after reading the file.


AnalysisException: cannot resolve '`entity`' given input columns: [_corrupt_record]; line 1 pos 6;

'Project ['LOWER('entity) AS entity#668391, 'extraction_model, 'part, 'pipeline_run_id, 'timestamp]

+- Relation[_corrupt_record#668389] json


The schema of the read DF looks like:


 |-- _corrupt_record: string (nullable = true)


I analyzed the file and it contains all requested columns. Actually I can fully parse it using ijson, as I thought the issue might relate syntax errors.


Any hints on the _corrupt_record?