Driver OOM when using writing parquet

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Driver OOM when using writing parquet

Nikhil Goyal
Hi guys,

I have a simple job which reads LZO thrift files and writes them in Parquet Format. Driver is going out of memory. Parquet writer does keep some meta info in memory but that should cause the executor to go out of memory. No computation is being done on the driver. Any idea what could be the reason?


Thanks
Nikhil