PickleException when collecting DataFrame containing empty bytearray()

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

PickleException when collecting DataFrame containing empty bytearray()

tot0
This post has NOT been accepted by the mailing list yet.
Hi All,

I tried creating a DataFrame containing an empty bytearray, bytearray(b''). The DataFrame can be created but upon collection the following Exception occurs, repro steps are also at the top of the pyspark shell:

pyspark gist

Bytearray's that aren't empty cause no issues, as can be seen at the end of the gist.

This doesn't seem right but I'm not sure if this is an undiscovered bug, a discovered bug I couldn't locate online or intended behavior.

Any help is appreciated.
Cheers, Lucas Pickup
Loading...