Get Request for very large file with S3A

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Get Request for very large file with S3A

hellosaferide
This post has NOT been accepted by the mailing list yet.
Hi, I am creating a dataframe by reading a json file from s3 that is about 3.2GB and spark is making a get request with the whole range and also making smaller range requests. I would like to better understand what is going on. What is determining the range (where in the code) and why would there be a request for the full file?

Thanks