I want to load a 10TB parquet File from S3 and I'm trying to decide what EC2 instances to use.Should I go for instances that in total have a larger memory size than 10TB? Or is it enough that they have in total enough SSD storage so that everything can be spilled to disk?
How many columns do you need from the big file?
Also how CPU / memory intensive are the computations you want to perform?
Basically there is one very big shuffle going on the rest is not that heavy. The CPU intense lifting was done before that.
I have a temporary result file ( the 10TB one) that looks like thisI have around 3 billion rows of (url,url_list,language,vector,
The idea at the moment is to load the URL and URL_list column from the parquet and give ever row an ID. Then exploded the URL_list and join the IDs to this on the now exploded rows. After that I drop the URLs from URL_list column. For the rest of the computation I only load those rows from the parquet that have values in (language,vector and text) and join them with the ID table.
In the end I will create 3 tables:
1. url, ID2. ID, ID
On Mon, Nov 27, 2017 at 12:03 PM, Alexander Czech <[hidden email]> wrote:
it would be much simpler in case you just provide two tables with the samples of input and output. Going through the verbose text and trying to read and figure out what is happening is a bit daunting.
Personally, given that you have your entire data in Parquet, I do not think that you will need to have a large cluster size at all. You can do it with a small size cluster as well, but depending on the cluster size, you might want to create intermediate staging tables or persist the data.
Also it will be of help if you could kindly provide the EMR version that you are using.
On another note also mention the AWS Region you are in. If Redshift Spectrum is available, or you can use Athena, or you can use Presto, then running massive aggregates over huge data sets at fraction of cost and at least 10x speed may be handy as well.
Let me know in case you need any further help.
On Mon, Nov 27, 2017 at 11:05 AM, Alexander Czech <[hidden email]> wrote:
I don't use EMR I spin my clusters up using flintrock (beeing a student my budget is slim), my code is writen in pyspark and my data is in the us-east-1 region (N. Virginia). I will do my best explaining it with tables:My input with a size of (10TB) sits in multiple (~150) parquets on S3
(link_list is a ArrayType(StringType()))
+-----------+--------------------------+-------+------+-------+ | uri| link_list|lang_id|vector|content| +-----------+--------------------------+-------+------+-------+ |www.123.com|[www.123.com,www.abc.com,]| null| null| null| |www.abc.com|[www.opq.com,www.456.com,]| null| null| null| |www.456.com|[www.xyz.com,www.abc.com,]| null| null| null|
Step1 : I only load the uri and link_list columns (but they make up the bulk of the data). Then every uri is given a unique ID with df.withColumn('uri_id', func.monotonically_increasing_id())
resulting in a dataframe looking like this
+-----------+--------------------------+-------+ | uri| link_list| uri_id| +-----------+--------------------------+-------+ |www.123.com|[www.123.com,www.abc.com,]| 1| |www.abc.com|[www.opq.com,www.456.com,]| 2| |www.456.com|[www.xyz.com,www.abc.com,]| 3|
Step 2: I create another dataframe containing only the uri and uri_id which is renamed to link_id fields
Step 3: Now I exploded the link_list field in DF_A with DF_A.select("uri_id", func.explode("link_list").alias("link"))
Lastly I Join DF_B DF_C DF_C.join(DF_B, DF_C.link==DF_B.uri, "left_outer").drop("uri") Which results in the final dataframe:
(in code the field link is also dropped but this makes it hopefully more intelligible this way)
the rest is to just join the uri_id with the lang_id,vector,content rows that are not null which is trivial.
I hope this makes it more readable. If there is an aws service that makes it easier for me to deal with the data, since it is basically "just" database operations I'm also happy to hear about it.
On Mon, Nov 27, 2017 at 12:32 PM, Gourav Sengupta <[hidden email]> wrote:
Hi,I think that I have mentioned all the required alternatives. However I am quite curious as to how did you conclude that processing using EMR is going to be more expensive than using any other stack. I have been using EMR since last 6 years (almost about the time it came out), and have always found it cheap, reliable, safe and stable (ofcourse its like fire, if you are not careful it can end up burning you financially).
On Mon, Nov 27, 2017 at 12:58 PM, Alexander Czech <[hidden email]> wrote:
Simply because you have to pay on top of every instance hour. I currently need about 4800h of r3.2xlarge EMR takes 0.18$ instance hour so it would be 864$ just in EMR costs (spot prices are around 0.12$/h).Just to stay on topic I thought about getting 40 i2.xlarge instances which have about 1TB of combined ram and 32TB of combined SSD space would this be enough to load a 10TB parquet or do I need more RAM/Disk spill space?
On Mon, Nov 27, 2017 at 6:06 PM, Gourav Sengupta <[hidden email]> wrote:
10 TB in Athena would cost $50. If your data is in Parquet, then it will cost even less because of columnar striping. So I am genuinely not quite sure what you are speaking about? Also what do you mean by "I currently need"? Are you already processing the data?
Since you mentioned that you are a student, may I please ask which University and College you are studying at? I may provide some additional information regarding the same.
On Mon, Nov 27, 2017 at 7:55 PM, Alexander Czech <[hidden email]> wrote:
|Free forum by Nabble||Edit this page|