Query execution in spark

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Query execution in spark

Ravi Hemnani
Hey,

Can anyone explain how the job basically runs in spark?

The number of mapper, reducers, the tmp files created and which tmp file contains what data and how to set the number of reducer tasks as we do in hadoop.

This would prove to be a big help. Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: Query execution in spark

Andrew Ash
Hi Ravi,

Have you read through the docs?  I'm not sure there's a page that directly answers your question but this one gives you an overview of the cluster.


Andrew


On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <[hidden email]> wrote:
Hey,

Can anyone explain how the job basically runs in spark?

The number of mapper, reducers, the tmp files created and which tmp file
contains what data and how to set the number of reducer tasks as we do in
hadoop.

This would prove to be a big help. Thank you.





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Query execution in spark

Mayur Rustagi
I have found this quite useful 




On Tue, Feb 11, 2014 at 10:16 AM, Andrew Ash <[hidden email]> wrote:
Hi Ravi,

Have you read through the docs?  I'm not sure there's a page that directly answers your question but this one gives you an overview of the cluster.


Andrew


On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <[hidden email]> wrote:
Hey,

Can anyone explain how the job basically runs in spark?

The number of mapper, reducers, the tmp files created and which tmp file
contains what data and how to set the number of reducer tasks as we do in
hadoop.

This would prove to be a big help. Thank you.





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Query execution in spark

Ognen Duzlevski-2
On 2/11/14, 12:57 PM, Mayur Rustagi wrote:
> I have found this quite useful
> http://www.youtube.com/watch?v=49Hr5xZyTEA

Thank you!
Ognen
Reply | Threaded
Open this post in threaded view
|

Re: Query execution in spark

Ravi Hemnani
In reply to this post by Mayur Rustagi
On 02/12/2014 12:29 AM, Mayur Rustagi [via Apache Spark User List] wrote:
I have found this quite useful 


Mayur Rustagi
Ph: +919632149971


On Tue, Feb 11, 2014 at 10:16 AM, Andrew Ash <[hidden email]> wrote:
Hi Ravi,

Have you read through the docs?  I'm not sure there's a page that directly answers your question but this one gives you an overview of the cluster.


Andrew


On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <[hidden email]> wrote:
Hey,

Can anyone explain how the job basically runs in spark?

The number of mapper, reducers, the tmp files created and which tmp file
contains what data and how to set the number of reducer tasks as we do in
hadoop.

This would prove to be a big help. Thank you.





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.





If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390p1405.html
To unsubscribe from Query execution in spark, click here.
NAML
I was watching the same video and it made me more curious about various things.
Currently i am working on what things i need to look into for optimization of my spark cluster. Ill share my notes with you soon. :)