Query on Profiling Spark Code

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Query on Profiling Spark Code

Aakash Basu-2
Hi guys,

I'm trying to profile my Spark code on cProfiler and check where more time is taken. I found the most time taken is by some socket object, which I'm quite clueless of, as to where it is used.

Can anyone shed some light on this?




Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

RE: Query on Profiling Spark Code

Patil, Prashasth

Hi Aakash,

On a related note, you may want to try SparkLens for profiling which is quite helpful in my opinion.

 

 

-Prash

 

From: Aakash Basu [mailto:[hidden email]]
Sent: Tuesday, July 17, 2018 12:41 PM
To: user
Subject: Query on Profiling Spark Code

 

Hi guys,

 

I'm trying to profile my Spark code on cProfiler and check where more time is taken. I found the most time taken is by some socket object, which I'm quite clueless of, as to where it is used.

 

Can anyone shed some light on this?

 

 

ncalls

tottime

percall

cumtime

percall

filename:lineno(function)

11789

479.8

0.0407

479.8

0.0407

~:0(<method 'recv_into' of '_socket.socket' objects>)

 

 

Thanks,

Aakash.




The information contained in this message is intended only for the recipient, and may be a confidential attorney-client communication or may otherwise be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, please be aware that any dissemination or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the message and deleting it from your computer. S&P Global Inc. reserves the right, subject to applicable local law, to monitor, review and process the content of any electronic message or information sent to or from S&P Global Inc. e-mail addresses without informing the sender or recipient of the message. By sending electronic message or information to S&P Global Inc. e-mail addresses you, as the sender, are consenting to S&P Global Inc. processing any of your personal data therein.
Reply | Threaded
Open this post in threaded view
|

Re: Query on Profiling Spark Code

Aakash Basu-2
Okay, sure!

On Tue, Jul 31, 2018 at 1:06 PM, Patil, Prashasth <[hidden email]> wrote:

Hi Aakash,

On a related note, you may want to try SparkLens for profiling which is quite helpful in my opinion.

 

 

-Prash

 

From: Aakash Basu [mailto:[hidden email]]
Sent: Tuesday, July 17, 2018 12:41 PM
To: user
Subject: Query on Profiling Spark Code

 

Hi guys,

 

I'm trying to profile my Spark code on cProfiler and check where more time is taken. I found the most time taken is by some socket object, which I'm quite clueless of, as to where it is used.

 

Can anyone shed some light on this?

 

 

ncalls

tottime

percall

cumtime

percall

filename:lineno(function)

11789

479.8

0.0407

479.8

0.0407

~:0(<method 'recv_into' of '_socket.socket' objects>)

 

 

Thanks,

Aakash.




The information contained in this message is intended only for the recipient, and may be a confidential attorney-client communication or may otherwise be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, please be aware that any dissemination or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the message and deleting it from your computer. S&P Global Inc. reserves the right, subject to applicable local law, to monitor, review and process the content of any electronic message or information sent to or from S&P Global Inc. e-mail addresses without informing the sender or recipient of the message. By sending electronic message or information to S&P Global Inc. e-mail addresses you, as the sender, are consenting to S&P Global Inc. processing any of your personal data therein.