no stdout output from worker

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

no stdout output from worker

Sen, Ranjan [USA]
Hi 
I have some System.out.println in my Java code that is working ok in a local environment. But when I run the same code on a standalone  mode in a EC2 cluster I do not see them at the worker stdout (in the worker node under <spark location>/work ) or at the driver console. Could you help me understand how do I troubleshoot?

Thanks 
Ranjan
Reply | Threaded
Open this post in threaded view
|

Re: no stdout output from worker

Patrick Wendell
Hey Sen,

Is your code in the driver code or inside one of the tasks?

If it's in the tasks, the place you would expect these to be is in
stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
at least stderr logs in that folder? If not then the tasks might not
be running on the workers machines. If you see stderr but not stdout
that's a bit of a puzzler since they both go through the same
mechanism.

- Patrick

On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]> wrote:
> Hi
> I have some System.out.println in my Java code that is working ok in a local
> environment. But when I run the same code on a standalone  mode in a EC2
> cluster I do not see them at the worker stdout (in the worker node under
> <spark location>/work ) or at the driver console. Could you help me
> understand how do I troubleshoot?
>
> Thanks
> Ranjan
Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: no stdout output from worker

Sen, Ranjan [USA]
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-

Š

Static boolean done=false;
Š

Public static void main(..

..

JavaRDD<String> lines = ..

..

While (!done) {

..
While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)


If (Š) {
        Done = true;

        !! Also here some System.out.println  (C)
       
        Break;
}

Else {

        If (Š) {
               
                !! More System.out.println  (D)

               
                labs = labs.map(Š) ;

                }
        }

} // outer while

!! Even more System.out.println  (E)

} // main

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.

Ranjan

On 3/9/14, 10:50 PM, "Patrick Wendell" <[hidden email]> wrote:

>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan

Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: no stdout output from worker

Sen, Ranjan [USA]
Just a correction - what the strange symbol which was auto generated is
for   “. . . . “

Thanks again,
Ranjan

On 3/9/14, 11:33 PM, "Sen, Ranjan [USA]" <[hidden email]> wrote:

>Hi Patrick
>
>How do I know which part of the code is in the driver and which in task?
>The structure of my code is as below-
>

>
>Static boolean done=false;

>
>Public static void main(..
>
>..
>
>JavaRDD<String> lines = ..
>
>..
>
>While (!done) {
>
>..
>While (..) {
>
>JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š
>);
>
>!! Here I have System.out.println (A)
>
>} // inner while
>
>!! Here I have System.out.println (B)
>
>
>If (Š) {
> Done = true;
>
> !! Also here some System.out.println  (C)
>
> Break;
>}
>
>Else {
>
> If (Š) {
>
> !! More System.out.println  (D)
>
>
> labs = labs.map(Š) ;
>
> }
> }
>
>} // outer while
>
>!! Even more System.out.println  (E)
>
>} // main
>
>} //class
>
>I get the console outputs on the master for (B) and (E). I do not see any
>stdout in the worker node. I find the stdout and stderr in the
><spark>/work/<appid>/0/. I see output
>in stderr but not in stdout.
>
>I do get all the outputs on the console when I run it in local mode.
>
>Sorry I am new and may be asking some naïve question but it is really
>confusing to me. Thanks for your help.
>
>Ranjan
>
>On 3/9/14, 10:50 PM, "Patrick Wendell" <[hidden email]> wrote:
>
>>Hey Sen,
>>
>>Is your code in the driver code or inside one of the tasks?
>>
>>If it's in the tasks, the place you would expect these to be is in
>>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>>at least stderr logs in that folder? If not then the tasks might not
>>be running on the workers machines. If you see stderr but not stdout
>>that's a bit of a puzzler since they both go through the same
>>mechanism.
>>
>>- Patrick
>>
>>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]>
>>wrote:
>>> Hi
>>> I have some System.out.println in my Java code that is working ok in a
>>>local
>>> environment. But when I run the same code on a standalone  mode in a
>>>EC2
>>> cluster I do not see them at the worker stdout (in the worker node
>>>under
>>> <spark location>/work ) or at the driver console. Could you help me
>>> understand how do I troubleshoot?
>>>
>>> Thanks
>>> Ranjan
>

Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: no stdout output from worker

Sourav Chandra
In reply to this post by Sen, Ranjan [USA]
Hi Ranjan,

Whatever code is being passed as closure to spark operations like map, flatmap, filter etc are part of task

All others are in driver.

Thanks,
Sourav


On Mon, Mar 10, 2014 at 12:03 PM, Sen, Ranjan [USA] <[hidden email]> wrote:
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-

Š

Static boolean done=false;
Š

Public static void main(..

..

JavaRDD<String> lines = ..

..

While (!done) {

..
While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)


If (Š) {
        Done = true;

        !! Also here some System.out.println  (C)

        Break;
}

Else {

        If (Š) {

                !! More System.out.println  (D)


                labs = labs.map(Š) ;

                }
        }

} // outer while

!! Even more System.out.println  (E)

} // main

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.

Ranjan

On 3/9/14, 10:50 PM, "Patrick Wendell" <[hidden email]> wrote:

>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan




--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: no stdout output from worker

Sen, Ranjan [USA]
Hi Sourav
That makes so much sense. Thanks much. 
Ranjan

From: Sourav Chandra <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Sunday, March 9, 2014 at 10:37 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: [External] Re: no stdout output from worker

Hi Ranjan,

Whatever code is being passed as closure to spark operations like map, flatmap, filter etc are part of task

All others are in driver.

Thanks,
Sourav


On Mon, Mar 10, 2014 at 12:03 PM, Sen, Ranjan [USA] <[hidden email]> wrote:
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-

Š

Static boolean done=false;
Š

Public static void main(..

..

JavaRDD<String> lines = ..

..

While (!done) {

..
While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)


If (Š) {
        Done = true;

        !! Also here some System.out.println  (C)

        Break;
}

Else {

        If (Š) {

                !! More System.out.println  (D)


                labs = labs.map(Š) ;

                }
        }

} // outer while

!! Even more System.out.println  (E)

} // main

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.

Ranjan

On 3/9/14, 10:50 PM, "Patrick Wendell" <[hidden email]> wrote:

>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan




--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: no stdout output from worker

Patrick Wendell
Hey Sen,

Suarav is right, and I think all of your print statements are inside of the driver program rather than inside of a closure. How are you running your program (i.e. what do you run that starts this job)? Where you run the driver you should expect to see the output.

- Patrick


On Mon, Mar 10, 2014 at 8:56 AM, Sen, Ranjan [USA] <[hidden email]> wrote:
Hi Sourav
That makes so much sense. Thanks much. 
Ranjan

From: Sourav Chandra <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Sunday, March 9, 2014 at 10:37 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: [External] Re: no stdout output from worker

Hi Ranjan,

Whatever code is being passed as closure to spark operations like map, flatmap, filter etc are part of task

All others are in driver.

Thanks,
Sourav


On Mon, Mar 10, 2014 at 12:03 PM, Sen, Ranjan [USA] <[hidden email]> wrote:
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-

Š

Static boolean done=false;
Š

Public static void main(..

..

JavaRDD<String> lines = ..

..

While (!done) {

..
While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)


If (Š) {
        Done = true;

        !! Also here some System.out.println  (C)

        Break;
}

Else {

        If (Š) {

                !! More System.out.println  (D)


                labs = labs.map(Š) ;

                }
        }

} // outer while

!! Even more System.out.println  (E)

} // main

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.

Ranjan

On 3/9/14, 10:50 PM, "Patrick Wendell" <[hidden email]> wrote:

>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[hidden email]>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan




--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com