How do you hit breakpoints using IntelliJ In functions used by an RDD

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How do you hit breakpoints using IntelliJ In functions used by an RDD

lordjoe
I was able to get JavaWordCount running with a local instance under IntelliJ.

In order to do so I needed to use maven to package my code and 
call
   String[] jars = { "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
    sparkConf.setJars(jars);

After that the sample ran properly and in the debugger I could set break points in the main.

However when I do 
something like 
   JavaRDD<String> words = lines.flatMap( new WordsMapFunction());

where WordsMapFunction is a separate class like

 public static class WordsMapFunction implements FlatMapFunction<String, String> {
     private static final Pattern SPACE = Pattern.compile(" ");
     public Iterable<String> call(String s) {
        String[] split = SPACE.split(s);
        for (int i = 0; i < split.length; i++) {
            split[i] = toUpperCase(split[i]);
        }
        return Arrays.asList(split);
    }
}

Breakpoints set in WordsMapFunction  are never hit.

Most interesting functionality in the problems I am trying to solve if in the FlatMapFunction and the Function2 code and this is the functionality I will need to examine in more detail.

Has anyone figured out how to configure a project to hit breakpoints in these functions?? 
Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

sowen
flatMap() is a transformation only. Calling it by itself does nothing,
and it just describes the relationship between one RDD and another.
You should see it swing into action if you invoke an action, like
count(), on the words RDD.

On Mon, Aug 25, 2014 at 6:32 PM, Steve Lewis <[hidden email]> wrote:

> I was able to get JavaWordCount running with a local instance under
> IntelliJ.
>
> In order to do so I needed to use maven to package my code and
> call
>    String[] jars = {
> "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
>     sparkConf.setJars(jars);
>
> After that the sample ran properly and in the debugger I could set break
> points in the main.
>
> However when I do
> something like
>    JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
>
> where WordsMapFunction is a separate class like
>
>  public static class WordsMapFunction implements FlatMapFunction<String,
> String> {
>      private static final Pattern SPACE = Pattern.compile(" ");
>      public Iterable<String> call(String s) {
>         String[] split = SPACE.split(s);
>         for (int i = 0; i < split.length; i++) {
>             split[i] = toUpperCase(split[i]);
>         }
>         return Arrays.asList(split);
>     }
> }
>
> Breakpoints set in WordsMapFunction  are never hit.
>
> Most interesting functionality in the problems I am trying to solve if in
> the FlatMapFunction and the Function2 code and this is the functionality I
> will need to examine in more detail.
>
> Has anyone figured out how to configure a project to hit breakpoints in
> these functions??

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

lordjoe
????
That was not quite in English

 
My Flatmap code is shown below

I know the code is called since the answers are correct but would like to
put a break point in dropNonLetters to make sure that code works properly

I am running in the IntelliJ debugger but believe the code is executing on a Spark Worker.
I am not sure what magic Intellij uses to hook up a debugger to a worker but hope it is possib;e

public class WordsMapFunction implements FlatMapFunction<String, String> {

    private static final Pattern SPACE = Pattern.compile(" ");

    public Iterable<String> call(String s) {
        String[] split = SPACE.split(s);
        for (int i = 0; i < split.length; i++) {
            split[i] = regularizeString(split[i]);
        }
        return Arrays.asList(split);
    }

    public static String dropNonLetters(String s) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < s.length(); i++) {
            char c = s.charAt(i);
            if (Character.isLetter(c))
                sb.append(c);
        }

        return sb.toString();
    }


    public static String regularizeString(String inp) {
        inp = inp.trim();
        inp = inp.toUpperCase();
        return dropNonLetters(inp);
    }

}


On Mon, Aug 25, 2014 at 10:35 AM, Sean Owen <[hidden email]> wrote:
flatMap() is a transformation only. Calling it by itself does nothing,
and it just describes the relationship between one RDD and another.
You should see it swing into action if you invoke an action, like
count(), on the words RDD.

On Mon, Aug 25, 2014 at 6:32 PM, Steve Lewis <[hidden email]> wrote:
> I was able to get JavaWordCount running with a local instance under
> IntelliJ.
>
> In order to do so I needed to use maven to package my code and
> call
>    String[] jars = {
> "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
>     sparkConf.setJars(jars);
>
> After that the sample ran properly and in the debugger I could set break
> points in the main.
>
> However when I do
> something like
>    JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
>
> where WordsMapFunction is a separate class like
>
>  public static class WordsMapFunction implements FlatMapFunction<String,
> String> {
>      private static final Pattern SPACE = Pattern.compile(" ");
>      public Iterable<String> call(String s) {
>         String[] split = SPACE.split(s);
>         for (int i = 0; i < split.length; i++) {
>             split[i] = toUpperCase(split[i]);
>         }
>         return Arrays.asList(split);
>     }
> }
>
> Breakpoints set in WordsMapFunction  are never hit.
>
> Most interesting functionality in the problems I am trying to solve if in
> the FlatMapFunction and the Function2 code and this is the functionality I
> will need to examine in more detail.
>
> Has anyone figured out how to configure a project to hit breakpoints in
> these functions??



--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com

Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

sowen
PS from an offline exchange -- yes more is being called here, the rest
is the standard WordCount example.

The trick was to make sure the task executes locally, and calling
setMaster("local") on SparkConf in the example code does that. That
seems to work fine in IntelliJ for debugging this.

On Mon, Aug 25, 2014 at 6:41 PM, Steve Lewis <[hidden email]> wrote:

> ????
> That was not quite in English
>
>
> My Flatmap code is shown below
>
> I know the code is called since the answers are correct but would like to
> put a break point in dropNonLetters to make sure that code works properly
>
> I am running in the IntelliJ debugger but believe the code is executing on a
> Spark Worker.
> I am not sure what magic Intellij uses to hook up a debugger to a worker but
> hope it is possib;e
>
> public class WordsMapFunction implements FlatMapFunction<String, String> {
>
>     private static final Pattern SPACE = Pattern.compile(" ");
>
>     public Iterable<String> call(String s) {
>         String[] split = SPACE.split(s);
>         for (int i = 0; i < split.length; i++) {
>             split[i] = regularizeString(split[i]);
>         }
>         return Arrays.asList(split);
>     }
>
>     public static String dropNonLetters(String s) {
>         StringBuilder sb = new StringBuilder();
>         for (int i = 0; i < s.length(); i++) {
>             char c = s.charAt(i);
>             if (Character.isLetter(c))
>                 sb.append(c);
>         }
>
>         return sb.toString();
>     }
>
>
>     public static String regularizeString(String inp) {
>         inp = inp.trim();
>         inp = inp.toUpperCase();
>         return dropNonLetters(inp);
>     }
>
> }
>
>
> On Mon, Aug 25, 2014 at 10:35 AM, Sean Owen <[hidden email]> wrote:
>>
>> flatMap() is a transformation only. Calling it by itself does nothing,
>> and it just describes the relationship between one RDD and another.
>> You should see it swing into action if you invoke an action, like
>> count(), on the words RDD.
>>
>> On Mon, Aug 25, 2014 at 6:32 PM, Steve Lewis <[hidden email]>
>> wrote:
>> > I was able to get JavaWordCount running with a local instance under
>> > IntelliJ.
>> >
>> > In order to do so I needed to use maven to package my code and
>> > call
>> >    String[] jars = {
>> > "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
>> >     sparkConf.setJars(jars);
>> >
>> > After that the sample ran properly and in the debugger I could set break
>> > points in the main.
>> >
>> > However when I do
>> > something like
>> >    JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
>> >
>> > where WordsMapFunction is a separate class like
>> >
>> >  public static class WordsMapFunction implements FlatMapFunction<String,
>> > String> {
>> >      private static final Pattern SPACE = Pattern.compile(" ");
>> >      public Iterable<String> call(String s) {
>> >         String[] split = SPACE.split(s);
>> >         for (int i = 0; i < split.length; i++) {
>> >             split[i] = toUpperCase(split[i]);
>> >         }
>> >         return Arrays.asList(split);
>> >     }
>> > }
>> >
>> > Breakpoints set in WordsMapFunction  are never hit.
>> >
>> > Most interesting functionality in the problems I am trying to solve if
>> > in
>> > the FlatMapFunction and the Function2 code and this is the functionality
>> > I
>> > will need to examine in more detail.
>> >
>> > Has anyone figured out how to configure a project to hit breakpoints in
>> > these functions??
>
>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

Akhil
You need to run your app in localmode ( aka master=local[2]) to get it debugged locally. If you are running it on a cluster, then you can use the remote debugging feature.

For remote debugging, you need to pass the following:

-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n

 jvm options and configure your ide on that given port (4000) for remote debugging.

Thanks
Best Regards


On Tue, Aug 26, 2014 at 1:32 AM, Sean Owen <[hidden email]> wrote:
PS from an offline exchange -- yes more is being called here, the rest
is the standard WordCount example.

The trick was to make sure the task executes locally, and calling
setMaster("local") on SparkConf in the example code does that. That
seems to work fine in IntelliJ for debugging this.

On Mon, Aug 25, 2014 at 6:41 PM, Steve Lewis <[hidden email]> wrote:
> ????
> That was not quite in English
>
>
> My Flatmap code is shown below
>
> I know the code is called since the answers are correct but would like to
> put a break point in dropNonLetters to make sure that code works properly
>
> I am running in the IntelliJ debugger but believe the code is executing on a
> Spark Worker.
> I am not sure what magic Intellij uses to hook up a debugger to a worker but
> hope it is possib;e
>
> public class WordsMapFunction implements FlatMapFunction<String, String> {
>
>     private static final Pattern SPACE = Pattern.compile(" ");
>
>     public Iterable<String> call(String s) {
>         String[] split = SPACE.split(s);
>         for (int i = 0; i < split.length; i++) {
>             split[i] = regularizeString(split[i]);
>         }
>         return Arrays.asList(split);
>     }
>
>     public static String dropNonLetters(String s) {
>         StringBuilder sb = new StringBuilder();
>         for (int i = 0; i < s.length(); i++) {
>             char c = s.charAt(i);
>             if (Character.isLetter(c))
>                 sb.append(c);
>         }
>
>         return sb.toString();
>     }
>
>
>     public static String regularizeString(String inp) {
>         inp = inp.trim();
>         inp = inp.toUpperCase();
>         return dropNonLetters(inp);
>     }
>
> }
>
>
> On Mon, Aug 25, 2014 at 10:35 AM, Sean Owen <[hidden email]> wrote:
>>
>> flatMap() is a transformation only. Calling it by itself does nothing,
>> and it just describes the relationship between one RDD and another.
>> You should see it swing into action if you invoke an action, like
>> count(), on the words RDD.
>>
>> On Mon, Aug 25, 2014 at 6:32 PM, Steve Lewis <[hidden email]>
>> wrote:
>> > I was able to get JavaWordCount running with a local instance under
>> > IntelliJ.
>> >
>> > In order to do so I needed to use maven to package my code and
>> > call
>> >    String[] jars = {
>> > "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
>> >     sparkConf.setJars(jars);
>> >
>> > After that the sample ran properly and in the debugger I could set break
>> > points in the main.
>> >
>> > However when I do
>> > something like
>> >    JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
>> >
>> > where WordsMapFunction is a separate class like
>> >
>> >  public static class WordsMapFunction implements FlatMapFunction<String,
>> > String> {
>> >      private static final Pattern SPACE = Pattern.compile(" ");
>> >      public Iterable<String> call(String s) {
>> >         String[] split = SPACE.split(s);
>> >         for (int i = 0; i < split.length; i++) {
>> >             split[i] = toUpperCase(split[i]);
>> >         }
>> >         return Arrays.asList(split);
>> >     }
>> > }
>> >
>> > Breakpoints set in WordsMapFunction  are never hit.
>> >
>> > Most interesting functionality in the problems I am trying to solve if
>> > in
>> > the FlatMapFunction and the Function2 code and this is the functionality
>> > I
>> > will need to examine in more detail.
>> >
>> > Has anyone figured out how to configure a project to hit breakpoints in
>> > these functions??
>
>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

miaoiao
This post has NOT been accepted by the mailing list yet.
In reply to this post by sowen
Hi, I occurred this problem too , have you get the answer?
Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

scorpio
This post has NOT been accepted by the mailing list yet.
While submitting your spark job, add the below argument to the command line:
--conf spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

You can change port (i.e. 5005) as per the availability on your executor machines. 
Once the job starts, you may use intellij to connect to port 5005 using jdwp which is available under "Remote Machine" option

On Wed, May 31, 2017 at 10:55 PM, miaoiao [via Apache Spark User List] <[hidden email]> wrote:
Hi, I occurred this problem too , have you get the answer?


To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

scorpio
This post has NOT been accepted by the mailing list yet.
In reply to this post by miaoiao

While submitting your spark job, add the below argument to the command line:
--conf
spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

You can change port (i.e. 5005) as per the availability on your executor
machines.
Once the job starts, you may use intellij to connect to port 5005 using
jdwp which is available under "Remote Machine" option

-cheers
Sachin

On Fri, Jun 2, 2017 at 8:25 AM, sachin garg <[hidden email]> wrote:
While submitting your spark job, add the below argument to the command line:
--conf spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

You can change port (i.e. 5005) as per the availability on your executor machines. 
Once the job starts, you may use intellij to connect to port 5005 using jdwp which is available under "Remote Machine" option

On Wed, May 31, 2017 at 10:55 PM, miaoiao [via Apache Spark User List] <[hidden email]> wrote:
Hi, I occurred this problem too , have you get the answer?


To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML


Reply | Threaded
Open this post in threaded view
|

Re: How do you hit breakpoints using IntelliJ In functions used by an RDD

miaoiao
This post has NOT been accepted by the mailing list yet.
In reply to this post by scorpio
Thanks, your answer seems to be  remote debug ,I want to debug local ,and agentlib is automaticly set( check below picture1 )

picture 1

 and I've setMaster("local[*]"), but still , I cannot debug the function used by an RDD, my code is :

val prodRDD = tensorRDD.flatMap{ case( blockSubIndex, denseMatrix ) =>
      TensorWeightTucker.localMMMany( blockSubIndex, denseMatrix, 1, bcBasisMatrixArray(1), tensorInfo.blockRank,
        tensorInfo.blockNum, tensorInfo.blockFinal ) }


I set breakpoint inside TensorWeightTucker.localMMMany, and it cannot stop there.