unable to make a custom class as a key in a pairrdd

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

unable to make a custom class as a key in a pairrdd

Jaonary Rabarisoa
Hi all,

I have the following case class that I want to use as a key in a key-value rdd. I defined the equals and hashCode methode but it's not working. What I'm doing wrong ?

case class PersonID(id: String) {
     
     override def hashCode = id.hashCode
     
     override def equals(other: Any) = other match {
         
         case that: PersonID => this.id == that.id && this.getClass == that.getClass
         case _ => false
     }   
 }   
 
 
 val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))


p.groupByKey.collect foreach println

(PersonID(1),CompactBuffer(5))
(PersonID(1),CompactBuffer(6))
(PersonID(1),CompactBuffer(7))
(PersonID(1),CompactBuffer(8, 9))
(PersonID(1),CompactBuffer(1))
(PersonID(1),CompactBuffer(2))
(PersonID(1),CompactBuffer(3))
(PersonID(1),CompactBuffer(4))


Best,

Jao
Reply | Threaded
Open this post in threaded view
|

Re: unable to make a custom class as a key in a pairrdd

Niklas Wilcke
Hi Jao,

I don't really know why this doesn't work but I have two hints.
You don't need to override hashCode and equals. The modifier case is doing that for you. Writing

case class PersonID(id: String)

would be enough to get the class you want I think.
If I change the type of the id param to Int it works for me but I don't know why.

case class PersonID(id: Int)

Looks like a strange behavior to me. Have a try.

Good luck,
Niklas

On 23.10.2014 21:52, Jaonary Rabarisoa wrote:
Hi all,

I have the following case class that I want to use as a key in a key-value rdd. I defined the equals and hashCode methode but it's not working. What I'm doing wrong ?

case class PersonID(id: String) {
     
     override def hashCode = id.hashCode
     
     override def equals(other: Any) = other match {
         
         case that: PersonID => this.id == that.id && this.getClass == that.getClass
         case _ => false
     }   
 }   
 
 
 val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))


p.groupByKey.collect foreach println

(PersonID(1),CompactBuffer(5))
(PersonID(1),CompactBuffer(6))
(PersonID(1),CompactBuffer(7))
(PersonID(1),CompactBuffer(8, 9))
(PersonID(1),CompactBuffer(1))
(PersonID(1),CompactBuffer(2))
(PersonID(1),CompactBuffer(3))
(PersonID(1),CompactBuffer(4))


Best,

Jao

Reply | Threaded
Open this post in threaded view
|

Re: unable to make a custom class as a key in a pairrdd

Prashant Sharma
Are you doing this in REPL ? Then there is a bug filed for this, I just can't recall the bug ID at the moment.

Prashant Sharma



On Fri, Oct 24, 2014 at 4:07 AM, Niklas Wilcke <[hidden email]> wrote:
Hi Jao,

I don't really know why this doesn't work but I have two hints.
You don't need to override hashCode and equals. The modifier case is doing that for you. Writing

case class PersonID(id: String)

would be enough to get the class you want I think.
If I change the type of the id param to Int it works for me but I don't know why.

case class PersonID(id: Int)

Looks like a strange behavior to me. Have a try.

Good luck,
Niklas


On 23.10.2014 21:52, Jaonary Rabarisoa wrote:
Hi all,

I have the following case class that I want to use as a key in a key-value rdd. I defined the equals and hashCode methode but it's not working. What I'm doing wrong ?

case class PersonID(id: String) {
     
     override def hashCode = id.hashCode
     
     override def equals(other: Any) = other match {
         
         case that: PersonID => this.id == that.id && this.getClass == that.getClass
         case _ => false
     }   
 }   
 
 
 val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))


p.groupByKey.collect foreach println

(PersonID(1),CompactBuffer(5))
(PersonID(1),CompactBuffer(6))
(PersonID(1),CompactBuffer(7))
(PersonID(1),CompactBuffer(8, 9))
(PersonID(1),CompactBuffer(1))
(PersonID(1),CompactBuffer(2))
(PersonID(1),CompactBuffer(3))
(PersonID(1),CompactBuffer(4))


Best,

Jao


Reply | Threaded
Open this post in threaded view
|

Re: unable to make a custom class as a key in a pairrdd

Jaonary Rabarisoa
In the documentation it's said that we need to override the hashCode and equals methods. Without overriding it does't work too. I get this error on REPL and stand alone application

On Fri, Oct 24, 2014 at 3:29 AM, Prashant Sharma <[hidden email]> wrote:
Are you doing this in REPL ? Then there is a bug filed for this, I just can't recall the bug ID at the moment.

Prashant Sharma



On Fri, Oct 24, 2014 at 4:07 AM, Niklas Wilcke <[hidden email]> wrote:
Hi Jao,

I don't really know why this doesn't work but I have two hints.
You don't need to override hashCode and equals. The modifier case is doing that for you. Writing

case class PersonID(id: String)

would be enough to get the class you want I think.
If I change the type of the id param to Int it works for me but I don't know why.

case class PersonID(id: Int)

Looks like a strange behavior to me. Have a try.

Good luck,
Niklas


On 23.10.2014 21:52, Jaonary Rabarisoa wrote:
Hi all,

I have the following case class that I want to use as a key in a key-value rdd. I defined the equals and hashCode methode but it's not working. What I'm doing wrong ?

case class PersonID(id: String) {
     
     override def hashCode = id.hashCode
     
     override def equals(other: Any) = other match {
         
         case that: PersonID => this.id == that.id && this.getClass == that.getClass
         case _ => false
     }   
 }   
 
 
 val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))


p.groupByKey.collect foreach println

(PersonID(1),CompactBuffer(5))
(PersonID(1),CompactBuffer(6))
(PersonID(1),CompactBuffer(7))
(PersonID(1),CompactBuffer(8, 9))
(PersonID(1),CompactBuffer(1))
(PersonID(1),CompactBuffer(2))
(PersonID(1),CompactBuffer(3))
(PersonID(1),CompactBuffer(4))


Best,

Jao



Reply | Threaded
Open this post in threaded view
|

Re: unable to make a custom class as a key in a pairrdd

maasg
There's an issue in the way case classes are handled on the REPL and you won't be able to use a case class as a key. See: https://issues.apache.org/jira/browse/SPARK-2620

BTW, case classes already implement equals and hashCode. It's not needed to implement those again.

Given that you already implement equals and hashCode, try just dropping "case"  and create a normal class. It might work that way.

-kr, Gerard.

On Fri, Oct 24, 2014 at 11:20 AM, Jaonary Rabarisoa <[hidden email]> wrote:
In the documentation it's said that we need to override the hashCode and equals methods. Without overriding it does't work too. I get this error on REPL and stand alone application

On Fri, Oct 24, 2014 at 3:29 AM, Prashant Sharma <[hidden email]> wrote:
Are you doing this in REPL ? Then there is a bug filed for this, I just can't recall the bug ID at the moment.

Prashant Sharma



On Fri, Oct 24, 2014 at 4:07 AM, Niklas Wilcke <[hidden email]> wrote:
Hi Jao,

I don't really know why this doesn't work but I have two hints.
You don't need to override hashCode and equals. The modifier case is doing that for you. Writing

case class PersonID(id: String)

would be enough to get the class you want I think.
If I change the type of the id param to Int it works for me but I don't know why.

case class PersonID(id: Int)

Looks like a strange behavior to me. Have a try.

Good luck,
Niklas


On 23.10.2014 21:52, Jaonary Rabarisoa wrote:
Hi all,

I have the following case class that I want to use as a key in a key-value rdd. I defined the equals and hashCode methode but it's not working. What I'm doing wrong ?

case class PersonID(id: String) {
     
     override def hashCode = id.hashCode
     
     override def equals(other: Any) = other match {
         
         case that: PersonID => this.id == that.id && this.getClass == that.getClass
         case _ => false
     }   
 }   
 
 
 val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))


p.groupByKey.collect foreach println

(PersonID(1),CompactBuffer(5))
(PersonID(1),CompactBuffer(6))
(PersonID(1),CompactBuffer(7))
(PersonID(1),CompactBuffer(8, 9))
(PersonID(1),CompactBuffer(1))
(PersonID(1),CompactBuffer(2))
(PersonID(1),CompactBuffer(3))
(PersonID(1),CompactBuffer(4))


Best,

Jao