Nested Case Classes (Found and Required Same)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Nested Case Classes (Found and Required Same)

iramaraju
I think this is a popular issue, but need help figuring a way around if this issue is unresolved. I have a dataset that has more than 70 columns. To have all the columns fit into my RDD, I am experimenting the following. (I intend to use the InputData to parse the file and have 3 or 4 columnsets to accommodate the full list of variables)

case class ColumnSet(C1: Double , C2: Double , C3: Double)
case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2 :ColumnSet)

val  set1 = ColumnSet(1,2,3)
val a = InputData("a","a",set1,set1)

returns the following

<console>:16: error: type mismatch;
 found   : ColumnSet
 required: ColumnSet
       val a = InputData("a","a",set1,set1)

Where as the same code works fine in my scala console.

Is there a work around for my problem ?

Regards
Ram
Reply | Threaded
Open this post in threaded view
|

Re: Nested Case Classes (Found and Required Same)

iramaraju
This is only a problem in shell, but works fine in batch mode though. I am also interested in how others are solving the problem of case class limitation on number of variables.

Regards
Ram

On Fri, Sep 12, 2014 at 12:12 PM, iramaraju <[hidden email]> wrote:
I think this is a popular issue, but need help figuring a way around if this
issue is unresolved. I have a dataset that has more than 70 columns. To have
all the columns fit into my RDD, I am experimenting the following. (I intend
to use the InputData to parse the file and have 3 or 4 columnsets to
accommodate the full list of variables)

case class ColumnSet(C1: Double , C2: Double , C3: Double)
case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2
:ColumnSet)

val  set1 = ColumnSet(1,2,3)
val a = InputData("a","a",set1,set1)

returns the following

<console>:16: error: type mismatch;
 found   : ColumnSet
 required: ColumnSet
       val a = InputData("a","a",set1,set1)

Where as the same code works fine in my scala console.

Is there a work around for my problem ?

Regards
Ram



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Case-Classes-Found-and-Required-Same-tp14096.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




--
------------------------------------------
Ramaraju Indukuri
Reply | Threaded
Open this post in threaded view
|

Re: Nested Case Classes (Found and Required Same)

Prashant Sharma
What is your spark version ?  This was fixed I suppose. Can you try it with latest release ? 

Prashant Sharma



On Fri, Sep 12, 2014 at 9:47 PM, Ramaraju Indukuri <[hidden email]> wrote:
This is only a problem in shell, but works fine in batch mode though. I am also interested in how others are solving the problem of case class limitation on number of variables.

Regards
Ram

On Fri, Sep 12, 2014 at 12:12 PM, iramaraju <[hidden email]> wrote:
I think this is a popular issue, but need help figuring a way around if this
issue is unresolved. I have a dataset that has more than 70 columns. To have
all the columns fit into my RDD, I am experimenting the following. (I intend
to use the InputData to parse the file and have 3 or 4 columnsets to
accommodate the full list of variables)

case class ColumnSet(C1: Double , C2: Double , C3: Double)
case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2
:ColumnSet)

val  set1 = ColumnSet(1,2,3)
val a = InputData("a","a",set1,set1)

returns the following

<console>:16: error: type mismatch;
 found   : ColumnSet
 required: ColumnSet
       val a = InputData("a","a",set1,set1)

Where as the same code works fine in my scala console.

Is there a work around for my problem ?

Regards
Ram



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Case-Classes-Found-and-Required-Same-tp14096.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




--
------------------------------------------
Ramaraju Indukuri

Reply | Threaded
Open this post in threaded view
|

Re: Nested Case Classes (Found and Required Same)

iramaraju
Upgraded to 1.1 and the issue is resolved. Thanks.

I still wonder if there is a better way to approach a large attribute dataset. 

On Fri, Sep 12, 2014 at 12:20 PM, Prashant Sharma <[hidden email]> wrote:
What is your spark version ?  This was fixed I suppose. Can you try it with latest release ? 

Prashant Sharma



On Fri, Sep 12, 2014 at 9:47 PM, Ramaraju Indukuri <[hidden email]> wrote:
This is only a problem in shell, but works fine in batch mode though. I am also interested in how others are solving the problem of case class limitation on number of variables.

Regards
Ram

On Fri, Sep 12, 2014 at 12:12 PM, iramaraju <[hidden email]> wrote:
I think this is a popular issue, but need help figuring a way around if this
issue is unresolved. I have a dataset that has more than 70 columns. To have
all the columns fit into my RDD, I am experimenting the following. (I intend
to use the InputData to parse the file and have 3 or 4 columnsets to
accommodate the full list of variables)

case class ColumnSet(C1: Double , C2: Double , C3: Double)
case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2
:ColumnSet)

val  set1 = ColumnSet(1,2,3)
val a = InputData("a","a",set1,set1)

returns the following

<console>:16: error: type mismatch;
 found   : ColumnSet
 required: ColumnSet
       val a = InputData("a","a",set1,set1)

Where as the same code works fine in my scala console.

Is there a work around for my problem ?

Regards
Ram



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Case-Classes-Found-and-Required-Same-tp14096.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




--
------------------------------------------
Ramaraju Indukuri




--
------------------------------------------
Ramaraju Indukuri
Reply | Threaded
Open this post in threaded view
|

Re: Nested Case Classes (Found and Required Same)

Bojan Kostic
This post has NOT been accepted by the mailing list yet.
Did you find any other way for this issue?
I just found out that i have >22 columns data set... And now i am searching for best solution.

Anyone else have experienced with this problem?

Best
Bojan