Dataframe vs dataset

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Dataframe vs dataset

MidwestMike
Hi, 

I use Spark everyday and I have a good grip on the basics of Spark, so this question isnt for myself.  But this came up and I wanted to see what other Spark users would say, and I dont want to influence your answer.  And SO is weird about polls. The question is

 "Which one do you feel is accurate... Dataset is a subset of DataFrame, or DataFrame a subset of Dataset?"
Reply | Threaded
Open this post in threaded view
|

Re: Dataframe vs dataset

Marco Mistroni
Imho .neither..I see datasets as typed df and therefore ds are enhanced df
Feel free to disagree..
Kr

On Sat, Apr 28, 2018, 2:24 PM Michael Artz <[hidden email]> wrote:
Hi, 

I use Spark everyday and I have a good grip on the basics of Spark, so this question isnt for myself.  But this came up and I wanted to see what other Spark users would say, and I dont want to influence your answer.  And SO is weird about polls. The question is

 "Which one do you feel is accurate... Dataset is a subset of DataFrame, or DataFrame a subset of Dataset?"
Reply | Threaded
Open this post in threaded view
|

Re: Dataframe vs dataset

MidwestMike
Ok from the language you used, you are saying kind of that Dataset is a subset of Dataframe.  I would disagree because to me a DataFrame is just a Dataset of org.spache.spark.sql.Row

On Sat, Apr 28, 2018, 8:34 AM Marco Mistroni <[hidden email]> wrote:
Imho .neither..I see datasets as typed df and therefore ds are enhanced df
Feel free to disagree..
Kr

On Sat, Apr 28, 2018, 2:24 PM Michael Artz <[hidden email]> wrote:
Hi, 

I use Spark everyday and I have a good grip on the basics of Spark, so this question isnt for myself.  But this came up and I wanted to see what other Spark users would say, and I dont want to influence your answer.  And SO is weird about polls. The question is

 "Which one do you feel is accurate... Dataset is a subset of DataFrame, or DataFrame a subset of Dataset?"
Reply | Threaded
Open this post in threaded view
|

Re: Dataframe vs dataset

JayeshLalwani
In reply to this post by MidwestMike

Neither.

 

All women are humans. Not all humans are women. You wouldn’t say that a woman is a subset of a human.

 

All DataFrames are DataSets. Not all Datasets are DataFrames. The “subset” relationship doesn’t apply here.  A DataFrame is a specialized type of DataSet

 

From: Michael Artz <[hidden email]>
Date: Saturday, April 28, 2018 at 9:24 AM
To: "user @spark" <[hidden email]>
Subject: Dataframe vs dataset

 

Hi, 

 

I use Spark everyday and I have a good grip on the basics of Spark, so this question isnt for myself.  But this came up and I wanted to see what other Spark users would say, and I dont want to influence your answer.  And SO is weird about polls. The question is

 

 "Which one do you feel is accurate... Dataset is a subset of DataFrame, or DataFrame a subset of Dataset?"



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|

Re: Dataframe vs dataset

MidwestMike
I get your point haha and I also think of it as DataFrame being a specific kind of Dataset. 
Mike

On Tue, May 1, 2018, 7:27 AM Lalwani, Jayesh <[hidden email]> wrote:

Neither.

 

All women are humans. Not all humans are women. You wouldn’t say that a woman is a subset of a human.

 

All DataFrames are DataSets. Not all Datasets are DataFrames. The “subset” relationship doesn’t apply here.  A DataFrame is a specialized type of DataSet

 

From: Michael Artz <[hidden email]>
Date: Saturday, April 28, 2018 at 9:24 AM
To: "user @spark" <[hidden email]>
Subject: Dataframe vs dataset

 

Hi, 

 

I use Spark everyday and I have a good grip on the basics of Spark, so this question isnt for myself.  But this came up and I wanted to see what other Spark users would say, and I dont want to influence your answer.  And SO is weird about polls. The question is

 

 "Which one do you feel is accurate... Dataset is a subset of DataFrame, or DataFrame a subset of Dataset?"



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.