Compute the Hash of each row in new column

6 messages
Open this post in threaded view
|

Compute the Hash of each row in new column

 Hi Spark Users,How can I compute Hash of each row and store in new column at Dataframe, could someone help me.Thanks
Open this post in threaded view
|

Re: Compute the Hash of each row in new column

 Hi Chetan,Would the sql function `hash` do the trick for your use-case ?Best,On Fri, Feb 28, 2020 at 1:56 PM Chetan Khatri <[hidden email]> wrote:Hi Spark Users,How can I compute Hash of each row and store in new column at Dataframe, could someone help me.Thanks
Open this post in threaded view
|

Re: Compute the Hash of each row in new column

 In reply to this post by Chetan Khatri This computes the md5 hash of a given column id of Dataset ds: ds.withColumn("id hash", md5(\$"id")).show(false) Test with this Dataset ds: import org.apache.spark.sql.types._ val ds = spark.range(10).select(\$"id".cast(StringType)) Available are md5, sha, sha1, sha2 and hash: https://spark.apache.org/docs/2.4.5/api/sql/index.htmlEnrico Am 28.02.20 um 13:56 schrieb Chetan Khatri: > Hi Spark Users, > How can I compute Hash of each row and store in new column at > Dataframe, could someone help me. > > Thanks --------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Open this post in threaded view
|

Re: Compute the Hash of each row in new column

 Thanks EnricoI want to compute hash of all the columns value in the row.On Fri, Feb 28, 2020 at 7:28 PM Enrico Minack <[hidden email]> wrote:This computes the md5 hash of a given column id of Dataset ds: ds.withColumn("id hash", md5(\$"id")).show(false) Test with this Dataset ds: import org.apache.spark.sql.types._ val ds = spark.range(10).select(\$"id".cast(StringType)) Available are md5, sha, sha1, sha2 and hash: https://spark.apache.org/docs/2.4.5/api/sql/index.html Enrico Am 28.02.20 um 13:56 schrieb Chetan Khatri: > Hi Spark Users, > How can I compute Hash of each row and store in new column at > Dataframe, could someone help me. > > Thanks