How to populate all possible combination values in columns using Spark SQL

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to populate all possible combination values in columns using Spark SQL

Aakash Basu-2
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: How to populate all possible combination values in columns using Spark SQL

Sonal Goyal
As mentioned in the comments on SO, can you provide a (masked) sample of the data? It will be easier to see what you are trying to do if you add the year column

Thanks,
Sonal
Nube Technologies 






On Thu, May 7, 2020 at 10:26 AM Aakash Basu <[hidden email]> wrote:
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: How to populate all possible combination values in columns using Spark SQL

Aakash Basu-2
Hi,

I've updated the SO question with masked data, added year column and other requirement. Please take a look.

Hope this helps in solving the problem.

Thanks and regards,
AB

On Thu 7 May, 2020, 10:59 AM Sonal Goyal, <[hidden email]> wrote:
As mentioned in the comments on SO, can you provide a (masked) sample of the data? It will be easier to see what you are trying to do if you add the year column

Thanks,
Sonal
Nube Technologies 






On Thu, May 7, 2020 at 10:26 AM Aakash Basu <[hidden email]> wrote:
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: How to populate all possible combination values in columns using Spark SQL

Edgardo Szrajber
Have you checked the pivot function?
Bentzi


On Thu, May 7, 2020 at 22:46, Aakash Basu
Hi,

I've updated the SO question with masked data, added year column and other requirement. Please take a look.

Hope this helps in solving the problem.

Thanks and regards,
AB

On Thu 7 May, 2020, 10:59 AM Sonal Goyal, <[hidden email]> wrote:
As mentioned in the comments on SO, can you provide a (masked) sample of the data? It will be easier to see what you are trying to do if you add the year column

Thanks,
Sonal
Nube Technologies 






On Thu, May 7, 2020 at 10:26 AM Aakash Basu <[hidden email]> wrote:
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: How to populate all possible combination values in columns using Spark SQL

Aakash Basu-2
I know how to pivot, but how to aggregate and pivot and build those True and False combination is the doubt.

On Fri 8 May, 2020, 1:31 PM Edgardo Szrajber, <[hidden email]> wrote:
Have you checked the pivot function?
Bentzi


On Thu, May 7, 2020 at 22:46, Aakash Basu
Hi,

I've updated the SO question with masked data, added year column and other requirement. Please take a look.

Hope this helps in solving the problem.

Thanks and regards,
AB

On Thu 7 May, 2020, 10:59 AM Sonal Goyal, <[hidden email]> wrote:
As mentioned in the comments on SO, can you provide a (masked) sample of the data? It will be easier to see what you are trying to do if you add the year column

Thanks,
Sonal
Nube Technologies 






On Thu, May 7, 2020 at 10:26 AM Aakash Basu <[hidden email]> wrote:
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: How to populate all possible combination values in columns using Spark SQL

Edgardo Szrajber
Once you pivot you can use CASE WHEN to calculate the True/false, then you can agregate.
Bentzi


On Sat, May 9, 2020 at 21:18, Aakash Basu
I know how to pivot, but how to aggregate and pivot and build those True and False combination is the doubt.

On Fri 8 May, 2020, 1:31 PM Edgardo Szrajber, <[hidden email]> wrote:
Have you checked the pivot function?
Bentzi


On Thu, May 7, 2020 at 22:46, Aakash Basu
Hi,

I've updated the SO question with masked data, added year column and other requirement. Please take a look.

Hope this helps in solving the problem.

Thanks and regards,
AB

On Thu 7 May, 2020, 10:59 AM Sonal Goyal, <[hidden email]> wrote:
As mentioned in the comments on SO, can you provide a (masked) sample of the data? It will be easier to see what you are trying to do if you add the year column

Thanks,
Sonal
Nube Technologies 






On Thu, May 7, 2020 at 10:26 AM Aakash Basu <[hidden email]> wrote:
Hi,

I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?


I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query.

Thanks,
Aakash.