PIG to Spark

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

PIG to Spark

Pralabh Kumar
Hi 

Is there a convenient way /open source project to convert PIG scripts to Spark. 


Regards
Pralabh Kumar
Reply | Threaded
Open this post in threaded view
|

Re: PIG to Spark

zjffdu

Pig support spark engine now, so you can leverage spark execution with pig script. 

I am afraid there's no solution to convert pig script to spark api code





Pralabh Kumar <[hidden email]>于2018年1月8日周一 下午11:25写道:
Hi 

Is there a convenient way /open source project to convert PIG scripts to Spark. 


Regards
Pralabh Kumar
Reply | Threaded
Open this post in threaded view
|

Re: PIG to Spark

Gourav Sengupta
In reply to this post by Pralabh Kumar
I may be wrong here, but when I see github and apache pig, it says that there are 8 contributors, and when I see github and look at apache spark it says there are more than 1000 contributors. And if the above is true I ask myself, why not shift to SPARK by learning it?

I also started with map reduce JAVA nightmarish coding, and then HIVE and then Pig and then realised the best time spent is the time used for actually solving data problems than programming problems. I know consultants who have ended up convincing their clients that its better to write JAVA programs than use SPARK SQL and then have spent close to 2.5 years not being able to deliver anything that works where the actual project was just a single SPARK SQL. 

Personally I prefer to learn and adapt and transfer existing code to a platform that gives me the maximum business benefit with least headaches.

But once again it is just a matter of opinion.


Regards,
Gourav Sengupta

On Mon, Jan 8, 2018 at 3:25 PM, Pralabh Kumar <[hidden email]> wrote:
Hi 

Is there a convenient way /open source project to convert PIG scripts to Spark. 


Regards
Pralabh Kumar