Fwd: Dose pyspark supports python3.6?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Dose pyspark supports python3.6?

shizihao123
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

RE: Dose pyspark supports python3.6?

van den Heever, Christian CC

Dear Spark users

 

I have been asked to provide a presentation / business case as to why to use spark and java as ingestion tool for HDFS and HIVE

And why to move away from an etl tool.

 

Could you be so kind as to provide with some pros and cons to this.

 

I have the following :

 

Pros:

In house build – code can be changes on the fly to suite business need.

Software is free

Can out of the box run on all nodes

Will support all Apache based software.

Fast deu to in memory processing

Spark UI can visualise execution

Support checkpoint data loads

Support echama regesty for custom schema and inference.

Support Yarn execution

Mlibs can be used in need.

Data linage support deu to spar usage.

 

Cons

Skills needed to maintain and build

In memory cabibility can become bottleneck if not managed

No ETL gui.

 

Maybe point be to an article if you have one.

 

Thanks a mill.

Christian

 



Standard Bank email disclaimer and confidentiality note
Please go to www.standardbank.co.za/site/homepage/emaildisclaimer.html to read our email disclaimer and confidentiality note. Kindly email [hidden email] (no content or subject line necessary) if you cannot view that page and we will email our email disclaimer and confidentiality note to you.


Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Dose pyspark supports python3.6?

tokoma
In reply to this post by shizihao123
I'm not sure whether pyspark supports python 3.6 but  pyspark and python 3.6 is working on my environment.

I found the following issue and it seems to be already resolved.


2017/11/02 午前11:54 "Jun Shi" <[hidden email]>:


Dear spark developers:
       It’s so exciting to send this email to you.
       I have encountered the problem that if pyspark supports python3.6? (I found some answer online is no.) Can you tell me the answer which  python versions does pyspark support?
       I’m looking forward for your answer. Thank you very much!

Best,
Jun


Reply | Threaded
Open this post in threaded view
|

Spark as ETL, was: Re: Dose pyspark supports python3.6?

Jean Georges Perrin
In reply to this post by van den Heever, Christian CC
Pros:
No need for Scala skills, Java can be used.
Other companies are already doing it.
> Support Yarn execution
But not only…
Complex use-case for import can easily be done in Java (see https://spark-summit.org/eu-2017/events/extending-apache-sparks-ingestion-building-your-own-java-data-source/ - sorry shameless self promo).
Can be parallelized on all nodes of the cluster

Cons:
> No ETL gui.
But works very well with libraries like log4j so we can track the process

His

jg

On Nov 1, 2017, at 22:58, van den Heever, Christian CC <[hidden email]> wrote:

Dear Spark users 
 
I have been asked to provide a presentation / business case as to why to use spark and java as ingestion tool for HDFS and HIVE
And why to move away from an etl tool.
 
Could you be so kind as to provide with some pros and cons to this.
 
I have the following :
 
Pros:
In house build – code can be changes on the fly to suite business need.
Software is free
Can out of the box run on all nodes
Will support all Apache based software.
Fast deu to in memory processing
Spark UI can visualise execution
Support checkpoint data loads
Support echama regesty for custom schema and inference.
Support Yarn execution
Mlibs can be used in need.
Data linage support deu to spar usage.
 
Cons
Skills needed to maintain and build
In memory cabibility can become bottleneck if not managed
No ETL gui.
 
Maybe point be to an article if you have one.
 
Thanks a mill.
Christian
 


Standard Bank email disclaimer and confidentiality note
Please go to www.standardbank.co.za/site/homepage/emaildisclaimer.html to read our email disclaimer and confidentiality note. Kindly email [hidden email] (no content or subject line necessary) if you cannot view that page and we will email our email disclaimer and confidentiality note to you.