Compression with DISK_ONLY persistence

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Compression with DISK_ONLY persistence

Surendranauth Hiraman
Hi,

Will spark.rdd.compress=true enable compression when using DISK_ONLY persistence? 


                                                            
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: [hidden email]elos.io
W: www.velos.io

Reply | Threaded
Open this post in threaded view
|

Re: Compression with DISK_ONLY persistence

Matei Zaharia
Administrator
Yes, actually even if you don’t set it to true, on-disk data is compressed. (This setting only affects serialized data in memory).

Matei

On Jun 11, 2014, at 2:56 PM, Surendranauth Hiraman <[hidden email]> wrote:

Hi,

Will spark.rdd.compress=true enable compression when using DISK_ONLY persistence? 


                                                            
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: [hidden email]elos.io
W: www.velos.io