how to config worker HA

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

how to config worker HA

qingyang li
i have one table in memery,  when one worker becomes dead, i can not query data from that table. Here is it's storage status:


RDD Name Storage LevelCached PartitionsFraction CachedSize in MemorySize on Disk


table01 Memory Deserialized 1x Replicated 119 88%       697.0 MB     0.0 B
so, my question is:
1. what meaning is " Memory Deserialized 1x Replicated" ?
2. how to config worker HA so that i can query data even one worker dead.
Reply | Threaded
Open this post in threaded view
|

Re: how to config worker HA

qingyang li
in addition:
on this site:https://spark.apache.org/docs/0.9.0/scala-programming-guide.html#hadoop-datasets,
i find RDD can be stored using a different storage level on the web,  and  also find StorageLevel's attribute MEMORY_ONLY_2 .
MEMORY_ONLY_2, Same as the levels above, but replicate each partition on two cluster nodes.
1. is this one point of fault-tolerance ?
2.if replicate each partition on two cluster nodes will help worker node HA ?
3. if there is MEMORY_ONLY_3 which could replicate each partition on three cluster nodes?




2014-03-12 12:11 GMT+08:00 qingyang li <[hidden email]>:
i have one table in memery,  when one worker becomes dead, i can not query data from that table. Here is it's storage status:


RDD Name Storage LevelCached PartitionsFraction CachedSize in MemorySize on Disk


table01 Memory Deserialized 1x Replicated 119 88%       697.0 MB     0.0 B
so, my question is:
1. what meaning is " Memory Deserialized 1x Replicated" ?
2. how to config worker HA so that i can query data even one worker dead.