Fair Scheduling

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Fair Scheduling

vinay Bajaj

I am trying for FAIR scheduling inside the pool that can be done by setting LocalProperty in sparkContext. But i am Using JavaSparkContext in my java Code and passing sparkContext Object for making JavaSparkObject.

System.setProperty("spark.scheduler.mode", "FAIR");
System.setProperty("spark.scheduler.allocation.file", "/opt/spark/spark-0.8.1-incubating/conf/fairscheduler.xml");
JavaSparkContext sc = new JavaSparkContext("spark://c8-data-store-1.srv.media.net:7077" , "sparkLogAnalyzer", "/opt/spark/spark-0.8.1-incubating", "/autoopt/vinay/sparkTesting/sparkLogAnalyser.jar");

 Seq<String> seq = new ArraySeq<String>(1);
 seq.addString(new StringBuilder("/autoopt/vinay/sparkTesting/sparkLogAnalyser.jar"));

 SparkContext context = new SparkContext(cluster URL , "sparkLogAnalyzer", "/opt/spark/spark-0.8.1-incubating",seq, null,null);
context.setLocalProperty("spark.scheduler.pool", "pool1");

JavaSparkContext sc = new JavaSparkContext(context);

String path = "hdfs://path/sparkTestingData/smallData.txt";
JavaRDD<String> lines = sc.textFile(path);

currently i am using spark 0.8.1, SetLocalProperty is only function for sparkContext and is not in JavaSparkContext.

This code is giving me some "class not found exception" but unable to figure out. Can anyone validate if i am doing it right way and how can one do FAIR scheduling inside the pool??

Vinay Bajaj