Setting spark memory limit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Setting spark memory limit

cuihenggang
Hi,

I'm trying to run the SimpleApp example (http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala) on a larger dataset.

The input file is about 1GB, but when I run the Spark program, it says:"java.lang.OutOfMemoryError: GC overhead limit exceeded", the full error output is attached at the end of the E-mail.

Then I tried multiple ways of setting the memory limit.

In SimpleApp.scala file, I set the following configurations:
    val conf = new SparkConf()
               .setAppName("Simple Application")
               .set("spark.executor.memory", "10g")

And I have also tried appending the following configuration to conf/spark-defaults.conf file:
    spark.executor.memory   10g

But neither of them works. In the error message, it claims "(estimated size 103.8 MB, free 191.1 MB)", so the total available memory is still 300MB. Why?

Thanks,
Cui


$ ~/spark-1.0.0-bin-hadoop1/bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.10/simple-project_2.10-1.0.jar /tmp/mdata0-10.tsd
Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/06/09 15:06:29 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/06/09 15:06:29 INFO SecurityManager: Changing view acls to: cuihe
14/06/09 15:06:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cuihe)
14/06/09 15:06:29 INFO Slf4jLogger: Slf4jLogger started
14/06/09 15:06:29 INFO Remoting: Starting remoting
14/06/09 15:06:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@...:40779]
14/06/09 15:06:30 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@...:40779]
14/06/09 15:06:30 INFO SparkEnv: Registering MapOutputTracker
14/06/09 15:06:30 INFO SparkEnv: Registering BlockManagerMaster
14/06/09 15:06:30 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140609150630-eaa9
14/06/09 15:06:30 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.
14/06/09 15:06:30 INFO ConnectionManager: Bound socket to port 47164 with id = ConnectionManagerId(131-1.bfc.hpl.hp.com,47164)
14/06/09 15:06:30 INFO BlockManagerMaster: Trying to register BlockManager
14/06/09 15:06:30 INFO BlockManagerInfo: Registering block manager 131-1.bfc.hpl.hp.com:47164 with 294.9 MB RAM
14/06/09 15:06:30 INFO BlockManagerMaster: Registered BlockManager
14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
14/06/09 15:06:30 INFO HttpBroadcast: Broadcast server started at http://16.106.36.131:48587
14/06/09 15:06:30 INFO HttpFileServer: HTTP File server directory is /tmp/spark-35e1c47b-bfa1-4fba-bc64-df8eee287bb7
14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
14/06/09 15:06:30 INFO SparkUI: Started SparkUI at http://131-1.bfc.hpl.hp.com:4040
14/06/09 15:06:30 INFO SparkContext: Added JAR file:/data/cuihe/spark-app/target/scala-2.10/simple-project_2.10-1.0.jar at http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp 1402351590741
14/06/09 15:06:30 INFO MemoryStore: ensureFreeSpace(32856) called with curMem=0, maxMem=309225062
14/06/09 15:06:30 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 32.1 KB, free 294.9 MB)
14/06/09 15:06:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/09 15:06:30 WARN LoadSnappy: Snappy native library not loaded
14/06/09 15:06:30 INFO FileInputFormat: Total input paths to process : 1
14/06/09 15:06:30 INFO SparkContext: Starting job: count at SimpleApp.scala:14
14/06/09 15:06:31 INFO DAGScheduler: Got job 0 (count at SimpleApp.scala:14) with 7 output partitions (allowLocal=false)
14/06/09 15:06:31 INFO DAGScheduler: Final stage: Stage 0(count at SimpleApp.scala:14)
14/06/09 15:06:31 INFO DAGScheduler: Parents of final stage: List()
14/06/09 15:06:31 INFO DAGScheduler: Missing parents: List()
14/06/09 15:06:31 INFO DAGScheduler: Submitting Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:14), which has no missing parents
14/06/09 15:06:31 INFO DAGScheduler: Submitting 7 missing tasks from Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:14)
14/06/09 15:06:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 7 tasks
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:0 as 1839 bytes in 2 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:1 as 1839 bytes in 0 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:2 as TID 2 on executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:2 as 1839 bytes in 1 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:3 as TID 3 on executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:3 as 1839 bytes in 1 ms
14/06/09 15:06:31 INFO Executor: Running task ID 0
14/06/09 15:06:31 INFO Executor: Running task ID 1
14/06/09 15:06:31 INFO Executor: Running task ID 2
14/06/09 15:06:31 INFO Executor: Running task ID 3
14/06/09 15:06:31 INFO Executor: Fetching http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp 1402351590741
14/06/09 15:06:31 INFO Utils: Fetching http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar to /tmp/fetchFileTemp7241193225836706654.tmp
14/06/09 15:06:31 INFO Executor: Adding file:/tmp/spark-68aa13c8-8146-4e6a-80a1-c406a4cef89f/simple-project_2.10-1.0.jar to class loader
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_2 not found, computing it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_0 not found, computing it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_1 not found, computing it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_3 not found, computing it
14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:67108864+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:33554432+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:100663296+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:0+33554432
14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108800293) called with curMem=32856, maxMem=309225062
14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_2 stored as values to memory (estimated size 103.8 MB, free 191.1 MB)
14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108716407) called with curMem=108833149, maxMem=309225062
14/06/09 15:06:50 ERROR Executor: Exception in task ID 1
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
        at java.nio.CharBuffer.toString(CharBuffer.java:1201)
        at org.apache.hadoop.io.Text.decode(Text.java:350)
        at org.apache.hadoop.io.Text.decode(Text.java:327)
        at org.apache.hadoop.io.Text.toString(Text.java:254)
        at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        at org.apache.spark.scheduler.Task.run(Task.scala:51)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_2 in memory on 131-1.bfc.hpl.hp.com:47164 (size: 103.8 MB, free: 191.1 MB)
14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_3 stored as values to memory (estimated size 103.7 MB, free 87.4 MB)
14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_2
14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_3 in memory on 131-1.bfc.hpl.hp.com:47164 (size: 103.7 MB, free: 87.5 MB)
14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_3
14/06/09 15:06:50 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-1,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
        at java.nio.CharBuffer.toString(CharBuffer.java:1201)
        at org.apache.hadoop.io.Text.decode(Text.java:350)
        at org.apache.hadoop.io.Text.decode(Text.java:327)
        at org.apache.hadoop.io.Text.toString(Text.java:254)
        at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        at org.apache.spark.scheduler.Task.run(Task.scala:51)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
14/06/09 15:06:50 INFO TaskSetManager: Starting task 0.0:4 as TID 4 on executor localhost: localhost (PROCESS_LOCAL)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Setting spark memory limit

Patrick Wendell
I you run locally then Spark doesn't launch remote executors. However,
in this case you can set the memory with --spark-driver-memory flag to
spark-submit. Does that work?

- Patrick

On Mon, Jun 9, 2014 at 3:24 PM, Henggang Cui <[hidden email]> wrote:

> Hi,
>
> I'm trying to run the SimpleApp example
> (http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala)
> on a larger dataset.
>
> The input file is about 1GB, but when I run the Spark program, it
> says:"java.lang.OutOfMemoryError: GC overhead limit exceeded", the full
> error output is attached at the end of the E-mail.
>
> Then I tried multiple ways of setting the memory limit.
>
> In SimpleApp.scala file, I set the following configurations:
>     val conf = new SparkConf()
>                .setAppName("Simple Application")
>                .set("spark.executor.memory", "10g")
>
> And I have also tried appending the following configuration to
> conf/spark-defaults.conf file:
>     spark.executor.memory   10g
>
> But neither of them works. In the error message, it claims "(estimated size
> 103.8 MB, free 191.1 MB)", so the total available memory is still 300MB.
> Why?
>
> Thanks,
> Cui
>
>
> $ ~/spark-1.0.0-bin-hadoop1/bin/spark-submit --class "SimpleApp" --master
> local[4] target/scala-2.10/simple-project_2.10-1.0.jar /tmp/mdata0-10.tsd
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> 14/06/09 15:06:29 INFO SecurityManager: Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 14/06/09 15:06:29 INFO SecurityManager: Changing view acls to: cuihe
> 14/06/09 15:06:29 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(cuihe)
> 14/06/09 15:06:29 INFO Slf4jLogger: Slf4jLogger started
> 14/06/09 15:06:29 INFO Remoting: Starting remoting
> 14/06/09 15:06:30 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://[hidden email]:40779]
> 14/06/09 15:06:30 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://[hidden email]:40779]
> 14/06/09 15:06:30 INFO SparkEnv: Registering MapOutputTracker
> 14/06/09 15:06:30 INFO SparkEnv: Registering BlockManagerMaster
> 14/06/09 15:06:30 INFO DiskBlockManager: Created local directory at
> /tmp/spark-local-20140609150630-eaa9
> 14/06/09 15:06:30 INFO MemoryStore: MemoryStore started with capacity 294.9
> MB.
> 14/06/09 15:06:30 INFO ConnectionManager: Bound socket to port 47164 with id
> = ConnectionManagerId(131-1.bfc.hpl.hp.com,47164)
> 14/06/09 15:06:30 INFO BlockManagerMaster: Trying to register BlockManager
> 14/06/09 15:06:30 INFO BlockManagerInfo: Registering block manager
> 131-1.bfc.hpl.hp.com:47164 with 294.9 MB RAM
> 14/06/09 15:06:30 INFO BlockManagerMaster: Registered BlockManager
> 14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
> 14/06/09 15:06:30 INFO HttpBroadcast: Broadcast server started at
> http://16.106.36.131:48587
> 14/06/09 15:06:30 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-35e1c47b-bfa1-4fba-bc64-df8eee287bb7
> 14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
> 14/06/09 15:06:30 INFO SparkUI: Started SparkUI at
> http://131-1.bfc.hpl.hp.com:4040
> 14/06/09 15:06:30 INFO SparkContext: Added JAR
> file:/data/cuihe/spark-app/target/scala-2.10/simple-project_2.10-1.0.jar at
> http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp
> 1402351590741
> 14/06/09 15:06:30 INFO MemoryStore: ensureFreeSpace(32856) called with
> curMem=0, maxMem=309225062
> 14/06/09 15:06:30 INFO MemoryStore: Block broadcast_0 stored as values to
> memory (estimated size 32.1 KB, free 294.9 MB)
> 14/06/09 15:06:30 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/09 15:06:30 WARN LoadSnappy: Snappy native library not loaded
> 14/06/09 15:06:30 INFO FileInputFormat: Total input paths to process : 1
> 14/06/09 15:06:30 INFO SparkContext: Starting job: count at
> SimpleApp.scala:14
> 14/06/09 15:06:31 INFO DAGScheduler: Got job 0 (count at SimpleApp.scala:14)
> with 7 output partitions (allowLocal=false)
> 14/06/09 15:06:31 INFO DAGScheduler: Final stage: Stage 0(count at
> SimpleApp.scala:14)
> 14/06/09 15:06:31 INFO DAGScheduler: Parents of final stage: List()
> 14/06/09 15:06:31 INFO DAGScheduler: Missing parents: List()
> 14/06/09 15:06:31 INFO DAGScheduler: Submitting Stage 0 (FilteredRDD[2] at
> filter at SimpleApp.scala:14), which has no missing parents
> 14/06/09 15:06:31 INFO DAGScheduler: Submitting 7 missing tasks from Stage 0
> (FilteredRDD[2] at filter at SimpleApp.scala:14)
> 14/06/09 15:06:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 7 tasks
> 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on
> executor localhost: localhost (PROCESS_LOCAL)
> 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:0 as 1839 bytes
> in 2 ms
> 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on
> executor localhost: localhost (PROCESS_LOCAL)
> 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:1 as 1839 bytes
> in 0 ms
> 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:2 as TID 2 on
> executor localhost: localhost (PROCESS_LOCAL)
> 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:2 as 1839 bytes
> in 1 ms
> 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:3 as TID 3 on
> executor localhost: localhost (PROCESS_LOCAL)
> 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:3 as 1839 bytes
> in 1 ms
> 14/06/09 15:06:31 INFO Executor: Running task ID 0
> 14/06/09 15:06:31 INFO Executor: Running task ID 1
> 14/06/09 15:06:31 INFO Executor: Running task ID 2
> 14/06/09 15:06:31 INFO Executor: Running task ID 3
> 14/06/09 15:06:31 INFO Executor: Fetching
> http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp
> 1402351590741
> 14/06/09 15:06:31 INFO Utils: Fetching
> http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar to
> /tmp/fetchFileTemp7241193225836706654.tmp
> 14/06/09 15:06:31 INFO Executor: Adding
> file:/tmp/spark-68aa13c8-8146-4e6a-80a1-c406a4cef89f/simple-project_2.10-1.0.jar
> to class loader
> 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
> 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
> 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
> 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
> 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_2 not found, computing
> it
> 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_0 not found, computing
> it
> 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_1 not found, computing
> it
> 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_3 not found, computing
> it
> 14/06/09 15:06:31 INFO HadoopRDD: Input split:
> file:/tmp/mdata0-10.tsd:67108864+33554432
> 14/06/09 15:06:31 INFO HadoopRDD: Input split:
> file:/tmp/mdata0-10.tsd:33554432+33554432
> 14/06/09 15:06:31 INFO HadoopRDD: Input split:
> file:/tmp/mdata0-10.tsd:100663296+33554432
> 14/06/09 15:06:31 INFO HadoopRDD: Input split:
> file:/tmp/mdata0-10.tsd:0+33554432
> 14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108800293) called with
> curMem=32856, maxMem=309225062
> 14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_2 stored as values to memory
> (estimated size 103.8 MB, free 191.1 MB)
> 14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108716407) called with
> curMem=108833149, maxMem=309225062
> 14/06/09 15:06:50 ERROR Executor: Exception in task ID 1
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at java.util.Arrays.copyOfRange(Arrays.java:2694)
>         at java.lang.String.<init>(String.java:203)
>         at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
>         at java.nio.CharBuffer.toString(CharBuffer.java:1201)
>         at org.apache.hadoop.io.Text.decode(Text.java:350)
>         at org.apache.hadoop.io.Text.decode(Text.java:327)
>         at org.apache.hadoop.io.Text.toString(Text.java:254)
>         at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
>         at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>         at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>         at
> org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
>         at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>         at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>         at org.apache.spark.scheduler.Task.run(Task.scala:51)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_2 in memory on
> 131-1.bfc.hpl.hp.com:47164 (size: 103.8 MB, free: 191.1 MB)
> 14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_3 stored as values to memory
> (estimated size 103.7 MB, free 87.4 MB)
> 14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_2
> 14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_3 in memory on
> 131-1.bfc.hpl.hp.com:47164 (size: 103.7 MB, free: 87.5 MB)
> 14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_3
> 14/06/09 15:06:50 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception
> in thread Thread[Executor task launch worker-1,5,main]
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at java.util.Arrays.copyOfRange(Arrays.java:2694)
>         at java.lang.String.<init>(String.java:203)
>         at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
>         at java.nio.CharBuffer.toString(CharBuffer.java:1201)
>         at org.apache.hadoop.io.Text.decode(Text.java:350)
>         at org.apache.hadoop.io.Text.decode(Text.java:327)
>         at org.apache.hadoop.io.Text.toString(Text.java:254)
>         at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
>         at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>         at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>         at
> org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
>         at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>         at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>         at org.apache.spark.scheduler.Task.run(Task.scala:51)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 14/06/09 15:06:50 INFO TaskSetManager: Starting task 0.0:4 as TID 4 on
> executor localhost: localhost (PROCESS_LOCAL)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Setting spark memory limit

cuihenggang
This post has NOT been accepted by the mailing list yet.
I find the correct flag should actually be --driver-memory. Then it works, thank you!

Cui
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Setting spark memory limit

marwan radwan
This post has NOT been accepted by the mailing list yet.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/04/03 03:44:54 INFO SparkContext: Running Spark version 1.3.0
17/04/03 03:44:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/03 03:44:54 INFO SecurityManager: Changing view acls to: cloudera
17/04/03 03:44:54 INFO SecurityManager: Changing modify acls to: cloudera
17/04/03 03:44:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera); users with modify permissions: Set(cloudera)
17/04/03 03:44:54 INFO Slf4jLogger: Slf4jLogger started
17/04/03 03:44:54 INFO Remoting: Starting remoting
17/04/03 03:44:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@quickstart.cloudera:56927]
17/04/03 03:44:55 INFO Utils: Successfully started service 'sparkDriver' on port 56927.
17/04/03 03:44:55 INFO SparkEnv: Registering MapOutputTracker
17/04/03 03:44:55 INFO SparkEnv: Registering BlockManagerMaster
17/04/03 03:44:55 INFO DiskBlockManager: Created local directory at /tmp/spark-61388fd5-1685-472a-b039-1094169393e4/blockmgr-c27fa491-ccc1-4cd1-a353-669c80e57c33
17/04/03 03:44:55 INFO MemoryStore: MemoryStore started with capacity 2.6 GB
17/04/03 03:44:55 INFO HttpFileServer: HTTP File server directory is /tmp/spark-2b2e3139-93c3-4d9d-a29c-8ca92b61a867/httpd-427efcdb-b097-40bd-942d-300659a7ceba
17/04/03 03:44:55 INFO HttpServer: Starting HTTP Server
17/04/03 03:44:55 INFO Server: jetty-8.y.z-SNAPSHOT
17/04/03 03:44:55 INFO AbstractConnector: Started SocketConnector@0.0.0.0:60207
17/04/03 03:44:55 INFO Utils: Successfully started service 'HTTP file server' on port 60207.
17/04/03 03:44:55 INFO SparkEnv: Registering OutputCommitCoordinator
17/04/03 03:44:55 INFO Server: jetty-8.y.z-SNAPSHOT
17/04/03 03:44:55 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:444)
        at sun.nio.ch.Net.bind(Net.java:436)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.spark-project.jetty.server.Server.doStart(Server.java:293)
        at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:199)
        at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:209)
        at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:209)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1832)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1823)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:209)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
        at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:307)
        at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:307)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:307)
        at org.FCI.spark.training.prefinal.Cluster$.main(Cluster.scala:22)
        at org.FCI.spark.training.prefinal.Cluster.main(Cluster.scala)
17/04/03 03:44:55 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@68d4ea21: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:444)
        at sun.nio.ch.Net.bind(Net.java:436)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.spark-project.jetty.server.Server.doStart(Server.java:293)
        at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:199)
        at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:209)
        at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:209)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1832)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1823)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:209)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
        at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:307)
        at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:307)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:307)
        at org.FCI.spark.training.prefinal.Cluster$.main(Cluster.scala:22)
        at org.FCI.spark.training.prefinal.Cluster.main(Cluster.scala)
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
17/04/03 03:44:55 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
17/04/03 03:44:55 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/04/03 03:44:55 INFO Server: jetty-8.y.z-SNAPSHOT
17/04/03 03:44:55 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
17/04/03 03:44:55 INFO Utils: Successfully started service 'SparkUI' on port 4041.
17/04/03 03:44:55 INFO SparkUI: Started SparkUI at http://quickstart.cloudera:4041
17/04/03 03:44:55 INFO Executor: Starting executor ID <driver> on host localhost
17/04/03 03:44:55 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@quickstart.cloudera:56927/user/HeartbeatReceiver
17/04/03 03:44:55 INFO NettyBlockTransferService: Server created on 34765
17/04/03 03:44:55 INFO BlockManagerMaster: Trying to register BlockManager
17/04/03 03:44:55 INFO BlockManagerMasterActor: Registering block manager localhost:34765 with 2.6 GB RAM, BlockManagerId(<driver>, localhost, 34765)
17/04/03 03:44:55 INFO BlockManagerMaster: Registered BlockManager
Loading...