NPE while reading broadcast variable.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

NPE while reading broadcast variable.

Archit Thakur
Hi,

I am getting NPE while I access broadcast variable.

I created an object:

object ClusterVariableEnumeration {
 
  var CURRENT_EXECUTION: org.apache.spark.broadcast.Broadcast[Any] = null.asInstanceOf[org.apache.spark.broadcast.Broadcast[Any]];
}

and then after creating SparkContext, I did ClusterVariableEnumeration.CURRENT_EXECUTION = sc.broadcast("value")

and then in map function When I tried to access it it gave me NPE. Idea?

Thanks and Regards,
Archit Thakur.
Reply | Threaded
Open this post in threaded view
|

Re: NPE while reading broadcast variable.

Archit Thakur
Accessed it:

val CURRENT_EXECUTING = ClusterVariableEnumeration.CURRENT_EXECUTION.value.asInstanceOf[String]


On Mon, Dec 23, 2013 at 11:27 AM, Archit Thakur <[hidden email]> wrote:
Hi,

I am getting NPE while I access broadcast variable.

I created an object:

object ClusterVariableEnumeration {
 
  var CURRENT_EXECUTION: org.apache.spark.broadcast.Broadcast[Any] = null.asInstanceOf[org.apache.spark.broadcast.Broadcast[Any]];
}

and then after creating SparkContext, I did ClusterVariableEnumeration.CURRENT_EXECUTION = sc.broadcast("value")

and then in map function When I tried to access it it gave me NPE. Idea?

Thanks and Regards,
Archit Thakur.

Reply | Threaded
Open this post in threaded view
|

Re: NPE while reading broadcast variable.

Archit Thakur
I am still getting it. I googled and found a similar open problem on stackoverflow: http://stackoverflow.com/questions/17794664/accumulator-fails-on-cluster-works-locally.

Thx, Archit_Thakur.


On Mon, Dec 23, 2013 at 11:32 AM, Archit Thakur <[hidden email]> wrote:
Accessed it:

val CURRENT_EXECUTING = ClusterVariableEnumeration.CURRENT_EXECUTION.value.asInstanceOf[String]


On Mon, Dec 23, 2013 at 11:27 AM, Archit Thakur <[hidden email]> wrote:
Hi,

I am getting NPE while I access broadcast variable.

I created an object:

object ClusterVariableEnumeration {
 
  var CURRENT_EXECUTION: org.apache.spark.broadcast.Broadcast[Any] = null.asInstanceOf[org.apache.spark.broadcast.Broadcast[Any]];
}

and then after creating SparkContext, I did ClusterVariableEnumeration.CURRENT_EXECUTION = sc.broadcast("value")

and then in map function When I tried to access it it gave me NPE. Idea?

Thanks and Regards,
Archit Thakur.


Reply | Threaded
Open this post in threaded view
|

Re: NPE while reading broadcast variable.

Aaron Davidson
Could you post the stack trace you see for the NPE?


On Mon, Dec 30, 2013 at 11:31 AM, Archit Thakur <[hidden email]> wrote:
I am still getting it. I googled and found a similar open problem on stackoverflow: http://stackoverflow.com/questions/17794664/accumulator-fails-on-cluster-works-locally.

Thx, Archit_Thakur.


On Mon, Dec 23, 2013 at 11:32 AM, Archit Thakur <[hidden email]> wrote:
Accessed it:

val CURRENT_EXECUTING = ClusterVariableEnumeration.CURRENT_EXECUTION.value.asInstanceOf[String]


On Mon, Dec 23, 2013 at 11:27 AM, Archit Thakur <[hidden email]> wrote:
Hi,

I am getting NPE while I access broadcast variable.

I created an object:

object ClusterVariableEnumeration {
 
  var CURRENT_EXECUTION: org.apache.spark.broadcast.Broadcast[Any] = null.asInstanceOf[org.apache.spark.broadcast.Broadcast[Any]];
}

and then after creating SparkContext, I did ClusterVariableEnumeration.CURRENT_EXECUTION = sc.broadcast("value")

and then in map function When I tried to access it it gave me NPE. Idea?

Thanks and Regards,
Archit Thakur.



Reply | Threaded
Open this post in threaded view
|

Re: NPE while reading broadcast variable.

Archit Thakur
I understood the problem. The object ClusterVariableEnumeration (or the class ClusterVariableEnumeration$) will be again loaded on Worker machine.

It will again instantiate the class variables ( var CURRENT_EXECUTING). Which will now have their default values (null). Hence NPE.

I guess, I need to pass ClusterVariableEnumeration's instance to Worker machine. Isn't it?