java.lang.NullPointerException

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

java.lang.NullPointerException

Mina Aslani
Hi,

I get 
java.lang.NullPointerException at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:128)

When I try to createDataFrame using the sparkSession, see below:

SparkConf conf = new SparkConf().setMaster(....).setAppName("test");
conf.set("spark.driver.allowMultipleContexts", "true");

//Read messages in batch of duration seconds
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
List<Row> data = Arrays.asList(
RowFactory.create(val1, val2, val3)
);
StructType schema =
new StructType(new StructField[]{
createStructField(
"field1", StringType, false),
createStructField(
"field2", StringType, false),
createStructField(
"field3", StringType, false)
});


Dataset<Row> df = spark.createDataFrame(data, schema);

Any idea? I am using JAVA therefore,I cannot convert my RDD to DF(). T
Therefore, I try to get specific field/values and createDataFrame manually.

Regards,
Mina