How to use window method with direct kafka streaming ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to use window method with direct kafka streaming ?

fat.wei
Hi everyone,

I have the following scenario , and I tried to use window method with direct kafka streaming. The program can run, but doesn't run right!

1. The data is stored in kafka.
2. Every single item of the data has its primary key.
3. Every single item of the data will be split into multipe parts,and these parts will arrive at kafka in order.

Here's my sample code:
 
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(20));
 
JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(jssc, String.class, String.class,
StringDecoder.class, StringDecoder.class, kafkaParams, topicsSet);

messages.window(Durations.seconds(60), Durations.seconds(40)).print();


I couldn't get the data of  RDD@40 ,when I tried to print the data of windowed RDD@80.

Can I have some  suggestions!