This post has NOT been accepted by the mailing list yet.
I am playing around with Spark structured streaming and we have a use case to use this as a CEP engine.
I am reading from 3 different kafka topics together. I want to perform windowing on this structured stream and then run some queries on this block on a sliding scale. Also, all of this needs to happen on the event time and I have my corresponding timestamp attached to each event.
Now I have something like this
val windowedEvents = filteredBamEvents.withWatermark("timestamp", "10 minutes")
.groupBy(functions.window($"timestamp", "10 minutes", "5 minutes"))
where timestamp is of type long in my case class as follows:
case class BAMIngestedEvent(id: String, eventName: String, eventID: String, correlationID: Seq[String], timestamp: Long)
But when I am running this example with my data from kafka I cam getting this following exception:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Event time must be defined on a window or a timestamp, but timestamp is of type bigint;;
|Free forum by Nabble||Edit this page|