Structured Streaming: stream-stream join with several equality conditions in a disjunction

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Structured Streaming: stream-stream join with several equality conditions in a disjunction

WILSON Frank

Hi,

 

I've just tried to do a stream-stream join with several equality conditions in a disjunction and got the following error:

 

"terminated with exception: Stream stream joins without equality predicate is not supported;;"

 

The query was in this sort of form:

 

    stream.as("origin").join(stream.as("remote"),

      ($"origin.a" === $"remote.a").or($"origin.a" === $"remote.b").or($"origin.a" === $"remote.c")

        .and($"remote.timestamp" <= $"origin.timestamp")

        .and($"remote.timestamp" >= ($"origin.timestamp" - expr("interval 100 seconds"))))

              

 

Is there a fundamental problem with combining the results of these equality comparisons? Or is it just that spark currently doesn't

dig deep enough into the query to find them?

 

The work around for now is I suppose to take the union of several queries with only one equality condition.

 

Thanks,

 

 

Frank