SparkStreaming getActiveOrCreate

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

SparkStreaming getActiveOrCreate

Justin Pihony
This post has NOT been accepted by the mailing list yet.
The docs on getActiveOrCreate makes it seem that you'll get an already started context:

> Either return the "active" StreamingContext (that is, started but not stopped), or create a new StreamingContext that is

However as far as I can tell from the code it is strictly dependent on the the implementation of the create function, and per the tests it is even expected that the create function will return a non-started function. So this makes for a bit awkward code that must check the state before starting or not. Was this considered when this was added? I couldn't find anything explicit

-Justin Pihony