Spark streaming for CEP

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark streaming for CEP

anna
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna


Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

Mich Talebzadeh
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna



Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

Thomas Bailet

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna




Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

Mich Talebzadeh
thanks Thomas.

do you have a summary write-up for this tool please?


regards,




Thomas

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 13:53, Thomas Bailet <[hidden email]> wrote:

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna





Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

Stephen Boesch
Hi Mich, the github link has a brief intro - including a link to the formal docs http://logisland.readthedocs.io/en/latest/index.html .   They have an architectural overview, developer guide, tutorial, and pretty comprehensive api docs.

2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <[hidden email]>:
thanks Thomas.

do you have a summary write-up for this tool please?


regards,




Thomas

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 13:53, Thomas Bailet <[hidden email]> wrote:

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna






Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

Mich Talebzadeh
Great thanks Steve

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 22:58, Stephen Boesch <[hidden email]> wrote:
Hi Mich, the github link has a brief intro - including a link to the formal docs http://logisland.readthedocs.io/en/latest/index.html .   They have an architectural overview, developer guide, tutorial, and pretty comprehensive api docs.

2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <[hidden email]>:
thanks Thomas.

do you have a summary write-up for this tool please?


regards,




Thomas

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 13:53, Thomas Bailet <[hidden email]> wrote:

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna







Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

lucas.gary@gmail.com
This looks really interesting, thanks for linking!

Gary Lucas

On 24 October 2017 at 15:06, Mich Talebzadeh <[hidden email]> wrote:
Great thanks Steve

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 22:58, Stephen Boesch <[hidden email]> wrote:
Hi Mich, the github link has a brief intro - including a link to the formal docs http://logisland.readthedocs.io/en/latest/index.html .   They have an architectural overview, developer guide, tutorial, and pretty comprehensive api docs.

2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <[hidden email]>:
thanks Thomas.

do you have a summary write-up for this tool please?


regards,




Thomas

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 13:53, Thomas Bailet <[hidden email]> wrote:

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna








Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming for CEP

anna
Thanks very much  Mich, Thomas and Stephan . I will look into it.

On Tue, Oct 24, 2017 at 8:02 PM, [hidden email] <[hidden email]> wrote:
This looks really interesting, thanks for linking!

Gary Lucas

On 24 October 2017 at 15:06, Mich Talebzadeh <[hidden email]> wrote:
Great thanks Steve

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 22:58, Stephen Boesch <[hidden email]> wrote:
Hi Mich, the github link has a brief intro - including a link to the formal docs http://logisland.readthedocs.io/en/latest/index.html .   They have an architectural overview, developer guide, tutorial, and pretty comprehensive api docs.

2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <[hidden email]>:
thanks Thomas.

do you have a summary write-up for this tool please?


regards,




Thomas

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 24 October 2017 at 13:53, Thomas Bailet <[hidden email]> wrote:

Hi

we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called logisland  (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look.


bye

Thomas Bailet

CTO : hurence


Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.

Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP)


HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 18 October 2017 at 20:52, anna stax <[hidden email]> wrote:
Hello all,

Has anyone used spark streaming for CEP (Complex Event processing).  Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. 

Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard.

Please share your experience. 
Thanks much.
-Anna