Use of Accumulators

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Use of Accumulators

kedarsdixit

Hi,


We need some way to toggle the flag of  a variable in transformation.


We are thinking to make use of spark  Accumulators for this purpose.


Can we use these as below:


Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0


In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.


Thanks!


With Regards,

~Kedar Dixit


[hidden email] | @kedarsdixit | M +91 90499 15588 | T +91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Reply | Threaded
Open this post in threaded view
|

Re: Use of Accumulators

vaquar khan
Confirmed ,you can use Accumulators :)

Regards,
Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]> wrote:

Hi,


We need some way to toggle the flag of  a variable in transformation.


We are thinking to make use of spark  Accumulators for this purpose.


Can we use these as below:


Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0


In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.


Thanks!


With Regards,

~Kedar Dixit


[hidden email] | @kedarsdixit | M <a href="tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
Reply | Threaded
Open this post in threaded view
|

Re: Use of Accumulators

Holden Karau
So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]> wrote:
Confirmed ,you can use Accumulators :)

Regards,
Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]> wrote:

Hi,


We need some way to toggle the flag of  a variable in transformation.


We are thinking to make use of spark  Accumulators for this purpose.


Can we use these as below:


Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0


In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.


Thanks!


With Regards,

~Kedar Dixit


[hidden email] | @kedarsdixit | M <a href="tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
--
Reply | Threaded
Open this post in threaded view
|

RE: Use of Accumulators

kedarsdixit

Hi,

 

Inside the transformation if there is any change for the Variable’s associated data, I want to just toggle it saying there is some change while processing the data.

 

Please let me know if we can runtime do this.

 

 

Thanks!

~Kedar Dixit

Bigdata Analytics at Persistent Systems Ltd.

 

From: Holden Karau [via Apache Spark User List] [[hidden email]]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit <[hidden email]>
Subject: Re: Use of Accumulators

 

So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?

 

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]> wrote:

Confirmed ,you can use Accumulators :)

 

Regards,

Vaquar khan

 

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]> wrote:

Hi,

 

We need some way to toggle the flag of  a variable in transformation.

 

We are thinking to make use of spark  Accumulators for this purpose.

 

Can we use these as below:

 

Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0

 

In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.

 

Thanks!

 

With Regards,

~Kedar Dixit

 

[hidden email] | @kedarsdixit | M <a href="<a href="tel:&#43;91%2090499%2015588">tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="<a href="tel:&#43;91%2020%206703%204783">tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.



 

--

Regards,

Vaquar Khan

+1 -224-436-0783

Greater Chicago

--

 


If you reply to this email, your message will be added to the discussion below:

http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html

To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Use of Accumulators

Holden Karau
And where do you want to read the toggle back from? On the driver?

On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit <[hidden email]> wrote:

Hi,

 

Inside the transformation if there is any change for the Variable’s associated data, I want to just toggle it saying there is some change while processing the data.

 

Please let me know if we can runtime do this.

 

 

Thanks!

~Kedar Dixit

Bigdata Analytics at Persistent Systems Ltd.

 

From: Holden Karau [via Apache Spark User List] [[hidden email]]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit <[hidden email]>


Subject: Re: Use of Accumulators

 

So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?

 

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]> wrote:

Confirmed ,you can use Accumulators :)

 

Regards,

Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]> wrote:

Hi,

 

We need some way to toggle the flag of  a variable in transformation.

 

We are thinking to make use of spark  Accumulators for this purpose.

 

Can we use these as below:

 

Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0

 

In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.

 

Thanks!

 

With Regards,

~Kedar Dixit

 

[hidden email] | @kedarsdixit | M <a href="<a href="tel:+91%2090499%2015588" target="_blank">tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="<a href="tel:+91%2020%206703%204783" target="_blank">tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

--

Regards,

Vaquar Khan

+1 -224-436-0783

Greater Chicago

--

 


If you reply to this email, your message will be added to the discussion below:

http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html

To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

--
Reply | Threaded
Open this post in threaded view
|

Re: Use of Accumulators

kedarsdixit


Yes!



Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.


From: Holden Karau <[hidden email]>
Sent: 14 November 2017 20:04:50
To: Kedarnath Dixit
Cc: [hidden email]
Subject: Re: Use of Accumulators
 
And where do you want to read the toggle back from? On the driver?

On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit <[hidden email]> wrote:

Hi,

 

Inside the transformation if there is any change for the Variable’s associated data, I want to just toggle it saying there is some change while processing the data.

 

Please let me know if we can runtime do this.

 

 

Thanks!

~Kedar Dixit

Bigdata Analytics at Persistent Systems Ltd.

 

From: Holden Karau [via Apache Spark User List] [[hidden email]]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit <[hidden email]>


Subject: Re: Use of Accumulators

 

So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?

 

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]> wrote:

Confirmed ,you can use Accumulators :)

 

Regards,

Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]> wrote:

Hi,

 

We need some way to toggle the flag of  a variable in transformation.

 

We are thinking to make use of spark  Accumulators for this purpose.

 

Can we use these as below:

 

Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0

 

In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.

 

Thanks!

 

With Regards,

~Kedar Dixit

 

[hidden email] | @kedarsdixit | M <a href="<a href="tel:&#43;91%2090499%2015588" target="_blank">tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="<a href="tel:&#43;91%2020%206703%204783" target="_blank">tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

--

Regards,

Vaquar Khan

+1 -224-436-0783

Greater Chicago

--

 


If you reply to this email, your message will be added to the discussion below:

http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html

To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

--