Re: implementing the VectorAccumulatorParam

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: implementing the VectorAccumulatorParam

sowen
(The user@ list might be a bit better but I can see why it might look
like a dev@ question.)

Did you import org.apache.spark.mllib.linalg.Vector ? I think you are
picking up Scala's Vector class instead.


On Mon, Jun 9, 2014 at 11:57 AM, dataginjaninja
<[hidden email]> wrote:

> The  programming-guide
> <http://spark.apache.org/docs/latest/programming-guide.html>   has the
> following:
>
>     object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>       def zero(initialValue: Vector): Vector = {
>         Vector.zeros(initialValue.size)
>       }
>       def addInPlace(v1: Vector, v2: Vector): Vector = {
>         v1 += v2
>       }
>     }
>
>
> // Then, create an Accumulator of this type:
> val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)
>
> However, when I try to use this I get an error:
>
> scala> import org.apache.spark.AccumulatorParam
> import org.apache.spark.AccumulatorParam
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>      |   def zero(initialValue: Vector): Vector = {
>      |     Vector.zeros(initialValue.size)
>      |   }
>      |   def addInPlace(v1: Vector, v2: Vector): Vector = {
>      |     v1 += v2
>      |   }
>      | }
> <console>:12: error: type Vector takes type parameters
>        object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>                                                               ^
>
>
> Last thing, am I posting on the wrong list?
>
>
>
> -----
> Cheers,
>
> Stephanie
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: implementing the VectorAccumulatorParam

sowen
(BCC dev@)

The example is out of date with respect to current Vector class. The
zeros() method is on "Vectors". There is not currently a += operation
for Vector anymore.

To be fair the example doesn't claim this illustrates use of the Spark
Vector class but it did work with the now-deprecated Vector.

Make sure you still have AccumulableParam imported.

You could make a PR to adjust the example to something that works with
the newer class once you have it working.

On Mon, Jun 9, 2014 at 12:27 PM, dataginjaninja
<[hidden email]> wrote:

> New error :-(
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>      |   def zero(initialValue: Vector): Vector = {
>      |     Vector.zeros(initialValue.size)
>      |   }
>      |   def addInPlace(v1: Vector, v2: Vector): Vector = {
>      |     v1 += v2
>      |   }
>      | }
> <console>:14: error: value zeros is not a member of object
> scala.collection.immutable.Vector
>            Vector.zeros(initialValue.size)
>                   ^
> <console>:17: error: value += is not a member of
> org.apache.spark.mllib.linalg.Vector
>            v1 += v2
>
>
>
> -----
> Cheers,
>
> Stephanie
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: implementing the VectorAccumulatorParam

dataginjaninja
This post has NOT been accepted by the mailing list yet.
It appears as though not only does += not work, but there isn't any Vector addition either? Is the trick to import breeze and use their Vector library?