Kotlin Spark API

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Kotlin Spark API

Maria Khalusova

Hi folks,


We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api


Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin language features. With Kotlin Spark API, you can use Kotlin data classes and lambda expressions. 


The API also adds some helpful extension functions. For example, you can use `withCached` to perform arbitrary transformations on a Dataset and not worry about the Dataset unpersisting at the end. 


If you like Kotlin and would like to try the API, we've prepared a Quick Start Guide to help you set up all the needed dependencies in no time using either Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md


In the repo, you’ll also find a few code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples


We’d love to see your feedback in the project’s GitHub issues:  https://github.com/JetBrains/kotlin-spark-api/issues


Thanks!



Reply | Threaded
Open this post in threaded view
|

Re: Kotlin Spark API

Anwar AliKhan


Is kotlin another new language ?

GRADY BOOCH;  The United States Department of defence (DOD) is perhaps the largest user of computers in the world. By the mid-1970s, software development for its systems had reached crisis proportions: projects were often late, over budget and they often failed to meet their stated requirements. It was evident that the problems would only worsen as software development costs continued to rise exponentially. To help resolve these problems which were further compounded by the proliferation of hundreds of different languages. The DOD sponsored the development of a single, common high order programming language. The winning design was originally called the Green Language (so called because of its team colour code during the competition), and was renamed ADA



On Tue, 14 Jul 2020, 18:42 Maria Khalusova, <[hidden email]> wrote:

Hi folks,


We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api


Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin language features. With Kotlin Spark API, you can use Kotlin data classes and lambda expressions. 


The API also adds some helpful extension functions. For example, you can use `withCached` to perform arbitrary transformations on a Dataset and not worry about the Dataset unpersisting at the end. 


If you like Kotlin and would like to try the API, we've prepared a Quick Start Guide to help you set up all the needed dependencies in no time using either Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md


In the repo, you’ll also find a few code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples


We’d love to see your feedback in the project’s GitHub issues:  https://github.com/JetBrains/kotlin-spark-api/issues


Thanks!



Reply | Threaded
Open this post in threaded view
|

Re: Kotlin Spark API

Stephen Boesch
In reply to this post by Maria Khalusova
I started with scala/spark in 2012 and scala has been my go-to language for six years. But I heartily applaud this direction. Kotlin is more like a simplified Scala - with the benefits that brings - than a simplified java. I particularly like the simplified / streamlined collections classes.

Really looking forward to this development.

On Tue, 14 Jul 2020 at 10:42, Maria Khalusova <[hidden email]> wrote:

Hi folks,


We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api


Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin language features. With Kotlin Spark API, you can use Kotlin data classes and lambda expressions. 


The API also adds some helpful extension functions. For example, you can use `withCached` to perform arbitrary transformations on a Dataset and not worry about the Dataset unpersisting at the end. 


If you like Kotlin and would like to try the API, we've prepared a Quick Start Guide to help you set up all the needed dependencies in no time using either Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md


In the repo, you’ll also find a few code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples


We’d love to see your feedback in the project’s GitHub issues:  https://github.com/JetBrains/kotlin-spark-api/issues


Thanks!



Reply | Threaded
Open this post in threaded view
|

Re: Kotlin Spark API

Stephen Boesch
I just looked at the examples. https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples  These look v nice!  V concise yet flexible.  I like the ability to do inline side-effects.  E.g. caching or printing or showDs()

package org.jetbrains.spark.api.examples
import org.apache.spark.sql.Row
import org.jetbrains.spark.api.*

fun main() {
    withSpark {
        val sd = dsOf(1, 2, 3)
        sd.createOrReplaceTempView("ds")
        spark.sql("select * from ds")
                .withCached {
                    println("asList: ${toList<Int>()}")
                    println("asArray: ${toArray<Int>().contentToString()}")
                    this
                }
                .to<Int>()
                .withCached {
                    println("typed collect: " + (collect() as Array<Int>).contentToString())
                    println("type collectAsList: " + collectAsList())
                }

        dsOf(1, 2, 3)
                .map { c(it, it + 1, it + 2) }
                .to<Row>()
                .select("_1")
                .collectAsList()
                .forEach { println(it) }
    }
}

So that shows some of the niceness of kotlin: intuitive type conversion `to<Int>`/`to<Row>` and `dsOf( list)`- and also the inlining of the side effects. Overall concise and pleasant to read.


On Tue, 14 Jul 2020 at 12:18, Stephen Boesch <[hidden email]> wrote:
I started with scala/spark in 2012 and scala has been my go-to language for six years. But I heartily applaud this direction. Kotlin is more like a simplified Scala - with the benefits that brings - than a simplified java. I particularly like the simplified / streamlined collections classes.

Really looking forward to this development.

On Tue, 14 Jul 2020 at 10:42, Maria Khalusova <[hidden email]> wrote:

Hi folks,


We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api


Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin language features. With Kotlin Spark API, you can use Kotlin data classes and lambda expressions. 


The API also adds some helpful extension functions. For example, you can use `withCached` to perform arbitrary transformations on a Dataset and not worry about the Dataset unpersisting at the end. 


If you like Kotlin and would like to try the API, we've prepared a Quick Start Guide to help you set up all the needed dependencies in no time using either Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md


In the repo, you’ll also find a few code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples


We’d love to see your feedback in the project’s GitHub issues:  https://github.com/JetBrains/kotlin-spark-api/issues


Thanks!



Reply | Threaded
Open this post in threaded view
|

Re: Kotlin Spark API

Pasha Finkelshteyn
Hi Stephen,

Thank you so much for finding time for looking at our examples! Yes, we've tried to implement as clean design of API as possible and are constantly looking for ways to make it even more readable, clear and friendly.

And as Maria already stated we welcome any feedback!

On 20/07/14 01:55PM, Stephen Boesch wrote:

> I just looked at the examples.
> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> These look v nice!  V concise yet flexible.  I like the ability to do
> inline *side-effects.  *E.g. caching or printing or showDs()
>
> package org.jetbrains.spark.api.examples
> import org.apache.spark.sql.Row
> import org.jetbrains.spark.api.*
>
> fun main() {
>     withSpark {
>         val sd = dsOf(1, 2, 3)
>         sd.createOrReplaceTempView("ds")
>         spark.sql("select * from ds")
>                 .withCached {
>                     println("asList: ${toList<Int>()}")
>                     println("asArray: ${toArray<Int>().contentToString()}")
>                     this
>                 }
>                 .to<Int>()
>                 .withCached {
>                     println("typed collect: " + (collect() as
> Array<Int>).contentToString())
>                     println("type collectAsList: " + collectAsList())
>                 }
>
>         dsOf(1, 2, 3)
>                 .map { c(it, it + 1, it + 2) }
>                 .to<Row>()
>                 .select("_1")
>                 .collectAsList()
>                 .forEach { println(it) }
>     }
> }
>
>
> So that shows some of the niceness of kotlin: intuitive type conversion
> `to<Int>`/`to<Row>` and `dsOf( list)`- and also the inlining of the side
> effects. Overall concise and pleasant to read.
>
>
> On Tue, 14 Jul 2020 at 12:18, Stephen Boesch <[hidden email]> wrote:
>
> > I started with scala/spark in 2012 and scala has been my go-to language
> > for six years. But I heartily applaud this direction. Kotlin is more like a
> > simplified Scala - with the benefits that brings - than a simplified java.
> > I particularly like the simplified / streamlined collections classes.
> >
> > Really looking forward to this development.
> >
> > On Tue, 14 Jul 2020 at 10:42, Maria Khalusova <[hidden email]> wrote:
> >
> >> Hi folks,
> >>
> >> We would love your feedback on the new Kotlin Spark API that we are
> >> working on: https://github.com/JetBrains/kotlin-spark-api.
> >>
> >> Why Kotlin Spark API? Kotlin developers can already use Kotlin with the
> >> existing Apache Spark Java API, however they cannot take full advantage of
> >> Kotlin language features. With Kotlin Spark API, you can use Kotlin data
> >> classes and lambda expressions.
> >>
> >> The API also adds some helpful extension functions. For example, you can
> >> use `withCached` to perform arbitrary transformations on a Dataset and not
> >> worry about the Dataset unpersisting at the end.
> >>
> >> If you like Kotlin and would like to try the API, we've prepared a Quick
> >> Start Guide to help you set up all the needed dependencies in no time using
> >> either Maven or Gradle:
> >> https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md
> >>
> >> In the repo, you’ll also find a few code examples to get an idea of what
> >> the API looks like:
> >> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> >>
> >> We’d love to see your feedback in the project’s GitHub issues:
> >> https://github.com/JetBrains/kotlin-spark-api/issues.
> >>
> >>
> >> Thanks!
> >>
> >>
> >>
--
Regards,
Pasha

Big Data Tools @ JetBrains

signature.asc (849 bytes) Download Attachment