Kotlin Spark Api Versions Save

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x

v1.2.4

9 months ago

What's Changed

Replace type check by pattern match by @cutiechi in https://github.com/Kotlin/kotlin-spark-api/pull/193
Set Jupyter version to Java 8 by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/200
Update spark to 3.3.2 by @devongleeson in https://github.com/Kotlin/kotlin-spark-api/pull/198
Spark yarn in Jupyter notebooks by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/201
Version bumps by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/202

New Contributors

@cutiechi made their first contribution in https://github.com/Kotlin/kotlin-spark-api/pull/193
@devongleeson made their first contribution in https://github.com/Kotlin/kotlin-spark-api/pull/198

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/compare/1.2.3...v1.2.4

1.2.3

1 year ago

Adds Java 8 support https://github.com/Kotlin/kotlin-spark-api/pull/189 requested by https://github.com/Kotlin/kotlin-spark-api/issues/186, https://github.com/Kotlin/kotlin-spark-api/issues/183, https://github.com/Kotlin/kotlin-spark-api/issues/190. All modules except jupyter now target Java (1.)8 and should be compatible with Java (1.)8 servers. Let us know if you have any issues with it!

1.2.2

1 year ago

Added BigInteger support in https://github.com/Kotlin/kotlin-spark-api/pull/182 thanks to https://github.com/Kotlin/kotlin-spark-api/issues/181
New Spark versions: 3.2.3, 3.3.1, 3.2.2
New Scala versions: 2.12.17, 2.13.10
Updated Kotlin to 1.7.20
Small bugfix regarding Map encoding

1.2.1

1 year ago

What's Changed

Reorganized project: Gradle by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/168.
Reorganized project: Now supporting builds for each patch-version of Spark from 3.0.0 to 3.3.0 and Scala 2.12 / 2.13 by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/168. NOTE: The new naming scheme.
Added: UDT Support for MLlib etc. by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/162. Example.
Added: Better UD(A)F support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/152. Example
Added: seqOf() function by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/172.
Added: dfOf() function by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/162.
Added: JavaRDD extension functions by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/174. Example / Use case.
Added: Spark properties now in use-magic in Jupyter in https://github.com/Kotlin/kotlin-spark-api/issues/176.
Fixed: Reading files from Datalore by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/165.
Fixed: Encoding "isSomething" names in data classes by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/171.
Fixed: Conversion methods for Scala 2.12 / 2.13 by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/175.
Updated wiki.

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/milestone/10?closed=1.

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.0_2.13/1.2.1/jar.

For other Spark- and Scala versions, follow the naming scheme.

Read more

Blog post: Coming soon!

3.2-1.1.0

1 year ago

What's Changed

Default master value to first check the system variables. by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/127
More documentation by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/128, https://github.com/Kotlin/kotlin-spark-api/pull/130
Rdd related helpers by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/132
3.2 Encoders, supporting scala case classes by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/134, https://github.com/Kotlin/kotlin-spark-api/pull/147
Reorganizing API and adding conversions by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/136
Sorting functions for key/value Datasets by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/138
Tuple first by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/144
Streaming support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/133, https://github.com/Kotlin/kotlin-spark-api/pull/157
Kotlin Jupyter notebook support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/137, https://github.com/Kotlin/kotlin-spark-api/pull/155, https://github.com/Kotlin/kotlin-spark-api/pull/158
Created and updated wiki

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/compare/kotlin-spark-3.2-1.0.3...3.2-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.2/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

3.1-1.1.0

1 year ago

What's Changed

Default master value to first check the system variables. by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/127
More documentation by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/128, https://github.com/Kotlin/kotlin-spark-api/pull/130
Rdd related helpers by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/132
3.2 Encoders, supporting scala case classes by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/134, https://github.com/Kotlin/kotlin-spark-api/pull/147
Reorganizing API and adding conversions by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/136
Sorting functions for key/value Datasets by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/138
Tuple first by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/144
Streaming support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/133, https://github.com/Kotlin/kotlin-spark-api/pull/157
Kotlin Jupyter notebook support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/137, https://github.com/Kotlin/kotlin-spark-api/pull/155, https://github.com/Kotlin/kotlin-spark-api/pull/158
Created and updated wiki

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/compare/kotlin-spark-3.2-1.0.3...3.1-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.1/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

3.0-1.1.0

1 year ago

What's Changed

Default master value to first check the system variables. by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/127
More documentation by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/128, https://github.com/Kotlin/kotlin-spark-api/pull/130
Rdd related helpers by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/132
3.2 Encoders, supporting scala case classes by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/134, https://github.com/Kotlin/kotlin-spark-api/pull/147
Reorganizing API and adding conversions by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/136
Sorting functions for key/value Datasets by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/138
Tuple first by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/144
Streaming support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/133, https://github.com/Kotlin/kotlin-spark-api/pull/157
Kotlin Jupyter notebook support by @Jolanrensen in https://github.com/Kotlin/kotlin-spark-api/pull/137, https://github.com/Kotlin/kotlin-spark-api/pull/155, https://github.com/Kotlin/kotlin-spark-api/pull/158
Created and updated wiki

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/compare/kotlin-spark-3.2-1.0.3...3.0-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.0/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

kotlin-spark-3.2-1.0.3

2 years ago

Bug Fixes

Adds test for nullable properties in data classes
Start to fix array of generics support
Adds support for List<Map> and List<List> and relative structures
Struct type for nullable field
Perform fixes in README.md (#113)

Build

Updates build.yml

Documentation

Fixes name of spark-3 API in README

Features

UDF support
Better selectTyped behavior
Spark 3.2 support

Miscellaneous Tasks

Updates Kotlin to 1.5.30

1.0.2

2 years ago

Features

Added support for enums (https://github.com/JetBrains/kotlin-spark-api/commit/a888ea420cc9c3eb3c62b9ea035787b0a419a264)

Bug

adds reduceK function to avoid resolution ambiguity for reduce (https://github.com/JetBrains/kotlin-spark-api/commit/72fb5ea2c6aa4ce081a64751db4a9cfd2c31b0d6)
fixes invalid package name kotlinx → kotlinx (https://github.com/JetBrains/kotlin-spark-api/commit/8e7523ab04005f4f6f94cc1348631c86a3dad4cd)

CI

Adds maven wrapper (https://github.com/JetBrains/kotlin-spark-api/commit/e951424f737b32a6f1c63c99541d517d3d4774c8)

1.0.1

2 years ago

Accidentally 1.0.0 release had no content, this release contains content, release to maven central with version 1.0.1

No other changes