Sbt Lighter Versions Save

SBT plugin for Apache Spark on AWS EMR

1.2.0

5 years ago
  • Add BootstrapAction Support
  • Update default EMR version

1.1.0

5 years ago
  • Upgrade AWS Java API and sbt-assembly
  • Upgrade to emr-5.14.0 with Spark 2.3.0
  • Change default instance type to m4.large with 32GB EBS.
  • Add EBS settings (sparkMasterEbsSize, sparkCoreEbsSize) for Master and Core nodes.

1.0.0

6 years ago
  • Change name to sbt-lighter.
  • Set visibleToAllUsers to true by default.
  • Change sparkSubmitJob to sparkSubmit.
  • Use EMR cluster id instead of cluster name to identify the cluster, allow multiple clusters with the same name to exist.
  • Separate Master and Core instance type / bid price settings.
  • Add SBT scripted test with ScalaMock.

0.15.1

6 years ago

Add the default value of sparkAwsRegion back but set it as "changeme". This prevent the error when using SBT Configs.

0.15.0

6 years ago

Refined several setting types and provided some shortcuts

  • Changed the types of sparkSecurityGroupIds and sparkEmrConfigs from Option[T] to [T].
  • Changed sparkInstanceBidPrice to Option[Double].
  • Changed sparkEc2KeyName to sparkInstanceKeyName.
  • Added sparkEmrApplications and sparkS3LogUri.
  • Removed default value of sparkAwsRegion and sparkS3JarFolder.
  • Upgraded default EMR release to 5.11.0.

0.14.0

6 years ago

Let user modify the Spark configs for each job using

sparkSubmitConfs := Map("spark.executor.memory" -> "10G", "spark.executor.instances" -> "2")

0.13.0

6 years ago

Upgraded to SBT 1.0