SparkRDMA Versions Save

This is archive of SparkRDMA project. The new repository with RDMA shuffle acceleration for Apache Spark is here: https://github.com/Nvidia/sparkucx

3.1

5 years ago
  • Improve ODP (on demand paging) performance by using explicit MR registration
  • Spark-2.4 support.
  • Bug fixes and performance improvements.

3.0

5 years ago
  • All-new shuffle protocol that essentially removes the driver from being the bottleneck for serving shuffle blocks metadata. Increases scalability and performance when using higher numbers of mappers, reducers, executors and/or nodes.
  • Many bug fixes and performance improvements.

v2.0

6 years ago

All-new implementation of SparkRDMA, redesigned from the ground up to further increase scalability, robustness and most importantly - performance.

Among the new features and capabilities introduced in this release:

  • All-new Metadata (Map Output) fetching protocol - now allows scaling to the tens of thousands of partitions, with superior performance and recoverability
  • Software-level flow control in RdmaChannel - eliminates pause storms in the fabric
  • ODP (On-Demand Paging) support - improves memory efficiency

Attached are pre-built binaries. Please follow the README page for instructions.

v1.0

6 years ago

This is the first production ready release of SparkRDMA. Release v1.0 went through rigorous testing for performance and functionality, as well as customer-side pre-production testing. This version addresses many bugs, and it is considered the best version yet to be published.

Attached are pre-built binaries. Please follow the README page for instructions.

v0.9

6 years ago

Pre-release version of SparkRDMA, precursor for version 1.0 that will include a set of fixes and improvements.