Incubator Uniffle Versions Save

Uniffle is a high performance, general purpose Remote Shuffle Service.

v0.8.0

5 months ago

v0.7.1

10 months ago

Apache Uniffle (Incubating) Release v0.7.1

Highlight

Improvements
- Refresh application when reading memory data to prevent application from being expired.
Major bug fixes
- Cache proxy user ugi to avoid memory leak.
- Make metric reporter usable by fixing some logic errors.
- Close ShuffleWriteClient after task was completed to exit container process.
- Make sure 'finishShuffle' invoked after all shuffle data sent.
- Update local storage metadata for all related events instead of just the first event.
- Correct a wrong metric, grpc_server_connection_number.
Minor bug fixes
- Avoid returning null in defaultUserApps when quote file doesn't config user.
- Fix LocalStorageManager divide by zero exception.

ChangeLog

[ISSUE-669] improvement: refresh application when reading memory data (#741)
Bump project version to 0.7.1-SNAPSHOT
[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
[ISSUE-796][0.7] bug: Fix the issues of MetricReporter (#821)
[MINOR][0.7] Avoid returning null in defaultUserApps when quota file doesn't config user (#822)
[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
[ISSUE-772][0.7] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
[ISSUE-715] fix(mr): The container does not exit because shuffleclient is not closed (#882)
[ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
Revert "[ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
[MINOR] fix: Fix LocalStorageManager divide by zero exception (#900)
[ISSUE-881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0 (#914)
[ISSUE-933][0.7] fix: incorrect metric grpc_server_connection_number (#941)
change version to 0.7.1 release

v0.7.0

1 year ago

Apache Uniffle (Incubating) Release v0.7.0

Highlight

Better support for Spark AQE
- Leveraging the LOCAL_ORDER data distribution to improve performance
- Estimating assignment number of shuffle servers to improve user experience
- Better assignment policy of assigning adjacent partitions to the same shuffle server to improve performance
Optimization of huge partition to improve stability of shuffle servers
More bug fixes and usability improvements of K8S operator
Add support of user quota management and more compression algorithms
Add support of spark data eviction mechanism of stage level
Some improvement of stability and performance

Changelog

Add more badges in README by @kaijchen in https://github.com/apache/incubator-uniffle/pull/219
Fix incorrect log format strings by @kaijchen in https://github.com/apache/incubator-uniffle/pull/220
Change total lines badge url to sloc.xyz in README by @kaijchen in https://github.com/apache/incubator-uniffle/pull/222
[MINOR] Fix warnings reported by lgtm by @kaijchen in https://github.com/apache/incubator-uniffle/pull/223
[MINOR] Simplify creating buffer logic by @zuston in https://github.com/apache/incubator-uniffle/pull/227
Support cancelling previous ci actions by @zuston in https://github.com/apache/incubator-uniffle/pull/225
Use the conf of shuffleNodesNumber from jobs to be as checking factor by @zuston in https://github.com/apache/incubator-uniffle/pull/208
Output the stderr and stdout to output file in startup script by @zuston in https://github.com/apache/incubator-uniffle/pull/226
[ISSUE-48][FEATURE][FOLLOW UP] Add controller component by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/214
Add more metrics about requiring read memory by @zuston in https://github.com/apache/incubator-uniffle/pull/231
Adjust the memory required times to match grpc max deadline conf by @zuston in https://github.com/apache/incubator-uniffle/pull/218
[MINOR] Fix flaky test by @jerqi in https://github.com/apache/incubator-uniffle/pull/238
[ISSUE-48][FEATURE][FOLLOW UP] Add yaml of components and crd exampes by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/236
Fix Flaky test GetShuffleReportForMultiPartTest by @leixm in https://github.com/apache/incubator-uniffle/pull/241
Set the default disk capacity to the total space by @zuston in https://github.com/apache/incubator-uniffle/pull/237
Add issue template by @jerqi in https://github.com/apache/incubator-uniffle/pull/8
[MINOR] Fix inefficient map iteration by @kaijchen in https://github.com/apache/incubator-uniffle/pull/245
Support deploy multiple shuffle servers in a single node by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/166
Fast fail when reading failed in ComposedClientReadHandler by @zuston in https://github.com/apache/incubator-uniffle/pull/213
Fix startup shell problem by @jerqi in https://github.com/apache/incubator-uniffle/pull/251
New version 0.7.0-snapshot by @jerqi in https://github.com/apache/incubator-uniffle/pull/252
[ISSUE-196] Fix flaky test about kerberos by @zuston in https://github.com/apache/incubator-uniffle/pull/250
[ISSUE-48][FEATURE][FOLLOW UP] add unit test for validating rss objects by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/248
[Improvement] Add hdfs path health check to AppBalanceSelectStorageStrategy by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/210
[TYPO] Replace Chinese colon by ASCII colon by @kaijchen in https://github.com/apache/incubator-uniffle/pull/255
Introduce startup-silent-period mechanism to avoid partial assignments by @zuston in https://github.com/apache/incubator-uniffle/pull/247
Replace DISCLAIMER with DISCLAIMER-WIP by @jerqi in https://github.com/apache/incubator-uniffle/pull/258
[ISSUE-244] Fix flaky test of CoordinatorGrpcTest.rpcMetricsTest by @zuston in https://github.com/apache/incubator-uniffle/pull/256
Fix flaky test of ClientConfManagerTest by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/260
[Refactor] Optimize creating shuffle handlers by @zuston in https://github.com/apache/incubator-uniffle/pull/259
Introduce data cleanup mechanism on stage level by @zuston in https://github.com/apache/incubator-uniffle/pull/249
[ISSUE-48][FEATURE][FOLLOW UP] add docs for operator by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/261
Fix potenial missing reads of exclude nodes by @zuston in https://github.com/apache/incubator-uniffle/pull/269
[ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long by @fpkgithub in https://github.com/apache/incubator-uniffle/pull/266
[ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle by @leixm in https://github.com/apache/incubator-uniffle/pull/274
Add enum type test about case insensitive by @zuston in https://github.com/apache/incubator-uniffle/pull/280
Support ZSTD by @zuston in https://github.com/apache/incubator-uniffle/pull/254
[ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file by @leixm in https://github.com/apache/incubator-uniffle/pull/275
Remove code quality badge and add release badge by @kaijchen in https://github.com/apache/incubator-uniffle/pull/284
[ISSUE-163][FEATURE] Write to hdfs when local disk can't be write by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/235
Upgrade Github actions for Node.js 16 by @kaijchen in https://github.com/apache/incubator-uniffle/pull/292
Fix NPE in WriteBufferManager.addRecord by @wForget in https://github.com/apache/incubator-uniffle/pull/296
Fix AbstractStorage#containsWriteHandler by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/281
Add more test cases on LocalStorageManager.selectStorage by @zuston in https://github.com/apache/incubator-uniffle/pull/298
[ISSUE-137][Improvement][AQE] Sort MapId before the data are flushed by @zuston in https://github.com/apache/incubator-uniffle/pull/293
[ISSUE-283][FEATURE] Support snappy compression/decompression by @amaliujia in https://github.com/apache/incubator-uniffle/pull/304
[ISSUE-290] Make RpcNodePort and HttpNodePort optional by @amaliujia in https://github.com/apache/incubator-uniffle/pull/305
[ISSUE-301][Subtask][Improvement][AQE] Merge continuous ShuffleDataSegment into single one by @zuston in https://github.com/apache/incubator-uniffle/pull/303
Cleanup RuntimeException and fetchRemoteStorage logic in ClientUtils by @kaijchen in https://github.com/apache/incubator-uniffle/pull/295
[ISSUE-135][FOLLOWUP][Improvement][AQE] Assign adjacent partitions to the same ShuffleServer by @leixm in https://github.com/apache/incubator-uniffle/pull/307
Correct the contributing guide link in pull-request template by @zuston in https://github.com/apache/incubator-uniffle/pull/314
Fix bug of "Comparison method violates its general contract" by @zuston in https://github.com/apache/incubator-uniffle/pull/315
[AQE][LocalOrder] Fix potenial bug when merging continuous segments by @zuston in https://github.com/apache/incubator-uniffle/pull/318
[AQE][LocalOrder] Fix wrong param of expectedTaskIds in LocalOrderSegmentSplit by @zuston in https://github.com/apache/incubator-uniffle/pull/319
[Feature] Support the estimated number of ShuffleServers required. by @leixm in https://github.com/apache/incubator-uniffle/pull/322
[Bug] Fix potenial bug when the index reading offset is greater than data length by @zuston in https://github.com/apache/incubator-uniffle/pull/320
[ISSUE-154][Improvement] Support Empty assignment to Shuffle Server by @rhh777 in https://github.com/apache/incubator-uniffle/pull/325
[Bug] Fix invalid owner of host path volumes by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/330
[ISSUE-309][FEATURE] Support ShuffleServer latency metrics. by @leixm in https://github.com/apache/incubator-uniffle/pull/327
[ISSUE-329]Catch NPE in ShuffleTaskManager#addFinishedBlockIds by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/331
[BUG] Fix wrong method name by @leixm in https://github.com/apache/incubator-uniffle/pull/335
[ISSUE-328] Cleanup unused shuffle servers after stage completed by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/334
[MINOR] Migrate RankValue to the package of the common class by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/265
[BUG] Fix incorrect spark metrics by @zuston in https://github.com/apache/incubator-uniffle/pull/324
[Improvement][LocalOrder] Add tests about keeping consistent with FixedSize when no skew optimization by @zuston in https://github.com/apache/incubator-uniffle/pull/336
[INFRA] Add k8s pipeline by @jerqi in https://github.com/apache/incubator-uniffle/pull/340
Remove unused class of RssShuffleUtils by @zuston in https://github.com/apache/incubator-uniffle/pull/345
[ISSUE-342][Improvement] Check Spark Serializer type by @chong0929 in https://github.com/apache/incubator-uniffle/pull/344
[Feature] Support user's app quota level limit by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/311
[BUG][AQE][LocalOrder] Fix the bug of missed data due to block sorting by @zuston in https://github.com/apache/incubator-uniffle/pull/347
[ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/349
[BUG] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in https://github.com/apache/incubator-uniffle/pull/350
Add collaborators by @jerqi in https://github.com/apache/incubator-uniffle/pull/351
[BUG][FOLLOWUP] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in https://github.com/apache/incubator-uniffle/pull/355
[BUG][AQE][LocalOrder] Remove check of discontinuous map task ids by @zuston in https://github.com/apache/incubator-uniffle/pull/354
[Improvement] Task fast fail once blocks fail to send by @zuston in https://github.com/apache/incubator-uniffle/pull/332
[ISSUE-228][FEATURE] Add a period local storage cleaner thread by @sfwang218 in https://github.com/apache/incubator-uniffle/pull/357
[Improvement] Optimize the use of QuotaManager by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/359
[ISSUE-339] Optimize retry logic in send shuffle data by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/361
[ISSUE-300] Make config type of RSS_CLIENT_TYPE as enum by @selectbook in https://github.com/apache/incubator-uniffle/pull/310
[ISSUE-228] Fix the problem of protobuf-java incorrect dependency at compile time by @tsface in https://github.com/apache/incubator-uniffle/pull/362
[ISSUE-124] Add fallback mechanism for blocks read inconsistent by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/276
[BUG] Fix potential memory leak when encountering disk unhealthy by @zuston in https://github.com/apache/incubator-uniffle/pull/370
[Improvement][AQE] Support getting memory data skip by upstream task ids by @zuston in https://github.com/apache/incubator-uniffle/pull/358
[ISSUE-369] Don't throw exception if blocks are corrupted but have multi replicas by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/374
[ISSUE-285][Improvement] Only use HDFS and LOCALFILE storageType in the test by @tiantingting5435 in https://github.com/apache/incubator-uniffle/pull/360
Fix typo of PreferDiffHostAssignmentStrategy by @zuston in https://github.com/apache/incubator-uniffle/pull/379
[ISSUE-376] Fix concurrency problems may occur when the ApplicationManager register app by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/382
[ISSUE-380] Refactor the flush process to fix fallback fail by @zuston in https://github.com/apache/incubator-uniffle/pull/383
[Refactor] Make coordinator class more organized by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/386
[ISSUE-392] Fix the bug in the shuffle data cleanup checker that causes false reports of disk corruption by @zuston in https://github.com/apache/incubator-uniffle/pull/393
[ISSUE-390] Print more infos after read finished by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/395
[Improvement] Skip blocks when read from memory by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/294
[Improvement] Small refactor for code quality by @advancedxy in https://github.com/apache/incubator-uniffle/pull/394
[BUG] Fix incorrect block info statistics after read finished by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/401
Revert "[Improvement] Skip blocks when read from memory (#294)" by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/403
[ISSUE-388][ISSUE-244][Bug] Fix incorrect usage of GRPCMetrics#setGauge by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/404
[MINOR] If there is no data flush to hdfs, return directly instead of throw exception by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/406
Support writing multi files of single partition to improve speed in HDFS storage by @zuston in https://github.com/apache/incubator-uniffle/pull/396
Fix incorrect metrics of event_queue_size and total_write_handler by @zuston in https://github.com/apache/incubator-uniffle/pull/411
[Improvement] Support skip memory data when use multiple replicas by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/400
[ISSUE-402] Flaky Test: QuorumTest#case1 by @Rembrant777 in https://github.com/apache/incubator-uniffle/pull/422
Fix flaky test QuotaManagerTest#testDetectUserResource by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/421
Improve README by @kaijchen in https://github.com/apache/incubator-uniffle/pull/427
Remove unused data structure and method by @zuston in https://github.com/apache/incubator-uniffle/pull/429
[FOLLOWUP] Remove unused methods in Storage interface by @zuston in https://github.com/apache/incubator-uniffle/pull/431
Flaky Test: AppBalanceSelectStorageStrategyTest#selectStorageTest by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/438
[Minor] Move GrpcServerTest to common.rpc package by @kaijchen in https://github.com/apache/incubator-uniffle/pull/439
[Minor] refactor test code by @advancedxy in https://github.com/apache/incubator-uniffle/pull/432
[Minor][Improvement] Introduce FileWriter interface for Localfile/HDFS file writer by @zuston in https://github.com/apache/incubator-uniffle/pull/444
[ISSUE-169] Support metric reporter and Support promethues push gateway by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/415
[Improvement] Avoid selecting storage which has reached the high watermark by @zuston in https://github.com/apache/incubator-uniffle/pull/424
[Bug] Fix potential negative preAllocatedSize variable by @advancedxy in https://github.com/apache/incubator-uniffle/pull/428
[Feature] add a configuration to control shuffle data flush by @advancedxy in https://github.com/apache/incubator-uniffle/pull/445
[Improvement] Refactor getPartitionRange to calculate range directly by @a-li in https://github.com/apache/incubator-uniffle/pull/447
Fix Flaky Test: AppBalanceSelectStorageStrategyTest#selectStorageTest by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/450
Fix Flaky Test: QuotaManagerTest#testDetectUserResource by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/453
[ISSUE-451][Improvement] Read HDFS data files with random sequence to distribute pressure by @zuston in https://github.com/apache/incubator-uniffle/pull/452
[ISSUE-455] Lazily create uncompressedData by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/457
[ISSUE-378][HugePartition][Part-1] Record every partition data size for one app by @zuston in https://github.com/apache/incubator-uniffle/pull/458
[ISSUE-456] Avoid removeResources for multiple times by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/459
[ISSUE-461] Support Spark 3.3 by @kaijchen in https://github.com/apache/incubator-uniffle/pull/463
[Deps] Bump slf4j to 1.7.36 to fix vulnerability in slf4j-log4j12 by @kaijchen in https://github.com/apache/incubator-uniffle/pull/464
[ISSUE-448][Feature] shuffle server report storage info by @advancedxy in https://github.com/apache/incubator-uniffle/pull/449
[ISSUE-472] Fix Flaky Test: LocalFileServerReadHandlerTest#testDataInconsistent by @zuston in https://github.com/apache/incubator-uniffle/pull/473
[Improvement] Remove some unused empty server metrics by @zuston in https://github.com/apache/incubator-uniffle/pull/474
[Improvement] Add more logs about data flush by @zuston in https://github.com/apache/incubator-uniffle/pull/482
Fix potential race condition when registering remote storage info by @zuston in https://github.com/apache/incubator-uniffle/pull/481
[Minor] Improve readability by replacing lambda with method reference by @iwangjie in https://github.com/apache/incubator-uniffle/pull/488
[ISSUE-489][Minor] Cleanup some code by @iwangjie in https://github.com/apache/incubator-uniffle/pull/490
[Test] Assume unknown blockID in LocalFileHandlerTestBase by @kaijchen in https://github.com/apache/incubator-uniffle/pull/478
[ISSUE-475][Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter by @zjf2012 in https://github.com/apache/incubator-uniffle/pull/480
[ISSUE-378][HugePartition][Part-2] Introduce memory usage limit and data flush by @zuston in https://github.com/apache/incubator-uniffle/pull/471
[ISSUE-484] Fix accidentally remove the storage of appId when unregistering partial shuffle in HdfsStorageManager by @zuston in https://github.com/apache/incubator-uniffle/pull/485
[Test] Cleanup tests with Files#createTempDir() by @kaijchen in https://github.com/apache/incubator-uniffle/pull/492
[Test] Add ConfigUtilsTest by @kaijchen in https://github.com/apache/incubator-uniffle/pull/500
[Deps] Bump protobuf to 3.19.6 to address vulnerability by @kaijchen in https://github.com/apache/incubator-uniffle/pull/499
[Minor] Make Constants final by @kaijchen in https://github.com/apache/incubator-uniffle/pull/501
Cleanup UnitConverter and improve UnitConverterTest by @kaijchen in https://github.com/apache/incubator-uniffle/pull/504
Fixes errors in doc header and operator install command by @zuston in https://github.com/apache/incubator-uniffle/pull/506
[ISSUE-378][HugePartition][Part-3] Introduce more metrics about huge partition by @zuston in https://github.com/apache/incubator-uniffle/pull/494
[ISSUE-378][HugePartition][Part-4] Supplement doc about huge partitions by @zuston in https://github.com/apache/incubator-uniffle/pull/505
[Deps] Switch to slf4j-reload4j by @kaijchen in https://github.com/apache/incubator-uniffle/pull/508
[ISSUE-512][Operator] Bump golang to 1.17 by @kaijchen in https://github.com/apache/incubator-uniffle/pull/515
[ISSUE-514] Fix flaky test: ShuffleServerGrpcTest#clearResourceTest by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/516
[ISSUE-507] Fix Flaky Test: ShuffleBufferManagerTest#cacheShuffleDataTest by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/511
[ISSUE-509] Fix Flaky Test: ShuffleBufferManagerTest#shuffleFlushThreshold by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/510
[ISSUE-479] [operator] refine operator's build system by @advancedxy in https://github.com/apache/incubator-uniffle/pull/491
[ISSUE_479][operator][followup] use exec form instead of shell form in Dockerfile by @advancedxy in https://github.com/apache/incubator-uniffle/pull/518
[SpotBugs] Set threshold to middle with exceptions by @kaijchen in https://github.com/apache/incubator-uniffle/pull/517
[ISSUE-522] [operator] pass RSS_IP to coordinator container env by @advancedxy in https://github.com/apache/incubator-uniffle/pull/523
[ISSUE-496][operator] infer resource request/limit from spec for init container by @advancedxy in https://github.com/apache/incubator-uniffle/pull/521
[SpotBugs] Remove unread protected field in Checker by @kaijchen in https://github.com/apache/incubator-uniffle/pull/520
[ISSUE-468] Put unavailable servers to the end of the list when sending shuffle data by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/470
chore: add new collaborator by @advancedxy in https://github.com/apache/incubator-uniffle/pull/535
[SpotBugs] Fix REC_CATCH_EXCEPTION by @kaijchen in https://github.com/apache/incubator-uniffle/pull/527
[ISSUE-524][operator] Upgrading rss could also be deleted by @advancedxy in https://github.com/apache/incubator-uniffle/pull/531
[ISSUE-553] Avoid removing buffer multiple times when clearing resources by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/534
[ISSUE-469][operator] feat: supports adding labels to rss pods. by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/528
[SpotBugs] Fix UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD by @kaijchen in https://github.com/apache/incubator-uniffle/pull/536
Result of mkdirs() is ignored in LocalFileWriteHandler#createBasePath() by @kaijchen in https://github.com/apache/incubator-uniffle/pull/537
[Minor] Optimize ShuffleServerInfo#hashCode by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/538
[SpotBugs] Enable SC_START_IN_CTOR check by @kaijchen in https://github.com/apache/incubator-uniffle/pull/541
[operator] fix error kind of ownerreference by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/540
[ISSUE-542] Ensure the elements of StatusCode and RssProtos.StatusCode are the same by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/543
Flaky Test: LowestIOSampleCostSelectStorageStrategyTest#selectStorageTest by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/544
[Improvement] Only report to the shuffle servers that owns the blocks by @zuston in https://github.com/apache/incubator-uniffle/pull/539
[Minor] Cleanup "throws RuntimeException" by @kaijchen in https://github.com/apache/incubator-uniffle/pull/549
[ISSUE-546] Replace ResponseStatusCode with StatusCode by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/547
[ISSUE-476][FEATURE] Respect spark.shuffle.compress configuration in Uniffle by @zjf2012 in https://github.com/apache/incubator-uniffle/pull/495
[ISSUE-545][operator] feat: support setting runtime class name and env for rss by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/548
[ISSUE #525][operator] refine svc creations by @advancedxy in https://github.com/apache/incubator-uniffle/pull/530
[#552] docs: add more doc about spark.serializer requirement by @advancedxy in https://github.com/apache/incubator-uniffle/pull/556
[#559] test: use withEnvironmentVariable to replace RssUtilsTest#setEnv by @advancedxy in https://github.com/apache/incubator-uniffle/pull/560
[Followup #249] refactor: cleanup code and unify interfaces by @kaijchen in https://github.com/apache/incubator-uniffle/pull/558
[#554] feat: infer rss base storage conf from env by @advancedxy in https://github.com/apache/incubator-uniffle/pull/555
[MINOR] Remove commiters from collaborators by @jerqi in https://github.com/apache/incubator-uniffle/pull/563
[#525][FOLLOWUP] fix: add omitempty tag by @advancedxy in https://github.com/apache/incubator-uniffle/pull/565
[#545][FOLLOWUP] update rbac rule for webhook by @advancedxy in https://github.com/apache/incubator-uniffle/pull/566
[#571] feat: skip init for empty writable base dir by @advancedxy in https://github.com/apache/incubator-uniffle/pull/573
[#567] feat: add a shuffle-server metric about read_used_buffer_size by @zuston in https://github.com/apache/incubator-uniffle/pull/568
[MINOR] test: fix assertion in tests by @kaijchen in https://github.com/apache/incubator-uniffle/pull/574
[#575] refactor: replace switch-case with EnumMap in ComposedClientReadHandler by @kaijchen in https://github.com/apache/incubator-uniffle/pull/570
[#580] chore: improve CI workflows by @kaijchen in https://github.com/apache/incubator-uniffle/pull/579
[#410] feat: support the hot reload of coordinator's configuration by @jerqi in https://github.com/apache/incubator-uniffle/pull/572
[MINOR] chore(deps): bump go-restful to 2.16.0 in operator by @dependabot in https://github.com/apache/incubator-uniffle/pull/577
[#580] chore: move deploy/kubernetes to a standalone workflow by @kaijchen in https://github.com/apache/incubator-uniffle/pull/578
[MINOR] refactor: simplify ShuffleWriteClientImpl#genServerToBlocks() by @kaijchen in https://github.com/apache/incubator-uniffle/pull/594
[MINOR] chore: remove duplicated dependency in rss-client-mr by @kaijchen in https://github.com/apache/incubator-uniffle/pull/599
[#580] chore: speed up CI workflows by @kaijchen in https://github.com/apache/incubator-uniffle/pull/602
[MINOR] chore(operator): bump prometheus/client_golang to 1.11.1 by @dependabot in https://github.com/apache/incubator-uniffle/pull/601
[MINOR] docs: correct the format of server_guide doc by @zuston in https://github.com/apache/incubator-uniffle/pull/608
[FOLLOWUP] fix: don't recreate base dir if it's already existed by @advancedxy in https://github.com/apache/incubator-uniffle/pull/622

New Contributors

@fpkgithub made their first contribution in https://github.com/apache/incubator-uniffle/pull/266
@amaliujia made their first contribution in https://github.com/apache/incubator-uniffle/pull/304
@rhh777 made their first contribution in https://github.com/apache/incubator-uniffle/pull/325
@chong0929 made their first contribution in https://github.com/apache/incubator-uniffle/pull/344
@sfwang218 made their first contribution in https://github.com/apache/incubator-uniffle/pull/357
@selectbook made their first contribution in https://github.com/apache/incubator-uniffle/pull/310
@tsface made their first contribution in https://github.com/apache/incubator-uniffle/pull/362
@tiantingting5435 made their first contribution in https://github.com/apache/incubator-uniffle/pull/360
@Rembrant777 made their first contribution in https://github.com/apache/incubator-uniffle/pull/422
@a-li made their first contribution in https://github.com/apache/incubator-uniffle/pull/447
@iwangjie made their first contribution in https://github.com/apache/incubator-uniffle/pull/488
@zjf2012 made their first contribution in https://github.com/apache/incubator-uniffle/pull/480
@dependabot made their first contribution in https://github.com/apache/incubator-uniffle/pull/577

Full Changelog: https://github.com/apache/incubator-uniffle/compare/v0.6.1...v0.7.0-rc1

v0.6.1

1 year ago

Apache Uniffle (Incubating) Release v0.6.1

Highlight

Major bug fixes
- Partition cannot be accessed in MapReduce when the reduce task number exceeds 1024.
- Get shuffle result failure caused by concurrent calls to registerShuffle.
- Inconsistent blocks caused by missing length in RssUtils#transIndexDataToSegments.
- Handle NPE in WriteBufferManager#addRecord in the same way as Spark.
- AbstractStorage#containsWriteHandler is checking the wrong Map.
- indexWriter isn't closed if exception is thrown when closing dataWriter.
- Incorrect dependency of protobuf-java at compile time.
- Potential memory leak when encountering disk unhealthy.
Minor bug fixes
- Potenial missing reads of exclude nodes.
- Incorrect contributing link in pull-request template.
- Incorrect spark metrics.

ChangeLog

[ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long (#266)
[ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle (#274)
Fix potenial missing reads of exclude nodes (#269)
[ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file (#275)
Fix NPE in WriteBufferManager.addRecord (#296)
Fix AbstractStorage#containsWriteHandler (#281)
Correct the pull-request contributing link in template (#314)
[BUG] Fix incorrect spark metrics (#324)
[ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter (#349)
[ISSUE-228] Fix the problem of protobuf-java incorrect dependency at compile time (#362)
Bump project version to 0.6.1
[BUG] Potenial memory leak when encountering disk unhealthy (#370)

v0.6.0

1 year ago

Apache Uniffle (Incubating) Release v0.6.0

Highlight

Optimize the assignment strategy
Some improvement of stability and performance
Add a plugin mechanism of SelectStorageStrategy
Add LowestIOSampleCostSelectStorageStrategy
Support Kerberos HDFS

ChangeLog

Change license owner to ASF by @kaijchen in https://github.com/apache/incubator-uniffle/pull/5
Trivial code improvements by @wForget in https://github.com/apache/incubator-uniffle/pull/7
[Minor] Store shuffleId int to be consistent with other data structure by @zuston in https://github.com/apache/incubator-uniffle/pull/10
Introduce the asList method in ConfigOptions by @zuston in https://github.com/apache/incubator-uniffle/pull/9
Rename package by @jerqi in https://github.com/apache/incubator-uniffle/pull/6
Minimize apache-rat excluded files by @kaijchen in https://github.com/apache/incubator-uniffle/pull/11
Update module names by @kaijchen in https://github.com/apache/incubator-uniffle/pull/12
Covert PartitionAssignmentInfo to static inner class by @pan3793 in https://github.com/apache/incubator-uniffle/pull/15
[Followup] Migrate to Junit5 by @zuston in https://github.com/apache/incubator-uniffle/pull/14
[Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in https://github.com/apache/incubator-uniffle/pull/16
[CI] Enable codecov report by @kaijchen in https://github.com/apache/incubator-uniffle/pull/17
Correct the config description and fix typo by @zuston in https://github.com/apache/incubator-uniffle/pull/19
Add CI and Codecov badges in README by @kaijchen in https://github.com/apache/incubator-uniffle/pull/20
[Followup] Use asList method in some existing configOptions by @zuston in https://github.com/apache/incubator-uniffle/pull/18
Move rss-integration-spark-common-test module package by @wForget in https://github.com/apache/incubator-uniffle/pull/23
[INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in https://github.com/apache/incubator-uniffle/pull/25
[TEST] Improve code coverage in rss-common by @kaijchen in https://github.com/apache/incubator-uniffle/pull/26
Remove redundant package by @wForget in https://github.com/apache/incubator-uniffle/pull/27
[CI] Switch to temurin JDK by @kaijchen in https://github.com/apache/incubator-uniffle/pull/24
[INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in https://github.com/apache/incubator-uniffle/pull/33
Bump commons-lang3 from 3.5 to 3.10 by @wForget in https://github.com/apache/incubator-uniffle/pull/28
Fix the log of incorrectly bound class by @wForget in https://github.com/apache/incubator-uniffle/pull/35
[TYPO] Fix misspelled word "integration" by @kaijchen in https://github.com/apache/incubator-uniffle/pull/34
Fix some hyperlink in README.md by @daugraph in https://github.com/apache/incubator-uniffle/pull/32
Upgrade gRPC to support Apple Silicon by @pan3793 in https://github.com/apache/incubator-uniffle/pull/13
Allow to specify custom tags to decide the assignment of servers by @zuston in https://github.com/apache/incubator-uniffle/pull/30
Optimize the bash script by @zuston in https://github.com/apache/incubator-uniffle/pull/29
[Improvement] reduce compiler warnings by @advancedxy in https://github.com/apache/incubator-uniffle/pull/46
[Chore]: document update and build time optimize by @advancedxy in https://github.com/apache/incubator-uniffle/pull/45
Supplement doc about assignment tags by @zuston in https://github.com/apache/incubator-uniffle/pull/47
[Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in https://github.com/apache/incubator-uniffle/pull/40
[improvement] Remove experimental feature with ShuffleUploader by @colinmjj in https://github.com/apache/incubator-uniffle/pull/51
[Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/49
Enable spotbugs and fix high priority bugs by @kaijchen in https://github.com/apache/incubator-uniffle/pull/38
[CI] Change default checkstyle severity to error by @kaijchen in https://github.com/apache/incubator-uniffle/pull/57
[Style] Check indentation by @kaijchen in https://github.com/apache/incubator-uniffle/pull/56
[Experimental Feature] MR Supports Remote Spill by @frankliee in https://github.com/apache/incubator-uniffle/pull/55
[Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in https://github.com/apache/incubator-uniffle/pull/58
Send commit concurrently in client side by @zuston in https://github.com/apache/incubator-uniffle/pull/59
Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in https://github.com/apache/incubator-uniffle/pull/43
[DOC] Replace Firestorm with Uniffle by @jerqi in https://github.com/apache/incubator-uniffle/pull/60
Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in https://github.com/apache/incubator-uniffle/pull/42
Log enhancement: Merge multiple logs into oneline and add more description by @zuston in https://github.com/apache/incubator-uniffle/pull/62
[TEST] Add more unit tests in rss-common by @kaijchen in https://github.com/apache/incubator-uniffle/pull/63
[MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/68
[Minor] Make config keys and default values finalized by @kaijchen in https://github.com/apache/incubator-uniffle/pull/70
[Log Improvment] Add more detailed debug info for MR client by @frankliee in https://github.com/apache/incubator-uniffle/pull/84
[Improvement] Shutdown the grpc executors pool when closing by @zuston in https://github.com/apache/incubator-uniffle/pull/83
Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in https://github.com/apache/incubator-uniffle/pull/64
[ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in https://github.com/apache/incubator-uniffle/pull/75
[Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in https://github.com/apache/incubator-uniffle/pull/82
[DOC] Remove Wechat group in README by @jerqi in https://github.com/apache/incubator-uniffle/pull/88
[Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in https://github.com/apache/incubator-uniffle/pull/91
[DOC] Update title and description in README by @kaijchen in https://github.com/apache/incubator-uniffle/pull/94
[Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/74
[IMPROVEMENT][COMMON] Fix common module code style by @jerqi in https://github.com/apache/incubator-uniffle/pull/99
[Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/72
[Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in https://github.com/apache/incubator-uniffle/pull/103
[Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in https://github.com/apache/incubator-uniffle/pull/102
[CI] Test compile in Java 11 and Java 17 by @kaijchen in https://github.com/apache/incubator-uniffle/pull/105
Sleep less time but try more times when stopping by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/112
[Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/104
[Improvement] Introduce config to customize assignment server numbers in client side by @zuston in https://github.com/apache/incubator-uniffle/pull/100
Assign partition again if registerShuffleServers failed by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/115
[ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/113
[MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple new Configuration() by @zwangsheng in https://github.com/apache/incubator-uniffle/pull/118
[Improve] Remove useless server id from StorageManagerFactory#createStorageManager by @zwangsheng in https://github.com/apache/incubator-uniffle/pull/119
[MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in https://github.com/apache/incubator-uniffle/pull/122
[Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/121
[JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in https://github.com/apache/incubator-uniffle/pull/126
Make the conf of rss.storage.basePath as list by @zuston in https://github.com/apache/incubator-uniffle/pull/130
[MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in https://github.com/apache/incubator-uniffle/pull/131
[Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/139
[MINOR] Fix flaky test testGetHostIp by @izchen in https://github.com/apache/incubator-uniffle/pull/141
[Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/147
[ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/132
[BUGFIX] Fix memory leak which cause oom by @summaryzb in https://github.com/apache/incubator-uniffle/pull/145
[Log Improvement] Output the registering/lost/exclude nodes in log by @zuston in https://github.com/apache/incubator-uniffle/pull/148
[MINOR] Tagged spark hadoop version in release package by @izchen in https://github.com/apache/incubator-uniffle/pull/149
[DOC] Migrate the coordinator doc from README to docs page by @zuston in https://github.com/apache/incubator-uniffle/pull/153
[MINOR][DOC] Remove spaces when reading file of excluded nodes by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/155
[Improvement] Filter null value when selecting remote storage in ApplicationManager by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/156
Introduce more grpc server metrics by @zuston in https://github.com/apache/incubator-uniffle/pull/150
[Improvement] Introduce a new class ShuffleTaskInfo by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/158
[ISSUE-76] Disallow sendShuffleData if requireBufferId expired by @xianjingfeng in https://github.com/apache/incubator-uniffle/pull/159
Support storing shuffle data to secured dfs cluster by @zuston in https://github.com/apache/incubator-uniffle/pull/53
[FOLLOWUP] Delete hdfs shuffle data files using proxy user by @zuston in https://github.com/apache/incubator-uniffle/pull/170
[ISSUE-48][FEATURE] Init Operator Directory by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/161
PID file name should contains program name by @zuston in https://github.com/apache/incubator-uniffle/pull/165
[BUGFIX] Fix resource leak when shuffle read by @izchen in https://github.com/apache/incubator-uniffle/pull/174
[Improvement] ShuffleBufferManager supports triggering flush according to the size of single ShuffleBuffer by @leixm in https://github.com/apache/incubator-uniffle/pull/176
[Improvement] Should match from pathToStorages when appId does not exist in appIdToStorages by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/168
[ISSUE-173][FOLLOWUP] The size of single buffer flush should reach rss.server.flush.cold.storage.threshold.size by @leixm in https://github.com/apache/incubator-uniffle/pull/178
Revert "[ISSUE-173][FOLLOWUP] The size of single buffer flush should reach rss.server.flush.cold.storage.threshold.size " by @jerqi in https://github.com/apache/incubator-uniffle/pull/179
[ISSUE-173][FOLLOWUP] The size of single buffer flush should reach rss.server.flush.cold.storage.threshold.size by @leixm in https://github.com/apache/incubator-uniffle/pull/180
[FOLLOWUP] Store app user in ShuffleTaskInfo by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/181
[ISSUE-123] Fix all test code style by @macroguo-ghy in https://github.com/apache/incubator-uniffle/pull/185
[ISSUE-48][FEATURE][FOLLOW UP] Add RemoteShuffleService CRD by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/175
[FOLLOWUP] Add the conf of rss.security.hadoop.krb5-conf.file by @zuston in https://github.com/apache/incubator-uniffle/pull/184
Fix flaky test about kerberos by @zuston in https://github.com/apache/incubator-uniffle/pull/191
[Improvement] Add optional environment variables by @izchen in https://github.com/apache/incubator-uniffle/pull/187
[MINOR] Fix some bad practices reported by spotbugs by @kaijchen in https://github.com/apache/incubator-uniffle/pull/177
[ISSUE-48][FEATURE][FOLLOW UP] Add webhook component by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/188
[Log-Improvement] Log the newly registered app id by @zuston in https://github.com/apache/incubator-uniffle/pull/193
[MINOR] Replace HashSet with ImmutableSet in configs by @kaijchen in https://github.com/apache/incubator-uniffle/pull/195
[IMPROVEMENT] Introduce the enumType in ConfigOptions by @zuston in https://github.com/apache/incubator-uniffle/pull/199
[ISSUE-48][FEATURE][FOLLOW UP] Generate informer and lister for crd by @wangao1236 in https://github.com/apache/incubator-uniffle/pull/202
[ISSUE-144] Fix flaky test RssShuffleUtilsTest#testDestroyDirectByteBuffer by @LuciferYang in https://github.com/apache/incubator-uniffle/pull/203
[Issue-194][Feature] Support spark 3.2.0 by @leixm in https://github.com/apache/incubator-uniffle/pull/201
[ISSUE-186][Feature] Use I/O cost time to select storage paths by @smallzhongfeng in https://github.com/apache/incubator-uniffle/pull/192
[Improvement][AQE] Avoid calling getShuffleResult multiple times by @leixm in https://github.com/apache/incubator-uniffle/pull/190
Fix flaky test of heartbeatTimeoutTest by @zuston in https://github.com/apache/incubator-uniffle/pull/206
[IMPROVEMENT] Add more metrics about local storage info by @zuston in https://github.com/apache/incubator-uniffle/pull/205
[MINOR][IMPROVEMENT] Return index-file size of n*SEGMENT_SIZE in HDFS reading by @zuston in https://github.com/apache/incubator-uniffle/pull/204
Add DISCLAIMER by @jerqi in https://github.com/apache/incubator-uniffle/pull/212
[TEST] Improve SimpleClusterManagerTest by @kaijchen in https://github.com/apache/incubator-uniffle/pull/216
[Minor] Modify the format of DISCLAIMER by @jerqi in https://github.com/apache/incubator-uniffle/pull/217
Add Notice and DISCLAMER file by @frankliee in https://github.com/apache/incubator-uniffle/pull/215

Credits

The release of Uniffle 0.6.0 is inseparable from the contributors of the Uniffle community. Thanks to all the community contributors!

v0.6.0-rc2

1 year ago

Apache Uniffle (Incubating) Release 0.6.0-rc1

v0.6.0-rc1

1 year ago

Release-0.6.0-incubating-rc1