Pravega - Streaming as a new software defined storage primitive
This release contains significant enhancements and fixes over the previous major release line (0.12.x). This page provides an overview of most important ones. The major goal of this release was support for more storage types like GCP, Azure, stability improvements, integrity checks, CLI commands and various other fixes.
In this release, Pravega Client along with providing new API's for better user experience, also brings in some performance fixes.
Issue #2166: Improve logging for outstanding checkpoint (#6974) (#6994) (#6996) Issue #6747 : Cache segment to host mapping (#6872) Issue #6891: Fix the wrong ScaleType value in the ScalingPolicy.ScaleType enum. (#6892) Issue #6853: Expose location details of a segment. (#6854) Issue #3658: FlowId and Request Id in InvalidEventNumber Wirecommand (#6774)
Major goal of this release has been support for more storage types, specifically support for Azure and GCP, along with other important fixes.
Issue #7002: LTS - Set default value of storage.readindex.block.size to 1 GB. (#7003) (#7006) Issue #6934: LTS - Update ECS client to v3.4.5. (#6936) Issue #6896: LTS - Refactor FlakyChunkStorage by moving to top level and using wrapper pattern. (#6897) Issue #6849: LTS - Remove deprecated RollingStorage implementations from bindings project. (#6858) Issue #6790: (LTS) Implement bindings for Azure Storage (#6791) Issue #6787: LTS - Implementation of GCP binding for LTS (#6786) Issue #6032: Update ECS Object Client SDK dependency to latest version (#6738) Issue #6734: LTS - Make CHUNKED_STORAGE default in ServiceConfig and InProcPravegaCluster. (#6735) Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726) Issue #6811: LTS - Always disable lazy commit functionality for all writes to all segments (#6841)
This release of SS majorly focuses on adding various integrity checks, that help preserve the integrity of data in the ingestion pipeline if enabled. Apart from that there are also stability improvements made, which were found during the development and testing cylce of this release.
Issue #5975: ZKSegmentContainerMonitor blocks ForkJoinPool threads when queuing Segment Container Starts (#6913 ) Issue #6811: LTS - Add data integrity checks. (#6820) Issue #6804: Add integrity checks in Pravega (Tier 1) (#6805) Issue #6878: Revert Bookkeeper 4.15 upgrade in master (#6879) Issue #6772: Unnecessary segment container restarts due to Bookkeeper client re-creation mechanism (#6773) Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741) Issue #6759: OperationProcessor may be left stuck during shutdown if operationQueue is unexpectedly closed (#6758)
The major goal of this release of CLI has been recovery commands. As we can see we have added various recovery commands that can help recover a Pravega cluster in various fatal scenarios, like Table Segment Index recovery, Ability to Update LTS Snapshots and most importantly ability to restore/recover Pravega off data stored in LTS via CLI. Some of the existing commands have also been enhanced in this release.
Issue #7011: Pravega LTS based Recovery (#7012) (#7026) Issue #6809: Add new delete ledgers command (#6815) Issue #6497: Adding admin cli for viewing pending events (#6828) Issue #6817 : Pravega-cli should pick up the default environment values from tls based configuration (#6818) Issue #6800: Add a new admin CLI command to delete the reader group. (#6802) Issue #6716: Add Query Capabilities to Admin CLI DurableLog Repair Command (#6767) Issue #6712: Table Segment index recovery tool (#6753) Issue #6825: Update LTS Snapshot CLI command (#6826) Issue #6792: Removing operation versioning for table-segment recovering command (#6793)
Issue #6999: Add default Retention Policy (#7013) (#7022) Issue #6488: No retry from Controller on NoSuchSegment exception (#6783) Issue #6874: Flaky test SecureStreamMetadataTasksTest.readerGroupsTest is failing (#6887) Issue #6366: New metrics for Controller Event Processor Service (#6749)
Issue #6914: Update pods names in k8s doc (#6927) Issue #6914: Update k8s getting started guide (#6920) Issue #6910: Update quick start guide to 0.12.0 (#6917) Issue #6901: Upload MkDocs and Javadocs snapshots to pravega.io (#6904) Issue #6900: Update beta.pravega.io references to cncf.pravega.io (#6903) Issue #6899: Update gradle-mkdocs-plugin to version 2.4.0 (#6902) Issue #6862: Create document to recover container exhibiting Table Segment credits exhausted (#6886) Issue #6829: Metadata Table Segment Attribute Index recovery document (#6831) Issue #6782: Table Segment Index recovery doc (#6760) Issue #6915: Move AWS and DC/OS deployment guides (#6719) Issue #6916: Update manual installation guide (#6919) Issue #5350: Documentation for Retention and Consumption Based Retention in Pravega (#6750)
Issue #6975: Upgrading jackson-databind, micrometer and woodstox libraries (#6976) (#6979) Issue #6898: Update dependencies (#6923) Issue #6985: Upgrading commons-net, netty and jackson libraries (#6986)
Issue #6911: DelayedProcessor warning too verbose (#6912) Issue #6850: Adding Java doc for fetchStreamInfo API (#6873) Issue #6861: Refactor LargeEventTest in System Tests (#6868) Issue #6845: Remove build warnings (#6856) Issue #6864: Disable Mkdocs gradle plugin by default (#6865) Issue #6851: Create a faster, unified hash function for BufferView (#6842) Issue #6839: LTS - Resolve NullPointerException when loading Azure (#6840) Issue #6836: Temporarily revert PR 6528 (#6837) Issue #6748: Renaming getStreaminfo() and changing return type (#6812) Issue #6810: Allow Log Level to be configurable (#6814) Issue #6806: Regressions detected with Fault Injection testing in Bookkeeper 4.15 (#6807) Issue #6803: Fix standalone log location prompt (#6788) Issue #6801: Fixed spelling of bindings (#6798) Issue #6796: update gRPC version to 1.47.0 (was 1.39.0) (#6794) Issue #6732: Unexpected Pravega pod restarted in system tests (#6733) Issue #6745: Handle Controller connectivity with ZK (#6746)
This release contains significant enhancements and fixes over the previous major release line (0.12.x). This page provides an overview of most important ones. The major goal of this release was support for more storage types like GCP, Azure, stability improvements, integrity checks, CLI commands and various other fixes.
In this release, Pravega Client along with providing new API's for better user experience, also brings in some performance fixes.
Issue #2166: Improve logging for outstanding checkpoint (#6974) (#6994) (#6996) Issue #6747 : Cache segment to host mapping (#6872) Issue #6891: Fix the wrong ScaleType value in the ScalingPolicy.ScaleType enum. (#6892) Issue #6853: Expose location details of a segment. (#6854) Issue #3658: FlowId and Request Id in InvalidEventNumber Wirecommand (#6774)
Major goal of this release has been support for more storage types, specifically support for Azure and GCP, along with other important fixes.
Issue #7002: LTS - Set default value of storage.readindex.block.size to 1 GB. (#7003) (#7006) Issue #6934: LTS - Update ECS client to v3.4.5. (#6936) Issue #6896: LTS - Refactor FlakyChunkStorage by moving to top level and using wrapper pattern. (#6897) Issue #6849: LTS - Remove deprecated RollingStorage implementations from bindings project. (#6858) Issue #6790: (LTS) Implement bindings for Azure Storage (#6791) Issue #6787: LTS - Implementation of GCP binding for LTS (#6786) Issue #6032: Update ECS Object Client SDK dependency to latest version (#6738) Issue #6734: LTS - Make CHUNKED_STORAGE default in ServiceConfig and InProcPravegaCluster. (#6735) Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726) Issue #6811: LTS - Always disable lazy commit functionality for all writes to all segments (#6841)
This release of SS majorly focuses on adding various integrity checks, that help preserve the integrity of data in the ingestion pipeline if enabled. Apart from that there are also stability improvements made, which were found during the development and testing cylce of this release.
Issue #5975: ZKSegmentContainerMonitor blocks ForkJoinPool threads when queuing Segment Container Starts (#6913 ) Issue #6811: LTS - Add data integrity checks. (#6820) Issue #6804: Add integrity checks in Pravega (Tier 1) (#6805) Issue #6878: Revert Bookkeeper 4.15 upgrade in master (#6879) Issue #6772: Unnecessary segment container restarts due to Bookkeeper client re-creation mechanism (#6773) Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741) Issue #6759: OperationProcessor may be left stuck during shutdown if operationQueue is unexpectedly closed (#6758)
The major goal of this release of CLI has been recovery commands. As we can see we have added various recovery commands that can help recover a Pravega cluster in various fatal scenarios, like Table Segment Index recovery, Ability to Update LTS Snapshots and most importantly ability to restore/recover Pravega off data stored in LTS via CLI. Some of the existing commands have also been enhanced in this release.
Issue #7011: Pravega LTS based Recovery (#7012) (#7026) Issue #6809: Add new delete ledgers command (#6815) Issue #6497: Adding admin cli for viewing pending events (#6828) Issue #6817 : Pravega-cli should pick up the default environment values from tls based configuration (#6818) Issue #6800: Add a new admin CLI command to delete the reader group. (#6802) Issue #6716: Add Query Capabilities to Admin CLI DurableLog Repair Command (#6767) Issue #6712: Table Segment index recovery tool (#6753) Issue #6825: Update LTS Snapshot CLI command (#6826) Issue #6792: Removing operation versioning for table-segment recovering command (#6793)
Issue #6999: Add default Retention Policy (#7013) (#7022) Issue #6488: No retry from Controller on NoSuchSegment exception (#6783) Issue #6874: Flaky test SecureStreamMetadataTasksTest.readerGroupsTest is failing (#6887) Issue #6366: New metrics for Controller Event Processor Service (#6749)
Issue #6914: Update pods names in k8s doc (#6927) Issue #6914: Update k8s getting started guide (#6920) Issue #6910: Update quick start guide to 0.12.0 (#6917) Issue #6901: Upload MkDocs and Javadocs snapshots to pravega.io (#6904) Issue #6900: Update beta.pravega.io references to cncf.pravega.io (#6903) Issue #6899: Update gradle-mkdocs-plugin to version 2.4.0 (#6902) Issue #6862: Create document to recover container exhibiting Table Segment credits exhausted (#6886) Issue #6829: Metadata Table Segment Attribute Index recovery document (#6831) Issue #6782: Table Segment Index recovery doc (#6760) Issue #6915: Move AWS and DC/OS deployment guides (#6719) Issue #6916: Update manual installation guide (#6919) Issue #5350: Documentation for Retention and Consumption Based Retention in Pravega (#6750)
Issue #6975: Upgrading jackson-databind, micrometer and woodstox libraries (#6976) (#6979) Issue #6898: Update dependencies (#6923) Issue #6985: Upgrading commons-net, netty and jackson libraries (#6986)
Issue #6911: DelayedProcessor warning too verbose (#6912) Issue #6850: Adding Java doc for fetchStreamInfo API (#6873) Issue #6861: Refactor LargeEventTest in System Tests (#6868) Issue #6845: Remove build warnings (#6856) Issue #6864: Disable Mkdocs gradle plugin by default (#6865) Issue #6851: Create a faster, unified hash function for BufferView (#6842) Issue #6839: LTS - Resolve NullPointerException when loading Azure (#6840) Issue #6836: Temporarily revert PR 6528 (#6837) Issue #6748: Renaming getStreaminfo() and changing return type (#6812) Issue #6810: Allow Log Level to be configurable (#6814) Issue #6806: Regressions detected with Fault Injection testing in Bookkeeper 4.15 (#6807) Issue #6803: Fix standalone log location prompt (#6788) Issue #6801: Fixed spelling of bindings (#6798) Issue #6796: update gRPC version to 1.47.0 (was 1.39.0) (#6794) Issue #6732: Unexpected Pravega pod restarted in system tests (#6733) Issue #6745: Handle Controller connectivity with ZK (#6746)
This release contains significant enhancements and fixes over the previous major release line (0.12.x). This page provides an overview of most important ones. The major goal of this release was support for more storage types like GCP, Azure, stability improvements, integrity checks and various other fixes.
In this release, Pravega Client along with providing new API's for better user experience, also brings in some performance fixes.
Issue #2166: Improve logging for outstanding checkpoint (#6974) (#6994) (#6996) Issue #6747 : Cache segment to host mapping (#6872) Issue #6891: Fix the wrong ScaleType value in the ScalingPolicy.ScaleType enum. (#6892) Issue #6853: Expose location details of a segment. (#6854) Issue #3658: FlowId and Request Id in InvalidEventNumber Wirecommand (#6774)
Major goal of this release has been support for more storage types, specifically support for Azure and GCP, along with other important fixes.
Issue #7002: LTS - Set default value of storage.readindex.block.size to 1 GB. (#7003) (#7006) Issue #6934: LTS - Update ECS client to v3.4.5. (#6936) Issue #6896: LTS - Refactor FlakyChunkStorage by moving to top level and using wrapper pattern. (#6897) Issue #6849: LTS - Remove deprecated RollingStorage implementations from bindings project. (#6858) Issue #6790: (LTS) Implement bindings for Azure Storage (#6791) Issue #6787: LTS - Implementation of GCP binding for LTS (#6786) Issue #6032: Update ECS Object Client SDK dependency to latest version (#6738) Issue #6734: LTS - Make CHUNKED_STORAGE default in ServiceConfig and InProcPravegaCluster. (#6735) Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726) Issue #6811: LTS - Always disable lazy commit functionality for all writes to all segments (#6841)
This release of SS majorly focuses on adding various integrity checks, that help preserve the integrity of data in the ingestion pipeline if enabled. Apart from that there are also stability improvements made, which were found during the development and testing cylce of this release.
Issue #5975: ZKSegmentContainerMonitor blocks ForkJoinPool threads when queuing Segment Container Starts (#6913 ) Issue #6811: LTS - Add data integrity checks. (#6820) Issue #6804: Add integrity checks in Pravega (Tier 1) (#6805) Issue #6878: Revert Bookkeeper 4.15 upgrade in master (#6879) Issue #6772: Unnecessary segment container restarts due to Bookkeeper client re-creation mechanism (#6773) Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741) Issue #6759: OperationProcessor may be left stuck during shutdown if operationQueue is unexpectedly closed (#6758)
The major goal of this release of CLI has been recovery commands. As we can see we have added various recovery commands that can help recover a Pravega cluster in various fatal scenarios, like Table Segment Index recovery, Ability to Update LTS Snapshots etc. Some of the existing commands have also been enhanced in this release.
Issue #6809: Add new delete ledgers command (#6815) Issue #6497: Adding admin cli for viewing pending events (#6828) Issue #6817 : Pravega-cli should pick up the default environment values from tls based configuration (#6818) Issue #6800: Add a new admin CLI command to delete the reader group. (#6802) Issue #6716: Add Query Capabilities to Admin CLI DurableLog Repair Command (#6767) Issue #6712: Table Segment index recovery tool (#6753) Issue #6825: Update LTS Snapshot CLI command (#6826) Issue #6792: Removing operation versioning for table-segment recovering command (#6793)
Issue #6488: No retry from Controller on NoSuchSegment exception (#6783) Issue #6874: Flaky test SecureStreamMetadataTasksTest.readerGroupsTest is failing (#6887) Issue #6366: New metrics for Controller Event Processor Service (#6749)
Issue #6914: Update pods names in k8s doc (#6927) Issue #6914: Update k8s getting started guide (#6920) Issue #6910: Update quick start guide to 0.12.0 (#6917) Issue #6901: Upload MkDocs and Javadocs snapshots to pravega.io (#6904) Issue #6900: Update beta.pravega.io references to cncf.pravega.io (#6903) Issue #6899: Update gradle-mkdocs-plugin to version 2.4.0 (#6902) Issue #6862: Create document to recover container exhibiting Table Segment credits exhausted (#6886) Issue #6829: Metadata Table Segment Attribute Index recovery document (#6831) Issue #6782: Table Segment Index recovery doc (#6760) Issue #6915: Move AWS and DC/OS deployment guides (#6719) Issue #6916: Update manual installation guide (#6919) Issue #5350: Documentation for Retention and Consumption Based Retention in Pravega (#6750)
Issue #6975: Upgrading jackson-databind, micrometer and woodstox libraries (#6976) (#6979) Issue #6898: Update dependencies (#6923) Issue #6985: Upgrading commons-net, netty and jackson libraries (#6986)
Issue #6911: DelayedProcessor warning too verbose (#6912) Issue #6850: Adding Java doc for fetchStreamInfo API (#6873) Issue #6861: Refactor LargeEventTest in System Tests (#6868) Issue #6845: Remove build warnings (#6856) Issue #6864: Disable Mkdocs gradle plugin by default (#6865) Issue #6851: Create a faster, unified hash function for BufferView (#6842) Issue #6839: LTS - Resolve NullPointerException when loading Azure (#6840) Issue #6836: Temporarily revert PR 6528 (#6837) Issue #6748: Renaming getStreaminfo() and changing return type (#6812) Issue #6810: Allow Log Level to be configurable (#6814) Issue #6806: Regressions detected with Fault Injection testing in Bookkeeper 4.15 (#6807) Issue #6803: Fix standalone log location prompt (#6788) Issue #6801: Fixed spelling of bindings (#6798) Issue #6796: update gRPC version to 1.47.0 (was 1.39.0) (#6794) Issue #6732: Unexpected Pravega pod restarted in system tests (#6733) Issue #6745: Handle Controller connectivity with ZK (#6746)
This release contains significant enhancements and fixes over the previous major release line (0.11.x). This page provides an overview of most important ones. The major goal of this release was stability improvements, testing, and extension of Pravega client API. For more details, please see below:
In this release, the Pravega client provides several new APIs to provide a better user experience to developers writing Pravega applications:
Issue #4097: Serialization Framework for Synchronizer Config (#6542) Issue #1398: Validated Controller URI at ClientConfig (#6559) Issue #6567: Reader to indicate stream from which data read from (#6568) Issue #3015: added flush async in segment output stream currently testing (#6556) Issue #6620: Make fetching tokens on segment reader creation async (#6622) Issue #4816: internal executor for checkpoint (#6604) Issue #4818: Reader group resume from last checkpoint (#6586) Issue #6651: EventRead can indicate end of stream (#6673) Issue #6649: Added fetchEvent() async version in StreamManagerImpl. (#6650)
Apart from relevant fixes and configuration improvements, in Pravega 0.12 we have added "SlowStorage" as new storage type devoted to test Pravega in harsh conditions where LTS exhibits delays:
Issue #6526: LTS - Implement ChunkedSegmentStorage::listSegments (#6561) Issue #6600: LTS - Ability to simulate slow LTS (#6601) Issue #6618: LTS - Remove unnecessary pessimistic check during read of System Journals. (#6619) Issue #6688: LTS - Fix integer overflow in TruncateOperation. (#6691) Issue #6699: LTS - Prevent relocation of huge data with truncate when cluster is configured with very large max chunk size. (#6700) Issue #6674: LTS - Remove unnecessary pessimistic check in doDelete for S3 and ExtendedS3 (#6675) Issue #6658: LTS - Enable using SlowChunkStorage with SlowStorageFactory (#6660) Issue #6683: LTS - Fix thread visibility issues (#6687) Issue #6775: Cherry-pick PR #6726 to r0.12 (#6779) - (Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726))
Deprecation notice: We plan to remove support for RollingStorage in Pravega 0.13. SLTS was introduced in Pravega 0.9.0 and it has been the default storage layout since Pravega 0.10.0.
During the development and testing cycle of this release we have been able to address few relevant issues in the Segment Store and port them to the most recent release branches:
Issue #6539: Table Segment tail-caching may populate tail cache with stale data (#6538) Issue #6523: ContainerEventProcessor internal segment continuous truncations prevent StorageWriter batching (#6527) Issue #6612: Hash Table Segment may serve gets while tail-caching recovery is not yet complete (#6613) Issue #6616: Pinned Segments my not be pinned after a container restart. (#6628) Issue #6598: Avoid throttling system critical operations (#6646) Issue #6761: Cherry pick PR #6741 to r0.12 (#6764) - (Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741))
We have contributed some improvements to the Admin CLI that will provide users with more visibility and understanding of the behavior of the system:
Issue #6311: Admin CLI Commands to view Reader Information in zookeeper for a specific Controller host (#6515) Issue #6520: CLI command to list chunks from metadata for a given segment (#6524) Issue #6657: Admin CLI : Run sanity test against LTS (#6659) Issue #6505: Tweaking the use of the cluster name property in the Admin CLI (#6662) Issue #6495: DataFrameReader does not allow to read a corrupted log in BookkeeperLog repair command (#6681)
Issue #6439: Use global suffix for global segmentstore operation latency (#6492) Issue #6563: Fix OperationLogSize metric (#6564)
Issue #4801 Validate Stream name with proper regex (#6404) Issue #6250: Provide a listCompletedTransactions API on Controller (#6569) Issue #6664: Caching of Request Ids for Request Tracker causes OOM on Controller (#6665) Issue #6751: Cherry Pick PR #6745 to r0.12 (#6752) - (Issue #6745: Handle Controller connectivity with ZK (#6746))
Issue #6537: Fix flakey test: ClusterCommandsTest. (#6548) Issue #6078: LTS update default value of writer.rollover.size.bytes.max (#6503) Issue #5637: Improved Constructors of StreamManagerImpl and added a unit Test. (#6532) Issue #6565: LTS - Reduce noise in SystemJournalTests that generates java.lang.IllegalStateException: taskQueue must not be null (#6566) Issue #6498: Hardcoded port 8000 in standalone mode (#6553) Issue #6574: Unable to build Pravega via Gradle (#6575) Issue #6512: Standalone config properties should be included when create local segment store (#6521) Issue #5530: Use grpc-netty-shaded to avoid conflict with other Netty versions (#5529) Issue #6593: Fix operationlogsize metric based test (#6594) Issue #6572: Prevent subclassing as a way to violate visibility rules (#6573) Issue #6599: Fix flaky operationlog metric tests (#6602) Issue #6625: LTS - Remove flakiness from ChunkedSegmentStorageMockTests::testReport. (#6626) Issue #6655: LTS - Correctly handle Storage initialization when using SlowStorage (#6656) Issue #6609: Bump up versions of dependencies (#6610) Issue #6666: Add documentation about right-sizing Bookkeeper storage in Pravega (#6667) Issue #6677: Adding integration test to cover timestampAggregationTimeout (#6678) Issue #6682: Bump up versions of third party libraries (#6684) Issue #6711: Gradle fails when building Pravega locally in the IDE (#6710) Issue #6714: Remove queued task when recurring job is canceled. (#6715) Issue #6723: System test multiple custom configs (#6722) Issue #6713: Fixing flakiness in progressingWatermarkWithTimestampAggregationTimeout Test (#6717) Issue #6683: Disallow synchronizedList to prevent synchronization errors (#6743) Issue #6739: Changed Kubernetes Version from 8.0.0 to 15.0.1. #6739 (#6740) Issue #6780: Cherry pick #6733 to r0.12 (#6781) - (Issue #6732: Unexpected Pravega pod restarted in system tests (#6733))
This release contains significant enhancements and fixes over the previous major release line (0.11.x). This page provides an overview of most important ones. The major goal of this release was stability improvements, testing, and extension of Pravega client API. For more details, please see below:
In this release, the Pravega client provides several new APIs to provide a better user experience to developers writing Pravega applications:
Issue #4097: Serialization Framework for Synchronizer Config (#6542) Issue #1398: Validated Controller URI at ClientConfig (#6559) Issue #6567: Reader to indicate stream from which data read from (#6568) Issue #3015: added flush async in segment output stream currently testing (#6556) Issue #6620: Make fetching tokens on segment reader creation async (#6622) Issue #4816: internal executor for checkpoint (#6604) Issue #4818: Reader group resume from last checkpoint (#6586) Issue #6651: EventRead can indicate end of stream (#6673) Issue #6649: Added fetchEvent() async version in StreamManagerImpl. (#6650)
Apart from relevant fixes and configuration improvements, in Pravega 0.12 we have added "SlowStorage" as new storage type devoted to test Pravega in harsh conditions where LTS exhibits delays:
Issue #6526: LTS - Implement ChunkedSegmentStorage::listSegments (#6561) Issue #6600: LTS - Ability to simulate slow LTS (#6601) Issue #6618: LTS - Remove unnecessary pessimistic check during read of System Journals. (#6619) Issue #6688: LTS - Fix integer overflow in TruncateOperation. (#6691) Issue #6699: LTS - Prevent relocation of huge data with truncate when cluster is configured with very large max chunk size. (#6700) Issue #6674: LTS - Remove unnecessary pessimistic check in doDelete for S3 and ExtendedS3 (#6675) Issue #6658: LTS - Enable using SlowChunkStorage with SlowStorageFactory (#6660) Issue #6683: LTS - Fix thread visibility issues (#6687) Issue #6775: Cherry-pick PR #6726 to r0.12 (#6779) - (Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726))
Deprecation notice: We plan to remove support for RollingStorage in Pravega 0.13. SLTS was introduced in Pravega 0.9.0 and it has been the default storage layout since Pravega 0.10.0.
During the development and testing cycle of this release we have been able to address few relevant issues in the Segment Store and port them to the most recent release branches:
Issue #6539: Table Segment tail-caching may populate tail cache with stale data (#6538) Issue #6523: ContainerEventProcessor internal segment continuous truncations prevent StorageWriter batching (#6527) Issue #6612: Hash Table Segment may serve gets while tail-caching recovery is not yet complete (#6613) Issue #6616: Pinned Segments my not be pinned after a container restart. (#6628) Issue #6598: Avoid throttling system critical operations (#6646) Issue #6761: Cherry pick PR #6741 to r0.12 (#6764) - (Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741))
We have contributed some improvements to the Admin CLI that will provide users with more visibility and understanding of the behavior of the system:
Issue #6311: Admin CLI Commands to view Reader Information in zookeeper for a specific Controller host (#6515) Issue #6520: CLI command to list chunks from metadata for a given segment (#6524) Issue #6657: Admin CLI : Run sanity test against LTS (#6659) Issue #6505: Tweaking the use of the cluster name property in the Admin CLI (#6662) Issue #6495: DataFrameReader does not allow to read a corrupted log in BookkeeperLog repair command (#6681)
Issue #6439: Use global suffix for global segmentstore operation latency (#6492) Issue #6563: Fix OperationLogSize metric (#6564)
Issue #4801 Validate Stream name with proper regex (#6404) Issue #6250: Provide a listCompletedTransactions API on Controller (#6569) Issue #6664: Caching of Request Ids for Request Tracker causes OOM on Controller (#6665) Issue #6751: Cherry Pick PR #6745 to r0.12 (#6752) - (Issue #6745: Handle Controller connectivity with ZK (#6746))
Issue #6537: Fix flakey test: ClusterCommandsTest. (#6548) Issue #6078: LTS update default value of writer.rollover.size.bytes.max (#6503) Issue #5637: Improved Constructors of StreamManagerImpl and added a unit Test. (#6532) Issue #6565: LTS - Reduce noise in SystemJournalTests that generates java.lang.IllegalStateException: taskQueue must not be null (#6566) Issue #6498: Hardcoded port 8000 in standalone mode (#6553) Issue #6574: Unable to build Pravega via Gradle (#6575) Issue #6512: Standalone config properties should be included when create local segment store (#6521) Issue #5530: Use grpc-netty-shaded to avoid conflict with other Netty versions (#5529) Issue #6593: Fix operationlogsize metric based test (#6594) Issue #6572: Prevent subclassing as a way to violate visibility rules (#6573) Issue #6599: Fix flaky operationlog metric tests (#6602) Issue #6625: LTS - Remove flakiness from ChunkedSegmentStorageMockTests::testReport. (#6626) Issue #6655: LTS - Correctly handle Storage initialization when using SlowStorage (#6656) Issue #6609: Bump up versions of dependencies (#6610) Issue #6666: Add documentation about right-sizing Bookkeeper storage in Pravega (#6667) Issue #6677: Adding integration test to cover timestampAggregationTimeout (#6678) Issue #6682: Bump up versions of third party libraries (#6684) Issue #6711: Gradle fails when building Pravega locally in the IDE (#6710) Issue #6714: Remove queued task when recurring job is canceled. (#6715) Issue #6723: System test multiple custom configs (#6722) Issue #6713: Fixing flakiness in progressingWatermarkWithTimestampAggregationTimeout Test (#6717) Issue #6683: Disallow synchronizedList to prevent synchronization errors (#6743) Issue #6739: Changed Kubernetes Version from 8.0.0 to 15.0.1. #6739 (#6740) Issue #6780: Cherry pick #6733 to r0.12 (#6781) - (Issue #6732: Unexpected Pravega pod restarted in system tests (#6733))
This release contains significant enhancements and fixes over the previous major release line (0.11.x). This page provides an overview of most important ones. The major goal of this release was stability improvements, testing, and extension of Pravega client API. For more details, please see below:
In this release, the Pravega client provides several new APIs to provide a better user experience to developers writing Pravega applications:
Issue #4097: Serialization Framework for Synchronizer Config (#6542) Issue #1398: Validated Controller URI at ClientConfig (#6559) Issue #6567: Reader to indicate stream from which data read from (#6568) Issue #3015: added flush async in segment output stream currently testing (#6556) Issue #6620: Make fetching tokens on segment reader creation async (#6622) Issue #4816: internal executor for checkpoint (#6604) Issue #4818: Reader group resume from last checkpoint (#6586) Issue #6651: EventRead can indicate end of stream (#6673) Issue #6649: Added fetchEvent() async version in StreamManagerImpl. (#6650)
Apart from relevant fixes and configuration improvements, in Pravega 0.12 we have added "SlowStorage" as new storage type devoted to test Pravega in harsh conditions where LTS exhibits delays:
Issue #6526: LTS - Implement ChunkedSegmentStorage::listSegments (#6561) Issue #6600: LTS - Ability to simulate slow LTS (#6601) Issue #6618: LTS - Remove unnecessary pessimistic check during read of System Journals. (#6619) Issue #6688: LTS - Fix integer overflow in TruncateOperation. (#6691) Issue #6699: LTS - Prevent relocation of huge data with truncate when cluster is configured with very large max chunk size. (#6700) Issue #6674: LTS - Remove unnecessary pessimistic check in doDelete for S3 and ExtendedS3 (#6675) Issue #6658: LTS - Enable using SlowChunkStorage with SlowStorageFactory (#6660) Issue #6683: LTS - Fix thread visibility issues (#6687) Issue #6775: Cherry-pick PR #6726 to r0.12 (#6779) - (Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726))
Deprecation notice: We plan to remove support for RollingStorage in Pravega 0.13. SLTS was introduced in Pravega 0.9.0 and it has been the default storage layout since Pravega 0.10.0.
During the development and testing cycle of this release we have been able to address few relevant issues in the Segment Store and port them to the most recent release branches:
Issue #6539: Table Segment tail-caching may populate tail cache with stale data (#6538) Issue #6523: ContainerEventProcessor internal segment continuous truncations prevent StorageWriter batching (#6527) Issue #6612: Hash Table Segment may serve gets while tail-caching recovery is not yet complete (#6613) Issue #6616: Pinned Segments my not be pinned after a container restart. (#6628) Issue #6598: Avoid throttling system critical operations (#6646) Issue #6761: Cherry pick PR #6741 to r0.12 (#6764) - (Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741))
We have contributed some improvements to the Admin CLI that will provide users with more visibility and understanding of the behavior of the system:
Issue #6311 : Admin CLI Commands to view Reader Information in zookeeper for a specific Controller host (#6515) Issue #6520: CLI command to list chunks from metadata for a given segment (#6524) Issue #6657 : Admin CLI : Run sanity test against LTS (#6659) Issue #6505: Tweaking the use of the cluster name property in the Admin CLI (#6662) Issue #6495: DataFrameReader does not allow to read a corrupted log in BookkeeperLog repair command (#6681)
Issue #6439: Use global suffix for global segmentstore operation latency (#6492) Issue #6563: Fix OperationLogSize metric (#6564)
Issue #4801 Validate Stream name with proper regex (#6404) Issue #6250: Provide a listCompletedTransactions API on Controller (#6569) Issue #6664: Caching of Request Ids for Request Tracker causes OOM on Controller (#6665) Issue #6751: Cherry Pick PR #6745 to r0.12 (#6752) - (Issue #6745: Handle Controller connectivity with ZK (#6746))
Issue #6537: Fix flakey test: ClusterCommandsTest. (#6548) Issue #6078: LTS update default value of writer.rollover.size.bytes.max (#6503) Issue #5637: Improved Constructors of StreamManagerImpl and added a unit Test. (#6532) Issue #6565: LTS - Reduce noise in SystemJournalTests that generates java.lang.IllegalStateException: taskQueue must not be null (#6566) Issue #6498: Hardcoded port 8000 in standalone mode (#6553) Issue #6574: Unable to build Pravega via Gradle (#6575) Issue #6512: Standalone config properties should be included when create local segment store (#6521) Issue #5530: Use grpc-netty-shaded to avoid conflict with other Netty versions (#5529) Issue #6593: Fix operationlogsize metric based test (#6594) Issue #6572: Prevent subclassing as a way to violate visibility rules (#6573) Issue #6599: Fix flaky operationlog metric tests (#6602) Issue #6625: LTS - Remove flakiness from ChunkedSegmentStorageMockTests::testReport. (#6626) Issue #6655: LTS - Correctly handle Storage initialization when using SlowStorage (#6656) Issue #6609: Bump up versions of dependencies (#6610) Issue #6666: Add documentation about right-sizing Bookkeeper storage in Pravega (#6667) Issue #6677: Adding integration test to cover timestampAggregationTimeout (#6678) Issue #6682: Bump up versions of third party libraries (#6684) Issue #6711: Gradle fails when building Pravega locally in the IDE (#6710) Issue #6714: Remove queued task when recurring job is canceled. (#6715) Issue #6723: System test multiple custom configs (#6722) Issue #6713: Fixing flakiness in progressingWatermarkWithTimestampAggregationTimeout Test (#6717) Issue #6683: Disallow synchronizedList to prevent synchronization errors (#6743) Issue #6739: Changed Kubernetes Version from 8.0.0 to 15.0.1. #6739 (#6740) Issue #6780: Cherry pick #6733 to r0.12 (#6781) - (Issue #6732: Unexpected Pravega pod restarted in system tests (#6733))
This release contains significant enhancements and fixes over the previous major release line (0.11.x). This page provides an overview of most important ones. The major goal of this release was stability improvements, testing, and extension of Pravega client API. For more details, please see below:
In this release, the Pravega client provides several new APIs to provide a better user experience to developers writing Pravega applications:
Issue #4097: Serialization Framework for Synchronizer Config (#6542) Issue #1398: Validated Controller URI at ClientConfig (#6559) Issue #6567: Reader to indicate stream from which data read from (#6568) Issue #3015: added flush async in segment output stream currently testing (#6556) Issue #6620: Make fetching tokens on segment reader creation async (#6622) Issue #4816: internal executor for checkpoint (#6604) Issue #4818: Reader group resume from last checkpoint (#6586) Issue #6651: EventRead can indicate end of stream (#6673) Issue #6649: Added fetchEvent() async version in StreamManagerImpl. (#6650)
Apart from relevant fixes and configuration improvements, in Pravega 0.12 we have added "SlowStorage" as new storage type devoted to test Pravega in harsh conditions where LTS exhibits delays:
Issue #6526: LTS - Implement ChunkedSegmentStorage::listSegments (#6561) Issue #6600: LTS - Ability to simulate slow LTS (#6601) Issue #6618: LTS - Remove unnecessary pessimistic check during read of System Journals. (#6619) Issue #6688: LTS - Fix integer overflow in TruncateOperation. (#6691) Issue #6699: LTS - Prevent relocation of huge data with truncate when cluster is configured with very large max chunk size. (#6700) Issue #6674: LTS - Remove unnecessary pessimistic check in doDelete for S3 and ExtendedS3 (#6675) Issue #6658: LTS - Enable using SlowChunkStorage with SlowStorageFactory (#6660) Issue #6683: LTS - Fix thread visibility issues (#6687) Issue #6775: Cherry-pick PR #6726 to r0.12 (#6779) - (Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726))
During the development and testing cycle of this release we have been able to address few relevant issues in the Segment Store and port them to the most recent release branches:
Issue #6539: Table Segment tail-caching may populate tail cache with stale data (#6538) Issue #6523: ContainerEventProcessor internal segment continuous truncations prevent StorageWriter batching (#6527) Issue #6612: Hash Table Segment may serve gets while tail-caching recovery is not yet complete (#6613) Issue #6616: Pinned Segments my not be pinned after a container restart. (#6628) Issue #6598: Avoid throttling system critical operations (#6646) Issue #6761: Cherry pick PR #6741 to r0.12 (#6764) - (Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741))
We have contributed some improvements to the Admin CLI that will provide users with more visibility and understanding of the behavior of the system:
Issue #6311 : Admin CLI Commands to view Reader Information in zookeeper for a specific Controller host (#6515) Issue #6520: CLI command to list chunks from metadata for a given segment (#6524) Issue #6657 : Admin CLI : Run sanity test against LTS (#6659) Issue #6505: Tweaking the use of the cluster name property in the Admin CLI (#6662) Issue #6495: DataFrameReader does not allow to read a corrupted log in BookkeeperLog repair command (#6681)
Issue #6439: Use global suffix for global segmentstore operation latency (#6492) Issue #6563: Fix OperationLogSize metric (#6564)
Issue #4801 Validate Stream name with proper regex (#6404) Issue #6250: Provide a listCompletedTransactions API on Controller (#6569) Issue #6664: Caching of Request Ids for Request Tracker causes OOM on Controller (#6665) Issue #6751: Cherry Pick PR #6745 to r0.12 (#6752) - (Issue #6745: Handle Controller connectivity with ZK (#6746))
Issue #6537: Fix flakey test: ClusterCommandsTest. (#6548) Issue #6078: LTS update default value of writer.rollover.size.bytes.max (#6503) Issue #5637: Improved Constructors of StreamManagerImpl and added a unit Test. (#6532) Issue #6565: LTS - Reduce noise in SystemJournalTests that generates java.lang.IllegalStateException: taskQueue must not be null (#6566) Issue #6498: Hardcoded port 8000 in standalone mode (#6553) Issue #6574: Unable to build Pravega via Gradle (#6575) Issue #6512: Standalone config properties should be included when create local segment store (#6521) Issue #5530: Use grpc-netty-shaded to avoid conflict with other Netty versions (#5529) Issue #6593: Fix operationlogsize metric based test (#6594) Issue #6572: Prevent subclassing as a way to violate visibility rules (#6573) Issue #6599: Fix flaky operationlog metric tests (#6602) Issue #6625: LTS - Remove flakiness from ChunkedSegmentStorageMockTests::testReport. (#6626) Issue #6655: LTS - Correctly handle Storage initialization when using SlowStorage (#6656) Issue #6609: Bump up versions of dependencies (#6610) Issue #6666: Add documentation about right-sizing Bookkeeper storage in Pravega (#6667) Issue #6677: Adding integration test to cover timestampAggregationTimeout (#6678) Issue #6682: Bump up versions of third party libraries (#6684) Issue #6711: Gradle fails when building Pravega locally in the IDE (#6710) Issue #6714: Remove queued task when recurring job is canceled. (#6715) Issue #6723: System test multiple custom configs (#6722) Issue #6713: Fixing flakiness in progressingWatermarkWithTimestampAggregationTimeout Test (#6717) Issue #6683: Disallow synchronizedList to prevent synchronization errors (#6743) Issue #6739: Changed Kubernetes Version from 8.0.0 to 15.0.1. #6739 (#6740) Issue #6780: Cherry pick #6733 to r0.12 (#6781) - (Issue #6732: Unexpected Pravega pod restarted in system tests (#6733))
This release contains significant enhancements and fixes over the previous major release line (0.11.x). This page provides an overview of most important ones. The major goal of this release was stability improvements, testing, and extension of Pravega client API. For more details, please see below:
In this release, the Pravega client provides several new APIs to provide a better user experience to developers writing Pravega applications:
Issue #4097: Serialization Framework for Synchronizer Config (#6542) Issue #1398: Validated Controller URI at ClientConfig (#6559) Issue #6567: Reader to indicate stream from which data read from (#6568) Issue #3015: added flush async in segment output stream currently testing (#6556) Issue #6620: Make fetching tokens on segment reader creation async (#6622) Issue #4816: internal executor for checkpoint (#6604) Issue #4818: Reader group resume from last checkpoint (#6586) Issue #6651: EventRead can indicate end of stream (#6673) Issue #6649: Added fetchEvent() async version in StreamManagerImpl. (#6650)
Apart from relevant fixes and configuration improvements, in Pravega 0.12 we have added "SlowStorage" as new storage type devoted to test Pravega in harsh conditions where LTS exhibits delays:
Issue #6526: LTS - Implement ChunkedSegmentStorage::listSegments (#6561) Issue #6600: LTS - Ability to simulate slow LTS (#6601) Issue #6618: LTS - Remove unnecessary pessimistic check during read of System Journals. (#6619) Issue #6688: LTS - Fix integer overflow in TruncateOperation. (#6691) Issue #6699: LTS - Prevent relocation of huge data with truncate when cluster is configured with very large max chunk size. (#6700) Issue #6674: LTS - Remove unnecessary pessimistic check in doDelete for S3 and ExtendedS3 (#6675) Issue #6658: LTS - Enable using SlowChunkStorage with SlowStorageFactory (#6660) Issue #6683: LTS - Fix thread visibility issues (#6687) Issue #6775: Cherry-pick PR #6726 to r0.12 (#6779) - (Issue #6704: LTS - Set writer.rollover.size.bytes.max=134217728 as default (#6726))
Deprecation notice: We plan to remove support for RollingStorage in Pravega 0.13. SLTS was introduced in Pravega 0.9.0 and it has been the default storage layout since Pravega 0.10.0.
During the development and testing cycle of this release we have been able to address few relevant issues in the Segment Store and port them to the most recent release branches:
Issue #6539: Table Segment tail-caching may populate tail cache with stale data (#6538) Issue #6523: ContainerEventProcessor internal segment continuous truncations prevent StorageWriter batching (#6527) Issue #6612: Hash Table Segment may serve gets while tail-caching recovery is not yet complete (#6613) Issue #6616: Pinned Segments my not be pinned after a container restart. (#6628) Issue #6598: Avoid throttling system critical operations (#6646) Issue #6761: Cherry pick PR #6741 to r0.12 (#6764) - (Issue #6444: Duplicate DataFrames detected upon ledger rollover (#6741))
We have contributed some improvements to the Admin CLI that will provide users with more visibility and understanding of the behavior of the system:
Issue #6311 : Admin CLI Commands to view Reader Information in zookeeper for a specific Controller host (#6515) Issue #6520: CLI command to list chunks from metadata for a given segment (#6524) Issue #6657 : Admin CLI : Run sanity test against LTS (#6659) Issue #6505: Tweaking the use of the cluster name property in the Admin CLI (#6662) Issue #6495: DataFrameReader does not allow to read a corrupted log in BookkeeperLog repair command (#6681)
Issue #6439: Use global suffix for global segmentstore operation latency (#6492) Issue #6563: Fix OperationLogSize metric (#6564)
Issue #4801 Validate Stream name with proper regex (#6404) Issue #6250: Provide a listCompletedTransactions API on Controller (#6569) Issue #6664: Caching of Request Ids for Request Tracker causes OOM on Controller (#6665) Issue #6751: Cherry Pick PR #6745 to r0.12 (#6752) - (Issue #6745: Handle Controller connectivity with ZK (#6746))
Issue #6537: Fix flakey test: ClusterCommandsTest. (#6548) Issue #6078: LTS update default value of writer.rollover.size.bytes.max (#6503) Issue #5637: Improved Constructors of StreamManagerImpl and added a unit Test. (#6532) Issue #6565: LTS - Reduce noise in SystemJournalTests that generates java.lang.IllegalStateException: taskQueue must not be null (#6566) Issue #6498: Hardcoded port 8000 in standalone mode (#6553) Issue #6574: Unable to build Pravega via Gradle (#6575) Issue #6512: Standalone config properties should be included when create local segment store (#6521) Issue #5530: Use grpc-netty-shaded to avoid conflict with other Netty versions (#5529) Issue #6593: Fix operationlogsize metric based test (#6594) Issue #6572: Prevent subclassing as a way to violate visibility rules (#6573) Issue #6599: Fix flaky operationlog metric tests (#6602) Issue #6625: LTS - Remove flakiness from ChunkedSegmentStorageMockTests::testReport. (#6626) Issue #6655: LTS - Correctly handle Storage initialization when using SlowStorage (#6656) Issue #6609: Bump up versions of dependencies (#6610) Issue #6666: Add documentation about right-sizing Bookkeeper storage in Pravega (#6667) Issue #6677: Adding integration test to cover timestampAggregationTimeout (#6678) Issue #6682: Bump up versions of third party libraries (#6684) Issue #6711: Gradle fails when building Pravega locally in the IDE (#6710) Issue #6714: Remove queued task when recurring job is canceled. (#6715) Issue #6723: System test multiple custom configs (#6722) Issue #6713: Fixing flakiness in progressingWatermarkWithTimestampAggregationTimeout Test (#6717) Issue #6683: Disallow synchronizedList to prevent synchronization errors (#6743) Issue #6739: Changed Kubernetes Version from 8.0.0 to 15.0.1. #6739 (#6740) Issue #6780: Cherry pick #6733 to r0.12 (#6781) - (Issue #6732: Unexpected Pravega pod restarted in system tests (#6733))
This release contains exciting new features and significant enhancements over the previous major release line (0.10.x). This page provides an overview of major ones. The major goal of this release was performance and stability improvements. There are several changes to further stabilize the core. There is a list of features and stability/performance improvements as documented below:-
This new feature introduces an alternative code path for any event writes whose size exceeds the maximum allowed (8 MB). This may be useful for many uses cases that require writing large events to Pravega, like video analytics.
This release of Pravega includes several improvements in the Segment Store component related to flexibility and validation of its configuration. Apart from that, in Pravega 0.11 we explicitly use the new Bookie ID scheme to decouple network address from identity of Bookies that we use as Tier 1 storage:
In Pravega 0.11, we have put significant effort to improve the debuggability and repair capabilities of the system. The Pravega CLI has been one of the main recipients of our contributions, which now contains new commands to manage KV Tables, repair Tier 1 logs, inspect Reader Group contents, and manage Controller Metadata. These efforts will help admins taking care of production clusters:
This release of Pravega includes several improvements on the Controller, specially in terms of bugfixes, logging and usability. Furthermore, we have contributed a major piece of work to improve transaction performance:
Finally, apart from bugfixes, in Pravega 0.11.0 we implemented an AWS S3 storage binding for SLTS:
This release of Pravega adds support for Prometheus metrics (thanks to the contributions of our community!):