Brooklin Versions Save

What's Changed

• Fixing Deadlock Bug Between ZK Callback & Event Thread On Acquiring Coordinator Object by @shrinandthakkar in (#964)
• Fixing the CPU Intensive RemoveAll with Lists in Sticky & Load Based Partition Assignment Strategies by @shrinandthakkar in (#965)
• Optimizing By Using Precomputed Set in Partition Assignment Routine by @shrinandthakkar in (#960)
• Standardize log messages for datastream operations by @amybubu in (#961)
• Upgrade Pre Assignment Clean Up To Remove Dependency Tasks Across Hosts by @shrinandthakkar in (#952)
• Quick Optimization For Partition Assignors by @shrinandthakkar in (#956)

Full Changelog: https://github.com/linkedin/brooklin/compare/5.4.0...5.4.3

What's Changed

• BMM Restart Improvements Part 2. Followers claim their tokens after handling stop assignments by @jzakaryan in https://github.com/linkedin/brooklin/pull/921
• BMM Restart Improvements Part 3. Leader polls for unclaimed tokens before declaring a datastream STOPPED by @jzakaryan in https://github.com/linkedin/brooklin/pull/922
• BMM Restart Improvements Part 4. Edge cases, cleanup and general improvements by @jzakaryan in https://github.com/linkedin/brooklin/pull/924
• Moved the datastream stop check poll to a separate thread to avoid blocking the event queue by @jzakaryan in https://github.com/linkedin/brooklin/pull/925
• BMM Restart Improvements: Bug fixes and logging improvements by @jzakaryan in https://github.com/linkedin/brooklin/pull/926
• Bump Kafka version by @vmaheshw in https://github.com/linkedin/brooklin/pull/929
• Handling Metrics and SLA Reporting for Throughput Violating Topics via Datastream Update API Part 1 by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/928
• Added Support and Handling For Generic Headers by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/931
• Minor improvements w.r.t #928 & Releasing a newer version by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/932
• Exposed the Kafka Connector's task interrupt timeout as config by @jzakaryan in https://github.com/linkedin/brooklin/pull/933
• Separated Metrics Handling for Throughput Violating Topics by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/930
• Released New Version 5.3.2 by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/935
• Increased the size of token claims thread pool by @jzakaryan in https://github.com/linkedin/brooklin/pull/938
• Audit START/END log events for CoordinatorEvent operations by @ehoner in https://github.com/linkedin/brooklin/pull/936
• Add metrics for CoordinatorEvent queue by @ehoner in https://github.com/linkedin/brooklin/pull/937
• Minor improvements for assignment tokens feature by @jzakaryan in https://github.com/linkedin/brooklin/pull/941
• Fix Metric Registration Issue for Throughput Violations Reporting by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/942
• Fix tests and reduce the overall test execution time by @vmaheshw in https://github.com/linkedin/brooklin/pull/939
• Fixes for NoOpTransportProvider by @jzakaryan in https://github.com/linkedin/brooklin/pull/940
• Add stop method to TransportProviderAdmin by @vmaheshw in https://github.com/linkedin/brooklin/pull/944
• Log existing datastream for update and delete call by @vmaheshw in https://github.com/linkedin/brooklin/pull/948
• EventQueue metrics are not exported by @ehoner in https://github.com/linkedin/brooklin/pull/945
• Added log messages to debug assignPartitions by @jzakaryan in https://github.com/linkedin/brooklin/pull/951
• Replacing Coordinator Queue With Deque & Fixing Usage Of toMap Util by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/950
• Fixing Flaky Test testLeaderDoAssignmentForNewlyElectedLeaderFailurePathVariation by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/953
• Minor version release - 5.4.0 by @ehoner in https://github.com/linkedin/brooklin/pull/954

Full Changelog: https://github.com/linkedin/brooklin/compare/5.1.0...5.4.0

The zookeeper dependency update in version 5 includes includes breaking changes. Some of the features below were included in untagged point releases.

What's Changed

• Add support for post datastream create,update,delete and state change action by @hshukla in https://github.com/linkedin/brooklin/pull/915
• Update zookeeper dependency to import from LinkedIn published version by @surajkn in https://github.com/linkedin/brooklin/pull/917
• Set datastream status to deleting for delete call by @hshukla in https://github.com/linkedin/brooklin/pull/918
• Bump version after release by @hshukla in https://github.com/linkedin/brooklin/pull/920
• BMM Restart Improvements Part 1. Leader Coordinator Issuing Assignment Tokens by @jzakaryan in https://github.com/linkedin/brooklin/pull/919

New Contributors

• @hshukla made their first contribution in https://github.com/linkedin/brooklin/pull/915

Full Changelog: https://github.com/linkedin/brooklin/compare/4.1.0...5.1.0

What's Changed

• Refactored StickyPartitionAssignmentStrategy and implemented task estimation logic in LoadBasedPartitionAssignmentStrategy by @jzakaryan in https://github.com/linkedin/brooklin/pull/835
• Fix flaky test testConsumeFromMultipleTopics by @vmaheshw in https://github.com/linkedin/brooklin/pull/838
• Refactor Optional parameters in the constructor of strategy by @vmaheshw in https://github.com/linkedin/brooklin/pull/836
• Implemented logic to prevent tasks from having more than specified number of partitions by @jzakaryan in https://github.com/linkedin/brooklin/pull/837
• Publishing artifacts to JFrog by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/839
• Make the Throughput based assignment and task estimation based on partition assignment configurable. by @vmaheshw in https://github.com/linkedin/brooklin/pull/841
• Implemented metrics for LoadBasedPartitionAssignmentStrategy by @jzakaryan in https://github.com/linkedin/brooklin/pull/840
• Clear the CallbackStatus entry from the map in FlushlessEventProducerHandler by @vmaheshw in https://github.com/linkedin/brooklin/pull/843
• Handle the new session after session expiry by @vmaheshw in https://github.com/linkedin/brooklin/pull/770
• Fixed issue with config key by @jzakaryan in https://github.com/linkedin/brooklin/pull/847
• Print count of orphaned tasks and orphaned task locks in the log message by @surajkn in https://github.com/linkedin/brooklin/pull/844
• Datastream stop transition redesign by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/842
• Fix flaky stopping simultaneous datastream test by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/849
• Update the DatabaseChunkedReader to take the Connection as input rather than the DataSource by @somandal in https://github.com/linkedin/brooklin/pull/850
• Handle leaking TP exception in handleAssignmentChange by @surajkn in https://github.com/linkedin/brooklin/pull/845
• Migrating from AdminUtils with AdminClient by @surajkn in https://github.com/linkedin/brooklin/pull/848
• Fix running task data structure logic in AbstractKafkaConnector for failing to stop task by @surajkn in https://github.com/linkedin/brooklin/pull/851
• Removing partial Helix ZkClient dependency by @surajkn in https://github.com/linkedin/brooklin/pull/852
• Stop the tasks in parallel in AbstractKafkaConnector by @vmaheshw in https://github.com/linkedin/brooklin/pull/853
• Add additional log in LoadBasedPartitionAssignmentStrategy by @vmaheshw in https://github.com/linkedin/brooklin/pull/856
• Fixing flaky test testCreateDatastreamHappyPathDefaultRetention by @surajkn in https://github.com/linkedin/brooklin/pull/854
• Added toString() override in PartitionThroughputInfo by @jzakaryan in https://github.com/linkedin/brooklin/pull/858
• Add Stats to DatastreamTaskImpl by @vmaheshw in https://github.com/linkedin/brooklin/pull/855
• Make PartitionThroughputProvider metricsAware by @vmaheshw in https://github.com/linkedin/brooklin/pull/859
• Added base strategy metric info in LoadBasedPartitionAssignmentStrategy by @jzakaryan in https://github.com/linkedin/brooklin/pull/857
• Add alert metrics to identify that elastic task configurations require adjustment by @vmaheshw in https://github.com/linkedin/brooklin/pull/860
• Fix restartDeadTask logic when the task thread has died by @surajkn in https://github.com/linkedin/brooklin/pull/861
• Fix metric infos in PartitionThroughputProvider by @vmaheshw in https://github.com/linkedin/brooklin/pull/862
• Fix the metrics deregistration in AbstractKafkaConnector when multiple stop are called by @vmaheshw in https://github.com/linkedin/brooklin/pull/865
• Fix logging in LoadBasedPartitionAssignmentStrategy by @vmaheshw in https://github.com/linkedin/brooklin/pull/866
• Make Default byte in rate and Msg in rate configurable by @vmaheshw in https://github.com/linkedin/brooklin/pull/864
• Metrics are getting emitted in LoadBasedPartitionAssignmentStrategy only when it needs adjustment by @vmaheshw in https://github.com/linkedin/brooklin/pull/867
• Use topic level throughput information when partition level information is unavailable. by @vmaheshw in https://github.com/linkedin/brooklin/pull/871
• Fix compilation error by @vmaheshw in https://github.com/linkedin/brooklin/pull/874
• Loadbased Partition assigner not using topic level metrics to recognize partitions by @vmaheshw in https://github.com/linkedin/brooklin/pull/876
• Flushless producer supporting both comparable and non comparable offsets by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/873
• LiveInstanceProvider subscription should be done only by the leader coordinator by @vmaheshw in https://github.com/linkedin/brooklin/pull/879
• Fixed issue with missing exception message during task initialization by @jzakaryan in https://github.com/linkedin/brooklin/pull/882
• Kafka upgrade by @srinagaraj in https://github.com/linkedin/brooklin/pull/881
• skipping onPartitionsRevoked during consumer.close() call by @srinagaraj in https://github.com/linkedin/brooklin/pull/886
• Scala 2.12 upgrade by @srinagaraj in https://github.com/linkedin/brooklin/pull/895
• Upgrade avro and move jackson from codehaus to fasterxml by @vmaheshw in https://github.com/linkedin/brooklin/pull/894
• Fix topic deletion when multiple duplicate streams expire at the same time by @surajkn in https://github.com/linkedin/brooklin/pull/897
• Use 2.4.1.57 kafka version by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/901
• tests for min/max partitions per task metrics and minor code quality improvements by @ryannedolan in https://github.com/linkedin/brooklin/pull/887
• fix rebalancing-tasks bug and added tests by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/900
• Refactor Stopping Tasks On Assignment Change of Tasks by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/868
• Change python dependency in commit-msg git hook by @surajkn in https://github.com/linkedin/brooklin/pull/904
• Remove Scala Dependencies by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/905
• Introduce broadcast API to TransportProvider by @surajkn in https://github.com/linkedin/brooklin/pull/903
• Dedupe tasks on LeaderDoAssignment by @jogrogan in https://github.com/linkedin/brooklin/pull/906
• Fix Stopping Logic and Maintain Stopping Latch Counter by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/877
• Fixing test OnAssignmentChangeMultipleReassignments by @shrinandthakkar in https://github.com/linkedin/brooklin/pull/908
• Update kafka version by @jogrogan in https://github.com/linkedin/brooklin/pull/910
• Replace 101tec ZkClient with Helix ZkClient by @surajkn in https://github.com/linkedin/brooklin/pull/909
• Add retry to query retention time for destination topic by @surajkn in https://github.com/linkedin/brooklin/pull/863
• Upgrade Zookeeper version to 3.6.3 by @jogrogan in https://github.com/linkedin/brooklin/pull/913

New Contributors

• @surajkn made their first contribution in https://github.com/linkedin/brooklin/pull/844
• @srinagaraj made their first contribution in https://github.com/linkedin/brooklin/pull/881
• @ryannedolan made their first contribution in https://github.com/linkedin/brooklin/pull/887
• @jogrogan made their first contribution in https://github.com/linkedin/brooklin/pull/906

Full Changelog: https://github.com/linkedin/brooklin/compare/1.1.0...4.1.0

Release Notes

Release Notes

First open-source release