Pykafka Versions Save

Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance.

2.8.1-dev.1

5 years ago

Bugfixes

  • Fixed a bug causing topics to be erroneously reported as present on Topic
  • Fixed noisy logging from consumer shutdown

Miscellaneous

  • Updated dependencies: gevent, xxhash, pytest, lz4

2.8.0

5 years ago

Minor Version Features

  • Added a deserializer kwarg to consumer components to facilitate unicode support
  • Added a reset_offset_on_fetch kwarg to consumer components to support read-only consumption
  • Changed the expected type of the consumer's consumer_group kwarg to str from bytes
  • Changed the expected type of TopicDict.__getitem__'s parameter to str from bytes
  • Added a pending_timeout_ms kwarg to Producer.__init__ to allow delivery report wait timeouts
  • Added a serializer kwarg to Producer.__init__ to facilitate unicode support
  • Deprecated the generation_id and consumer_id parameters on SimpleConsumer
  • Added a partition_offsets kwarg to consumers' commit_offsets method to decouple the notions of "committed" vs "consumed" messages
  • Added an attempts kwarg to Broker.connect that controls retries during broker connection
  • Added a queue_empty_timeout_ms kwarg to Producer.__init__ that creates an "empty wait" state in the Producer when no messages are available to produce
  • Added the zookeeper_hosts kwarg to BalancedConsumer to standardize kwarg naming
  • Implemented versioning for ListOffsetRequest
  • Changed the behavior of integer arguments passed to reset_offsets

Bugfixes

  • Changed consumers to handle valid ascii strings for consumer group names instead of bytes
  • Handled NoNodeException during consumer ZK node releases
  • Used api_versions to select the version-appropriate implementation for OffsetFetchRequest
  • Adjusted synchronous production logic to avoid infinite blocking when delivery report is lost
  • Fixed a bug in FetchResponseV1 causing throttle_time to be returned as a tuple instead of an integer
  • Implemented support for all current versions of OffsetFetchRequest and OffsetFetchResponse
  • Updated some cli.print_managed_consumer_groups to be Py3 compatible
  • Updated the topic creation/deletion CLI to avoid trying to talk to 0.10.0 brokers
  • Improved error handling in Cluster.get_group_coordinator
  • Added retry logic to BrokerConnection.connect
  • Handled some nuisance errors when shutting down in handlers.py
  • Added a threading.Event instance to Producer indicating the presence of at least one message in the queue to enable nonbusy "empty waiting"
  • Added logic to SimpleConsumer.commit_offsets and OwnedPartition.build_offset_commit_request that handles user-specified offset information and sends it in requests
  • Fixed the internal logic of reset_offsets to be more predictable and user-friendly, and to better handle the case where the topic has a single log segment
  • Standardized the handling of offsets_before arguments across the API
  • Added cluster update attempts to produce() retries
  • Added a more descriptive error message on certain broker version mismatch errors

Miscellaneous

  • Used logging.NullHandler to remove nuisance logs
  • Added stock unicode serde to utils
  • Added README to pypi info
  • Updated version of Kafka used in Travis tests to 1.0.1
  • Added usage guide section on connection loss
  • Updated test harness to allow simulated killing of brokers
  • Added a giant explanatory docstring to Topic.fetch_offset_limits clarifying how the ListOffsets API works
  • Pinned gevent dependency to avoid breaking change in kazoo, which depends on it
  • Added tests for retrying broker connections
  • Added tests for user-specified offset commits
  • Added usage example on consuming the last N messages from a topic
  • Deprecated the zookeeper_connect kwarg on BalancedConsumer
  • Split the protocol.py file into multiple smaller files via the protocol module
  • Changed the lag monitor CLI to avoid resetting offsets
  • Added InvalidTopic to the list of supported exceptions
  • Updated requirement versions: lz4, pytest, xxhash
  • Removed hacky test-skipping logic from test suite
  • xfail test_update_cluster, since it occasionally fails

2.8.0-rc.1

5 years ago

This is a multi-tagging of 2.8.0-dev.5 for the purpose of indicating that this release is slated to become 2.8.0 after a bit of testing.

Changes since 2.7.0

Minor Version Features

  • Added a deserializer kwarg to consumer components to facilitate unicode support
  • Added a reset_offset_on_fetch kwarg to consumer components to support read-only consumption
  • Changed the expected type of the consumer's consumer_group kwarg to str from bytes
  • Changed the expected type of TopicDict.__getitem__'s parameter to str from bytes
  • Added a pending_timeout_ms kwarg to Producer.__init__ to allow delivery report wait timeouts
  • Added a serializer kwarg to Producer.__init__ to facilitate unicode support
  • Deprecated the generation_id and consumer_id parameters on SimpleConsumer
  • Added a partition_offsets kwarg to consumers' commit_offsets method to decouple the notions of "committed" vs "consumed" messages
  • Added an attempts kwarg to Broker.connect that controls retries during broker connection
  • Added a queue_empty_timeout_ms kwarg to Producer.__init__ that creates an "empty wait" state in the Producer when no messages are available to produce
  • Added the zookeeper_hosts kwarg to BalancedConsumer to standardize kwarg naming
  • Implemented versioning for ListOffsetRequest
  • Changed the behavior of integer arguments passed to reset_offsets

Bugfixes

  • Changed consumers to handle valid ascii strings for consumer group names instead of bytes
  • Handled NoNodeException during consumer ZK node releases
  • Used api_versions to select the version-appropriate implementation for OffsetFetchRequest
  • Adjusted synchronous production logic to avoid infinite blocking when delivery report is lost
  • Fixed a bug in FetchResponseV1 causing throttle_time to be returned as a tuple instead of an integer
  • Implemented support for all current versions of OffsetFetchRequest and OffsetFetchResponse
  • Updated some cli.print_managed_consumer_groups to be Py3 compatible
  • Updated the topic creation/deletion CLI to avoid trying to talk to 0.10.0 brokers
  • Improved error handling in Cluster.get_group_coordinator
  • Added retry logic to BrokerConnection.connect
  • Handled some nuisance errors when shutting down in handlers.py
  • Added a threading.Event instance to Producer indicating the presence of at least one message in the queue to enable nonbusy "empty waiting"
  • Added logic to SimpleConsumer.commit_offsets and OwnedPartition.build_offset_commit_request that handles user-specified offset information and sends it in requests
  • Fixed the internal logic of reset_offsets to be more predictable and user-friendly, and to better handle the case where the topic has a single log segment
  • Standardized the handling of offsets_before arguments across the API
  • Added cluster update attempts to produce() retries
  • Added a more descriptive error message on certain broker version mismatch errors

Miscellaneous

  • Used logging.NullHandler to remove nuisance logs
  • Added stock unicode serde to utils
  • Added README to pypi info
  • Updated version of Kafka used in Travis tests to 1.0.1
  • Added usage guide section on connection loss
  • Updated test harness to allow simulated killing of brokers
  • Added a giant explanatory docstring to Topic.fetch_offset_limits clarifying how the ListOffsets API works
  • Pinned gevent dependency to avoid breaking change in kazoo, which depends on it
  • Added tests for retrying broker connections
  • Added tests for user-specified offset commits
  • Added usage example on consuming the last N messages from a topic
  • Deprecated the zookeeper_connect kwarg on BalancedConsumer
  • Split the protocol.py file into multiple smaller files via the protocol module
  • Changed the lag monitor CLI to avoid resetting offsets
  • Added InvalidTopic to the list of supported exceptions
  • Updated requirement versions: lz4, pytest, xxhash
  • Removed hacky test-skipping logic from test suite
  • xfail test_update_cluster, since it occasionally fails

2.8.0-dev.5

5 years ago

(Duplicate of 2.8.0-dev.4)

2.8.0-dev.4

5 years ago

Bugfixes

  • Added a more descriptive error message on certain broker version mismatch errors

Miscellaneous

  • xfail test_update_cluster, since it occasionally fails

2.8.0-dev.3

5 years ago

Minor Version Features

  • Added the zookeeper_hosts kwarg to BalancedConsumer to standardize kwarg naming
  • Implemented versioning for ListOffsetRequest
  • Changed the behavior of integer arguments passed to reset_offsets

Bugfixes

  • Fixed the internal logic of reset_offsets to be more predictable and user-friendly, and to better handle the case where the topic has a single log segment
  • Standardized the handling of offsets_before arguments across the API
  • Added cluster update attempts to produce() retries

Miscellaneous

  • Added usage example on consuming the last N messages from a topic
  • Deprecated the zookeeper_connect kwarg on BalancedConsumer
  • Split the protocol.py file into multiple smaller files via the protocol module
  • Changed the lag monitor CLI to avoid resetting offsets
  • Added InvalidTopic to the list of supported exceptions
  • Updated requirement versions: lz4, pytest, xxhash
  • Removed hacky test-skipping logic from test suite

2.8.0-dev.2

5 years ago

Minor Version Features

  • Added a partition_offsets kwarg to consumers' commit_offsets method to decouple the notions of "committed" vs "consumed" messages
  • Added an attempts kwarg to Broker.connect that controls retries during broker connection
  • Added a queue_empty_timeout_ms kwarg to Producer.__init__ that creates an "empty wait" state in the Producer when no messages are available to produce

Bugfixes

  • Updated some cli.print_managed_consumer_groups to be Py3 compatible
  • Updated the topic creation/deletion CLI to avoid trying to talk to 0.10.0 brokers
  • Improved error handling in Cluster.get_group_coordinator
  • Added retry logic to BrokerConnection.connect
  • Handled some nuisance errors when shutting down in handlers.py
  • Added a threading.Event instance to Producer indicating the presence of at least one message in the queue to enable nonbusy "empty waiting"
  • Added logic to SimpleConsumer.commit_offsets and OwnedPartition.build_offset_commit_request that handles user-specified offset information and sends it in requests

Miscellaneous

  • Updated version of Kafka used in Travis tests to 1.0.1
  • Added usage guide section on connection loss
  • Updated test harness to allow simulated killing of brokers
  • Added a giant explanatory docstring to Topic.fetch_offset_limits clarifying how the ListOffsets API works
  • Pinned gevent dependency to avoid breaking change in kazoo, which depends on it
  • Added tests for retrying broker connections
  • Added tests for user-specified offset commits

2.8.0-dev.1

6 years ago

Minor Version Features

  • Added a deserializer kwarg to consumer components to facilitate unicode support
  • Added a reset_offset_on_fetch kwarg to consumer components to support read-only consumption
  • Changed the expected type of the consumer's consumer_group kwarg to str from bytes
  • Changed the expected type of TopicDict.__getitem__'s parameter to str from bytes
  • Added a pending_timeout_ms kwarg to Producer.__init__ to allow delivery report wait timeouts
  • Added a serializer kwarg to Producer.__init__ to facilitate unicode support
  • Deprecated the generation_id and consumer_id parameters on SimpleConsumer

Bugfixes

  • Changed consumers to handle valid ascii strings for consumer group names instead of bytes
  • Handled NoNodeException during consumer ZK node releases
  • Used api_versions to select the version-appropriate implementation for OffsetFetchRequest
  • Adjusted synchronous production logic to avoid infinite blocking when delivery report is lost
  • Fixed a bug in FetchResponseV1 causing throttle_time to be returned as a tuple instead of an integer
  • Implemented support for all current versions of OffsetFetchRequest and OffsetFetchResponse

Miscellaneous

  • Used logging.NullHandler to remove nuisance logs
  • Added stock unicode serde to utils
  • Added README to pypi info

2.7.0

6 years ago

Minor Version Features

  • Added a broker_version kwarg to Broker.__init__ for the purpose of setting api_version in FetchResponse
  • Added a topic_name argument to Broker.join_group for use in protocol metadata, visible via the Administrative API
  • Added a function print_managed_consumer_groups to the CLI
  • Added a timestamp kwarg to Producer.produce to pass on messages when the broker supports newer message formats
  • Changed Producer.produce to return the produced Message instance
  • Added protocol_version and timestamp kwargs to Message
  • Added support for the fetch_error_backoff_ms kwarg on SimpleConsumer
  • Added an unblock_event kwarg to SimpleConsumer.consume used to notify the consumer that its parent BalancedConsumer is in the process of rebalancing
  • Added a general-purpose cleanup function to SimpleConsumer
  • Added a membership_protocol kwarg to BalancedConsumer that allows switchable and user-defined membership protocols to be used
  • Implemented GroupMembershipProtocol objects for the two standard partition assignment strategies
  • Added an api_versions kwarg to Broker to facilitate switchable API protocol versions
  • Added support for all versions of the MetadataRequest to Broker
  • Added the controller_broker attribute to Cluster
  • Added create_topics and delete_topics to Broker
  • Added fetch_api_versions to Broker and Cluster
  • Added a CLI for creating and deleting topics on the cluster to kafka_tools
  • Added support for LZ4 compression to the Producer and SimpleConsumer

Bug Fixes

  • Added an Event that notifies the internal SimpleConsumer of a BalancedConsumer that a rebalance is in progress, fixing a bug causing partitions to be unreleased
  • Fixed a bug causing busywaiting in the BalancedConsumer when there are no partitions available
  • Updated the protocol implementation to send non-empty GroupMembershipProtocol objects and become compatible with the Administrative API
  • Fixed a bytestring bug causing kafka_tools.reset_offsets not to work in python 3
  • Added a separate retry limit on connections to the offset manager
  • Improved logging on socket errors
  • Fixed a bug causing API version not to be passed on certain requests
  • Handled new MessageSet compression scheme in API v1
  • Fixed a bug in rdkafka.SimpleConsumer causing exceptions not to be raised from worker threads
  • Fixed a bug causing fetch_offsets not to raise exceptions under certain conditions when it should
  • Adjusted Cluster to become aware of supported API versions immediately upon instantiation
  • Refactored code in Cluster related to metadata requests to make logic reusable for pre-bootstrap communication with the cluster
  • Added the ability to pass arguments to protocol.Response instances when waiting on a future
  • Adjusted the RandomPartitioner to avoid actually calling random.choice to improve performance
  • Removed some calls in Producer.procuce to isinstance to improve performance
  • Simplified retry logic in SimpleConsumer.fetch_offsets

Miscellaneous

  • Separated gevent tests from other builds in Travis
  • Made dependency on gevent optional
  • Added a convenient CLI entry point via __main__
  • Fixed exception naming convention to align with naming in the broker
  • Avoided building the rdkafka extension on platforms that don't support it
  • Fixed a bug in test harness causing some tests not to be inherited from parent classes
  • Used sudo: required to get around dead Travis machines
  • Upgraded Travis tests to use Kafka 1.0.0
  • Added Code of Conduct
  • Documented release process
  • Made PyKafka available via conda-forge
  • Fleshed out the beginning of the usage guide
  • Made kafka_instance fetch its binary from archive.apache.org instead of mirror.reverse.net because the latter removed old versions of Kafka

2.7.0-dev.2

6 years ago

Minor Version Features

  • Added a membership_protocol kwarg to BalancedConsumer that allows switchable and user-defined membership protocols to be used
  • Implemented GroupMembershipProtocol objects for the two standard partition assignment strategies
  • Added an api_versions kwarg to Broker to facilitate switchable API protocol versions
  • Added support for all versions of the MetadataRequest to Broker
  • Added the controller_broker attribute to Cluster
  • Added create_topics and delete_topics to Broker
  • Added fetch_api_versions to Broker and Cluster
  • Added a CLI for creating and deleting topics on the cluster to kafka_tools
  • Added support for LZ4 compression to the Producer and SimpleConsumer

Bug Fixes

  • Adjusted Cluster to become aware of supported API versions immediately upon instantiation
  • Refactored code in Cluster related to metadata requests to make logic reusable for pre-bootstrap communication with the cluster
  • Added the ability to pass arguments to protocol.Response instances when waiting on a future
  • Adjusted the RandomPartitioner to avoid actually calling random.choice to improve performance
  • Removed some calls in Producer.procuce to isinstance to improve performance
  • Simplified retry logic in SimpleConsumer.fetch_offsets

Miscellaneous

  • Used sudo: required to get around dead Travis machines
  • Upgraded Travis tests to use Kafka 1.0.0
  • Added Code of Conduct
  • Documented release process
  • Made PyKafka available via conda-forge
  • Fleshed out the beginning of the usage guide
  • Made kafka_instance fetch its binary from archive.apache.org instead of mirror.reverse.net because the latter removed old versions of Kafka