Kq Versions Save

Kafka-based Job Queue for Python

2.2.1

2 years ago
  • Fixed a bug where the worker terminates prematurely when max_messages is not set (or set to None).

2.2.0

3 years ago
  • Added Python type hint annotations.

2.1.0

3 years ago
  • Added support for Python 3.9 and removed support for Python 3.5.
  • Converted kq.Job and kq.Message to dataclasses from namedtuples.
  • Added type annotations (mypy friendly).
  • Moved CD/CI to GitHub Actions from TravisCI.

2.0.1

3 years ago
  • Added official support for Python 3.8.
  • Bumped up dependency versions (kafka-python>=2.0.0 and dill>=0.3.2).
  • Flush producer to ensure jobs are sent to the topic immediately.

2.0.0

5 years ago
  • Dropped support for Python 2.7 and Python 3.4. This was to keep the code cleaner, reduce future development/maintenance workload for myself, and be ready for Python 2.x phaseout (albeit a little early).
  • Completely overhauled the library and improved the API. Changes are not backward compatible. Some notable changes include:
    • Ability to inject your own kafka-python producers and consumers into KQ queues and workers respectively. See here for an example.
    • Ability to specify the timeout, key and partition when enqueueing a job with new Queue.using method.
    • Ability to inject your own serializer/deserializer. See here for more details.
    • Removed the kq command line tool. Users should define an executable Python module themselves (see example here). Main reason for this was that the command line tool was not very useful: if users wanted to configure their consumers and callback functions, they had to define a configuration module and import it using the kq command line tool. This only added more room for user errors and hard-to-debug exceptions during the import, without providing much benefits over simply defining a module and calling Worker.start directly.
  • Please check out the documentation for new usage examples and API specifications.

1.3.2

6 years ago

1.3.1

6 years ago
  • Added tests_require and license information in setup.py.

1.3.0

7 years ago
  • Added parameter offset_policy to the Worker class to allow the user to control the Kafka consumer offset reset policy (earliest vs latest).
  • Added option offset to the kq command-line tool.
  • Added documentation on logging.

1.2.0

7 years ago
  • Added parameter proc_ttl to the Worker class to allow re-spawning of worker process pool.
  • Added option proc-ttl to the kq command-line tool.

1.1.0

7 years ago
  • Added Queue.enqueue_with_key(key, func, *args, **kwargs) method for adding jobs with specific Kafka message keys for sequential processing.
  • Fixed a bug where consumer offsets are committed when jobs fail.
  • Minor documentation updates.