Kafka-based Job Queue for Python
max_messages
is not set (or set to None
).Queue.using
method.kq
command line tool. Users should define an executable Python module themselves (see example here). Main reason for this was that the command line tool was not very useful: if users wanted to configure their consumers and callback functions, they had to define a configuration module and import it using the kq command line tool. This only added more room for user errors and hard-to-debug exceptions during the import, without providing much benefits over simply defining a module and calling Worker.start
directly.tests_require
and license
information in setup.py.offset_policy
to the Worker
class to allow the user to control the Kafka consumer offset reset policy (earliest
vs latest
).offset
to the kq command-line tool.proc_ttl
to the Worker
class to allow re-spawning of worker process pool.proc-ttl
to the kq
command-line tool.Queue.enqueue_with_key(key, func, *args, **kwargs)
method for adding jobs with specific Kafka message keys for sequential processing.