Streamparse Versions Save

This release simply makes it possible to override more settings that are in config.json at the Topology level. You can now add config = {'virtualenv_flags': '-p /path/to/python3'} to have some topologies in your project using one version of Python and others using another (Issue #399, PR #402)

This release just adds options to pre_submit_hook and post_submit_hook arguments. This is mostly so you can use the storm workers list inside hooks. (PR #396)

This release simply adds a new feature where the storm.workers.list topology configuration option is now set when you submit a topology, so if some part of your topology needs to know the list of Storm workers, you do not need to resort to connecting to Nimbus with each executor to find it out. (PR #395)

Another small release, but fixes issues with sparse tail and sparse remove_logs.

Features

• Can now specify the number of simultaneous SSH connections to allow when updating virtualenvs or accessing/removing logs from your workers with the --pool_size argument. (PR #393)
• The user that is used to remove logs from the Storm workers with sparse remove_logs can now be specified with --user (PR #393)

Fixes

• sparse tail and sparse remove_logs do a much better job of only finding the logs that relate to your specified topology in Storm 1.x (PR #393)
• sparse run will no longer crash if you have par set to a dict in your topology. (bafb72b)

Fix issue where sparse run would not work without Nimbus properly configured in config.json (Issue #391, PR #392)

Small release, but a big convenience feature was added.

Features

• No longer need to specify workers in your config.json! They will be looked up dynamically by communicating with your Nimbus server. If for some reason you would like to restrict where streamparse creates virtualenvs to a subset of your workers, you can still specify the worker list in config.json and that will take precedence. (PR #389)
• Added better error messages for when util.get_ui_jsons fails (commit b2a8219)
• Can now pass config_file file-like objects to util.get_config for if you need to do something like retrieve the config at runtime from a wheel. Not very common, but it is now supported. (PR #390)
• Removed the now unused decorators module.

Fixes

• Fixed a documentation issue where the FAQ still referenced the old ext module that was removed a long time ago (Issue #388)

This has bunch of bugfixes, but a few new features too.

Features

• Added virtualenv_name as a setting in config.json for users who want to reuse the same virtualenv for multiple topologies. (Issue #371, PR #373)
• Support ruamel.yaml>=0.15 (PR #379)
• You can now inherit Storm Thrift types directly from streamparse.thrift instead of needing the extra from streamparse.thrift import storm_thrift (PR #380)
• Added --timeout option to sparse run, sparse list, and sparse submit so that you can control how long to wait for Nimbus to respond before timing out. This is very useful on slow connections. (Issue #341, PR #381)

Fixes

• Fixed failing fabfile.py and tasks.py imports in get_user_tasks with Python 3. (Issue #376, PR #378)
• Made Storm version parsing a little more lenient to fix some rare crashes (Issue #356)
• Added documentation on how to pass string arguments to Java constructors (Issue #357)
• Added documentation on how to create topologies with cycles in them (Issue #339)
• Fixed issue where we were writing invalid YAML when a string contained a colon. (Issue #361)
• We now pass on the serialized properly with sparse run (Issue #340, PR #382)
• sparse no longer crashes on Windows (Issues #346 and pystorm/pystorm#40, PR pystorm/pystorm#45)

This release brings some compatibility fixes for Storm 1.0.3+.

Features

• Propagate topology parallelism settings to Flux when using sparse run (Issue #364, PR #365)
• Pass component configs on to Flux when using sparse run (PR #266)

Fixes

• Make sure output shows up when using sparse run (PR #363)
• Handle new resources nesting added in Storm 1.0.3 (Issue #362, PR #366)

This release fixes a few bugs and adds a few new features that require pystorm 3.1.0 or greater.

Features

• Added a ReliableSpout implementation that can be used to have spouts that will automatically replay failed tuples up to a specified number of times before giving up on them. (pystorm/pystorm#39)
• Added Spout.activate and Spout.deactivate methods that will be called in Storm 1.1.0 and above when a spout is activated or deactivated. This is handy if you want to close database connections on deactivation and reconnect on activation. (Issue #351, PR pystorm/pystorm#42)
• Can now override config.json Nimbus host and port with the STREAMPARSE_NIMBUS environment variable (PR #347)
• Original topology name will now be sent to Storm as topology.original_name even when you're using sparse --override_name. (PR #354)

Fixes

• Fixed an issue where batching bolts would fail all batches they had received when they encountered an exception, even when exit_on_exception was False. Now they will only fail the current batch when exit_on_exception is False; if it is True, all batches are still failed. (PR pystorm/pystorm#43)
• No longer call lein jar twice when creating jars. (PR #348)
• We now use yaml.safe_load instead of yaml.load when parsing command line options. (commit 6e8c4d8)

This release fixes a few bugs and adds the ability to pre-build JARs for submission to Storm/Nimbus..

Features

• Added --local_jar_path and --remote_jar_path options to submit to allow the re-use of pre-built JARs. This should make deploying topologies that are all within the same Python project much faster. (Issue #332)
• Added help subcommand, since it's not immediately obvious to users that sparse -h submit and sparse submit -h will return different help messages. (Issue #334)
• We now provide a universal wheel on PyPI (commit f600c98)
• sparse kill can now kill any topology and not just those that have a definition in your topologies folder. (commit 66b3a70)

Fixes

• Fixed Python 3 compatibility issue in sparse stats (Issue #333) an issue where name was being used instead of override_name when calling pre- and post-submit hooks. (10e8ce3)
• sparse will no longer hang without any indication of why when you run it as root. (Issue #324)
• RedisWordCount example topology works again (PR #336)
• Fix an issue where updating virtualenvs could be slow because certain versions of fabric would choke on the pip output (commit 9b1978f)