Cernan Versions Save

This release includes a new version of the Avro source that can carry arbitrary metadata out of band. Additionally rustfmt and clippy have been run over the codebase and are now ran in Travis to ensure code is kept formatted and free of lint violations, this accounts for the vast majority of changes in this release.

• Kinesis & Firehose sinks have been removed as they were not being actively used.

• BufferedPayload allocations have been capped at 1MB. This prevents malicious users from running Cernan out of memory with bad requests.

• Timestamps are now explicitly set in the Kafka sink. See - https://github.com/postmates/cernan/pull/441.

• Migrated from travis-ci.org to travis-ci.com.

This release contains a memory-leak fix for our kafka dependency as well as a change to prometheus internal telemetry. The bulk of the diff is in automatic upgrades to our crates.

• 12520e9 :: cernan.sinks.prometheus.aggregation.reportable as SET, not SUM (#435)
• ed4339e :: Upgrade rdkafka to 0.11.4 (rustrdkafka to 0.16.0) (#436)

This release includes improvements to the programmable_filter's error handling, enabling of the logging subsystem before configuration file parsing happens and quieting of the kafka sink when reporting hangup errors.

• e78bf80 :: Add a context that ignores "Receive failed: Disconnected" error logs
• 8b171f7 :: Remove extra lazy_statics
• 408722d :: Add error handling to calling lua functions
• 604d7d7 :: Enable the logging subsystem before config parsing

This release corrects an issue with the Elasticsearch sink losing tags. We expect memory efficiency of the logging subsystem will also be improved. The json_encode filter is now hooked up and the kafka sink has seen its error handling improved.

• 29a6db5 :: Promote MessageTimedOut to an error that will be retried
• c9c6f01 :: json_encode filter configuration
• 1a8fa81 :: Fix the lack of tags in ES sink, introduced in 0.8.13

This release contains a host of small fixes to cernan. The wavefront sink live-stall is, we believe, addressed. We also no longer emit Elasticsearch debug messages at error level, which turned out to cause a feedback cycle for some users.

• a14146e :: Remove redundant extern
• 68a63bb :: Emit debug, not error, in ES sink
• ac3d847 :: Address one possible WF sink live-stall cause

This release incorporates significant improvements to cernan and is a recommended upgrade. In particular, cernan now uses hopper 0.4. This update of hopper is significantly more efficient, being only about 3x slower than built-in MPSC while maintaining resource quotas. These quotas are now exposed as configuration to cernan. An experimental kafka sink is also added but care should be taken with its use. Mike Lang has been landing improvements to the programmable filters and it is hoped they are now more useful.

Cernan's baseline acceptance test is now 272k statsd PPS, held in 10Mb. This is up from 45k statsd PPS at 150Mb.

• dcbeb7d :: Reduce high PPS memory consumption
• 4d2d118 :: Update hopper to 0.4
• d10d195 :: Added json_encode filter
• ecc2fdf :: Kafka sink introduced

This release contains a bug fix for 0.8.11, which double-counted its internal telemetry. Anyone using 0.8.11 and the internal source is recommended to update. Elsewise, you're more or less blind.

• dac169c :: Don't double-count internal telemetry

This release switches our internal telemetry from a Set aggregation to Sum, removing the reset to zero between flushes. In release mode cernan will wrap-around when the counters are fully saturated. Don't saturate your counters in debug mode. We also correct telemetry in the kinesis sink.

• c6abe6b :: Switch internal telemetry to Sum from Set
• f50a892 :: Fixes Kinesis Internal Telem. on Batch Success

This release improves warnings around truncation of bin_width when flush_interval is lower than bin_width for a given sink. The native source will no longer space explode if the far-side does something goofy, panics have been removed in the statsd/tcp sources and kinesis is now more durable to errors.

• d0bb60b :: Warn if Wavefront bin_width > flush_interval
• 0f4756b :: Resolve space explosion in the native source
• 44fcabc :: Removes Undesirable Panics From statsd/tcp Sources
• 8c3084f :: Improves Durability of Kinesis Sink.