Snowplow Versions Save

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

22.01

2 years ago

22.01 Western Ghats brings togethe latest updates to the Snowplow Open Source Projects as part of the Snowplow OS Distribution 22.01 Western Ghats. This announcement is the third distribution in our new format, and as before, we clarify our recommended component versions and discuss the latest Snowplow features.

Features

So what’s new in 22.01 Western Ghats? Since the release of 21.08 North Cascades, we have focused on:

  • Making it easier to get started on Google Cloud Platform with new Terraform modules
  • Getting your Snowplow data to more destinations with Google Tag Manager Server Side
  • Getting your data from more sources with new Trackers
  • Landing your Snowplow events to core destinations more reliably, cost effectively and with observability with our updated Loaders.

There’s some other goodies in 22.01 too, so without any further delay head over to the announcement on the Snowplow Blog for all the details.

21.08

2 years ago

Today we bring together and share the latest updates to the Snowplow Open Source Projects as part of the Snowplow OS Distribution 21.08 North Cascades. This announcement is the second distribution in our new format, building on the updates we discussed in the 21.04 Pennine Alps annoucement. As before, we clarify our recommended component versions and discuss the latest Snowplow features.

Features

So what’s new in 21.08 North Cascades? Over the past few months, we have focused on:

  • Improved mobile analytics options: Continuing to improve our mobile offering through a number of mobile tracker updates (inc. the ability to remotely update configurations), as well as the release of the mobile data models.
  • Open Source quick start for AWS: Making it easier than ever to get started with Snowplow open source on AWS via a suite of terraform modules (GCP coming soon).
  • Optimizing the core pipeline applications: Optimising the cost, performance & observability of the Snowplow pipeline through a re-write of some of the core pipeline applications. Read more about our strategy to move our pipeline applications away from big data frameworks.
  • Support for dbt: The first release of the official Snowplow dbt package for our web data model.
  • Snowplow Micro experience improvements: Load your schemas directly into Snowplow Micro and various other developer experience improvements.

21.04

3 years ago

Overview

This release had a focus on a number of reliability and general hardening improvements to the platform, alongside exciting updates such as Surge Protection for AWS, Anonymous Tracking capabilities, brand new Data Models for the web, a major release of the JavaScript Trackers and automated updates to our GitHub home, snowplow/snowplow.

Features

r119-tycho-magnetic-anomaly-two

4 years ago

After R118 beta release, R119 contains the first production versions of components emitting the new bad row format :

  • Scala Common Enrich : Scala library that contains the validation and enrichments logic, used by Stream Enrich and Beam Enrich
  • Stream Enrich : Scala application that performs the enrichments with either Kafka, NSQ or Kinesis as input and output
  • Beam Enrich : Dataflow job that performs the enrichments on GCP with PubSub as input and ouput
  • Scala Stream Collector : Scala application that receives the events sent by the trackers

This release also includes a new version of EmrEtlRunner with improved fault tolerance.

Beam Enrich gets extracted to its own repo.

Scala Common Enrich

  • Use recent timestamp for unit tests in WeatherEnrichmentSpec (#4339)
  • Change maxColumns to 140 in .scalafmt.conf (#4314)
  • Add Snowplow Bintray to resolvers (#4325)
  • Return capitalised Unknown value in YAUAA enrichment (#4114)
  • Fix abandon assertions in PiiPseudonymizerEnrichmentSpec (#4322)
  • Handle query string parameters that don't have a value in IgluAdapter (#4330)
  • Bump snowplow-badrows to 1.0.0 (#4292)
  • Fix EtlPipeline short-circuiting on a first bad row (#4320)
  • Add validation for contexts added by enrichments (#3795)
  • Bump to 1.1.0 (#4294)

Stream Enrich

  • Add the possibility to override the DynamoDB endpoint (#3942)
  • Use info generated by BuildInfo for Processor in Source (#4335)
  • Remove sbt version from README (#4303)
  • Bump Scala Common Enrich to 1.1.0 (#4295)
  • Allow to download data from private S3 or GCS (#4269)
  • Bump to 1.1.0 (#4296)

Scala Stream Collector

  • Add Snowplow Bintray to resolvers (#4326)
  • Publish Docker image for stdout via Travis (#4333)
  • Fix config example (#4332)
  • Fix incompatible jackson dependencies to enable CBOR (#4266)
  • Bump to 1.0.1 (#4338)

EmrEtlRunner

  • Switch AMI bootstrap scripts to HTTPS (#4256)
  • Set ig.count (#4285)
  • Catch and retry EMR connection exceptions (#4290)
  • Bump to 0.37.0 (#4297)

Miscellaneous

  • Cloudfront Collector: deprecate (#4319)
  • Clojure Collector: deprecate (#4264)
  • Spark Enrich: deprecate (#4263)
  • Beam Enrich: extract into separate repo (#4282)
  • Common: fix dead links in collectors' README (#4310)
  • Enrich: update example config for referer parser enrichment to 2-0-0 (#4309)

r118-morgantina

4 years ago

Beam Enrich

  • bump to 1.0.0 (#4194)
  • migrate enrichment specs from Spark Enrich (#4200)
  • migrate miscellaneous specs from Spark Enrich (#4199)
  • migrate adapters specs from Spark Enrich (#4198)
  • bump SBT to 1.3.3 (#4174)
  • use sbt-tpolecat (#4172)
  • use scalafmt (#4171)
  • bump sbt-buildinfo to 0.9.0 (#4170)
  • bump scalatest to 3.0.8 (#4169)
  • bump paradise to 2.1.1 (#4168)
  • bump beam to 2.11.0 (#4167)
  • bump scio version to 0.7.4 (#4166)
  • bump scala-common-enrich to 1.0.0 (#4165)
  • bump Scala to 2.12.10 (#4164)
  • bump sbt-native-packager to 1.3.22 (#4163)
  • extend copyright notice to 2020 (#4259)

Stream Enrich

  • bump to 1.0.0 (#4162)
  • bump snowplow-scala-tracker to 0.6.1 (#4145)
  • use sbt-tpolecat (#4173)
  • bump sbt-buildinfo to 0.9.0 (#4161)
  • bump sbt-assembly to 0.14.9 (#4160)
  • bump jinjava to 2.5.0 (#4159)
  • bump pureconfig to 0.11.0 (#4158)
  • bump scopt to 3.7.1 (#4157)
  • bump slf4j to 1.7.26 (#4156)
  • bump jackson to 2.9.9 (#4155)
  • bump config to 1.3.4 (#4154)
  • bump kafka-clients to 2.2.1 (#4153)
  • bump amazon-kinesis-client to 1.10.0 (#4152)
  • bump aws-java-sdk to 1.11.566 (#4151)
  • add custom .jvmopts file (#4104)
  • bump specs2 to 4.5.1 (#4149)
  • bump scala-common-enrich to 1.0.0 (#4148)
  • bump Scala to 2.12.10 (#4147)
  • bump scalacheck to 1.14.0 (#4144)
  • bump SBT to 1.3.3 (#4098)
  • replace sbt-scalafmt-coursier with sbt-scalafmt (#4097)
  • extend copyright notice to 2020 (#4258)

Scala Common Enrich

  • extend copyright notice to 2020 (#4257)
  • bump to 1.0.0 (#4026)
  • wrap all contexts and unstruct events into SelfDescribingData (#4241)
  • remove shred function (#4233)
  • make test specifications formatting consistent (#4234)
  • use snowplow-badrows (#4106)
  • add custom .jvmopts file (#4103)
  • bump SBT to 1.2.8 (#4096)
  • update WeatherEnrichmentSpec (#4073)
  • bump scala-uri to 1.4.5 (#4072)
  • bump scalaj-http to 2.4.1 (#4071)
  • bump mysql-connector-java to 8.0.16 (#4070)
  • bump postgresql to 42.2.5 (#4069)
  • bump uap-java to 1.4.3 (#4068)
  • bump jackson-databind to 2.9.8 (#4067)
  • bump joda-time to 2.10.1 (#4066)
  • bump commons-codec to 1.12 (#4064)
  • clean up dependencies (#4052)
  • parameterize ApiRequestEnrichment over the effect type (#4046)
  • parameterize SqlQueryEnrichment over the effect type (#4045)
  • bump scala-weather to 0.5.0 (#4044)
  • bump scala-maxmind-iplookups to 0.6.1 (#4043)
  • use sbt-scalafmt (#4040)
  • separate EnrichmentRegistry parsing from its construction (#4033)
  • externalize referer-parser yml file (#3830)
  • bump scala-forex to 0.7.0 (#4031)
  • bump scala-referer-parser to 1.0.0 (#4030)
  • bump iglu-scala-client to 0.6.1 (#4029)
  • bump jsonpath to 0.6.14 (#4028)
  • bump Scala to 2.12.10 (#4027)
  • bump specs2 to 4.5.1 (#4024)
  • replace scalaz by cats (#4018)
  • replace json4s with circe (#3602)
  • use sbt-tpolecat (#4010)

Scala Stream Collector

  • bump to 1.0.0 (#4193)
  • introduce sbt-scalafmt (#4192)
  • bump sbt-buildinfo to 0.9.0 (#4191)
  • use sbt-tpolecat (#4190)
  • bump sbt-assembly to 0.14.9 (#4189)
  • bump specs2 to 4.5.1 (#4188)
  • bump pureconfig to 0.11.1 (#4187)
  • bump akka to 2.5.19 (#4186)
  • bump prometheus-simpleclient to 0.6.0 (#4184)
  • bump config to 1.3.4 (#4183)
  • bump slf4j to 1.7.26 (#4182)
  • bump joda-time to 2.10.2 (#4181)
  • bump kafka-clients to 2.2.1 (#4180)
  • bump google-cloud-pubsub to 1.78.0 (#4179)
  • bump aws-java-sdk to 1.11.573 (#4178)
  • integrate the size violation bad row type (#4177)
  • bump SBT to 1.3.3 (#4176)
  • bump Scala to 2.12.10 (#4175)
  • extend copyright notice to 2020 (#4261)

Common

  • bump Travis Scala version to 2.12.10 (#4150)

Spark Enrich

  • bump to 2.0.0 (#4236)
  • bump scala-common-enrich to 1.0.0 (#4235)
  • replace scopt by decline (#4245)
  • extend copyright notice to 2020 (#4260)

Hadoop Event Recovery

  • remove (#3908)

r117-biskupin

4 years ago

Blog post describing the changes in R117

EmrEtlRunner:

  • add support to the spot market for core instances (#3487)
  • bump to 0.36.0 (#4143)

Scala Common Enrich:

  • bump referer-parser to 0.3.1 (#4135)
  • add anonymization for IPv6 (#4222)
  • add additional event fingerprint hashing methods (#4226)
  • bump to 0.38.0 (#4136)

Beam Enrich:

  • bump scala-common-enrich to 0.38.0 (#4137)
  • fix unit tests failing after update of MaxMind database (#4230)
  • fix docker deployment authorization (#4231)
  • bump to 0.4.0 (#4138)

Stream Enrich:

  • bump scala-common-enrich to 0.38.0 (#4139)
  • bump to 0.22.0 (#4140)

Spark Enrich:

  • bump scala-common-enrich to 0.38.0 (#4141)
  • use hadoop-lzo 0.4.20 from Snowplow Bintray maven (#4238)
  • bump to 1.19.0 (#4142)

Common:

  • change Travis distribution to Trusty (#4214)
  • publish docker images for scala-stream-collector, beam-enrich and stream-enrich to DockerHub (#4237)

Scala Stream Collector:

  • allow users to disable the default redirect endpoint (#4211)
  • bump Scala version to 2.11.12 (#4206)
  • bump akka-http to 10.1.10 (#4185)
  • add support for TLS port binding and certificate (#4085)
  • remove duplicate section in example hocon config file (#4210)
  • bump to 0.17.0 (#4208)

r116-madara-rider

4 years ago

This release adds new features to the Scala Stream Collector.

Beam Enrich

  • Fix unit tests failing after update of MaxMind database (#4129)

EmrEtlRunner

  • Add support for shredded TSV data (#4074)
  • Update spark_enrich version in config samples to 1.18.0 (#4091)
  • Bump to 0.35.0 (#4112)

Scala Stream Collector

  • Add options to configure secure, same-site and http-only for the cookie (#3753)
  • Allow multiple cookie domains to be used (#3994)
  • Provide a way to specify custom path mappings (#4087)
  • Send back a Cache-Control header (#4017)
  • Add sbt-native-packager and Docker config (#4128)
  • Bump Akka HTTP to 10.0.15 (#4131)
  • Bump version to 0.16.0 (#4134)

r115-sigiriya

4 years ago

This release concerns mainly 2 small updates for EmrEtlRunner.

EmrEtlRunner

  • Update Contracts for get_failure_details (#4088)
  • Make sure all steps are successfully submitted in case of a transient cluster (#4092)
  • Bump to 0.34.3 (#4089)

Event Manifest Populator

  • Bump to 0.1.2 (#4082)
  • Remove part-* pattern (#4081)

r114-polonnaruwa

4 years ago

This release focuses on a new user agent enrichment using YAUAA as well as providing a way to plug a remote adapter.

Scala Common Enrich

  • Bump to 0.37.0 (#4057)
  • Make IpAddressExtractor fall back to the Forwarded: for= header as a last resort (#4014)
  • Update Sendgrid integration (#4002)
  • Add HTTP remote adapter #3760
  • Add YAUAA enrichment (#4009)
  • Create tutorial for adding an enrichment (#4039)
  • Update WeatherEnrichmentSpec (#4073)
  • Explore more relaxed URL parsing (#3880)
  • Add support to IPs (v4) with port to IP lookup enrichment (#4048)
  • Fix incompatibility between IAB enrichment and Iglu Webhook (#3952)
  • Skip IAB enrichment for IPs v6 addresses (#4078)

Stream Enrich

  • Bump to 0.21.0 (#4059)
  • Bump scala-common-enrich to 0.37.0 (#4058)

Beam Enrich

  • Bump to 0.3.0 (#4061)
  • Bump scala-common-enrich to 0.37.0 (#4060)
  • Fix unit tests failing after update of MaxMind database (#4037)

Spark Enrich

  • Bump to 1.18.0 (#4063)
  • Bump scala-common-enrich to 0.37.0 (#4062)

EmrEtlRunner

  • Extend backoff periods (#4049)
  • Limit requests to the EMR API (#4056)
  • Initialize S3::Aws constant (#4036)
  • Bump to 0.34.2 (#4050)

Common

  • Mention the contributing guide in the readme (#4007)
  • Bump release-manager to 0.4.1 (#4005)

r113-filitosa

5 years ago

This release focuses on improvements to the Scala Stream Collector as well as new features for Scala Common Enrich, the library powering all the different enrichment platforms.

Scala Stream Collector

  • Bump to 0.15.0 (#3983)
  • Expose Prometheus metrics (#3421)
  • Bump kafka client to 2.1.1 (#3981)
  • Provide a way to add arbitrary Kafka configuration settings (#3968)
  • Add support for an Access-Control-Max-Age header (#3904)
  • Allow for the do not track cookie value configuration to be a regex (#3966)
  • Showcase the usage of env variables in the configuration example (#3971)
  • Extend copyright notice to 2019 (#3997)

Scala Common Enrich

  • Bump to 0.36.0 (#3984)
  • Add adapter to pre-process Hubspot webhooks (#3282)
  • Change MarketoAdapter's last_interesting_moment_date type to date-time (#3967)
  • Bump CallRail's call_complete to 1-0-2 (#2501)
  • Support POST requests in API Request enrichment (#3857)
  • Warn users of the user-agent-utils enrichment (#3964)
  • Disable parallel test execution (#3970)
  • Extend copyright notice to 2019 (#3998)

Beam Enrich

  • Bump to 0.2.0 (#3990)
  • Bump scala-common-enrich to 0.36.0 (#3989)
  • Extend copyright notice to 2019 (#4001)

Stream Enrich

  • Bump to 0.20.0 (#3986)
  • Bump scala-common-enrich to 0.36.0 (#3985)
  • Bump kafka client to 2.1.1 (#3992)
  • Provide a way to add arbitrary Kafka configuration settings (#3969)
  • Showcase the usage of env variables in the configuration example (#3972)
  • Extend copyright notice to 2019 (#4000)

Spark Enrich

  • Bump to 1.17.0 (#3988)
  • Bump scala-common-enrich to 0.36.0 (#3987)
  • Add test for the Hubspot adapter (#3977)
  • Add test for the Marketo adapter (#3976)
  • Use sbt-buildinfo (#3628)
  • Extend copyright notice to 2019 (#3999)

EmrEtlRunner

  • Bump to 0.34.1 (#3996)
  • Add exponential backoff when getting cluster statuses (#3995)
  • Update spark_enrich version in config sample to 1.17.0 (#3991)