Snowplow Versions Save

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

r112-baalbek

5 years ago

This release brings reliability improvements to the batch pipeline

EmrEtlRunner

  • Bump to 0.34.0 (#3935)
  • Add support for running steps on a persistent EMR cluster (#3930)
  • Recover from request timeout (#3943)
  • Set timeout to 120 seconds (#3949)
  • Leverage compaction steps when copying to S3 (#3940)
  • Recover from S3 internal errors (#3950)
  • Rename EMR steps (#3925)
  • Scrub credentials from stderr (#2815)
  • Bump elasticity to 6.0.14 (#3948)
  • Extend copyright notice to 2019 (#3978)

Clojure Collector

  • Bump to 2.1.3 (#3947)
  • Add ebextension to increase the number of file descriptors to 65536 (#3876)
  • Extend copyright notice to 2019 (#3979)

Storage

  • Redshift: widen geo_region to 3 characters (#3822)
  • Postgres: widen geo_region to 3 characters (#3946)
  • Event Manifest Populator: support s3a uri scheme (#3870)

r111-selinunte

5 years ago

This release brings a bug fix to the Clojure Collector

Clojure Collector

  • Bump to 2.1.2 (#3900)
  • Extend access control headers to all types of requests (#3899)

Common

  • Remove Stream Enrich PubSub deployment (#3893)
  • Fix Spark Enrich version in R109 CHANGELOG entry (#3895)

r110-valle-dei-templi

5 years ago

This release brings a new enrichment process running on Google Cloud Dataflow: Beam Enrich

Beam Enrich

  • Implement a barebone port of Stream Enrich (#3735)
  • Support enrichments relying on local files (#3736)
  • Add support for the PII enrichment (#3888)
  • Add metrics (#3737)
  • Build Docker image (#3815)
  • Add README (#3773)
  • Add CI/CD (#3757)

Stream Enrich

  • Bump to 0.19.1 (#3889)
  • Remove GCP module (#3865)
  • Fix parent event context in PII events (#3886)

Clojure Collector

  • Bump to 2.1.1 (#3879)
  • Update CORS configuration (#3875)
  • Extend copyright notice to 2018 (#3891)

Common

  • Add Bintray Docker registry credentials to .travis.yml (#3814)
  • Remove Vagrantfile (#3877)

r109-lambaesis

5 years ago

This release is a maintenance real-time release although it also upgrades Spark Enrich as well as brings a bugfix to EmrEtlRunner.

Scala Stream Collector

  • Bump to 0.14.0 (#3862)
  • Respect a do not track cookie (#3825)
  • Add a way to customize the response from the root path (#3670)
  • Support HEAD requests (#3827)
  • Allow for multiple domains in crossdomain.xml (#3740)
  • Allow overriding of the kinesis endpoint url in the configuration (#3846)
  • Turn BufferConfig's byteLimit and recordLimit into longs (#3807)

Scala Common Enrich

  • Bump to 0.35.0 (#3861)
  • Externalize ua-parser rule file (#3793)
  • Bump ua-parser to 1.4.0 (#3811)
  • Make the list of files to cache available from the EnrichmentRegistry (#3789)
  • Change Iglu adapter to consider arrays as multiple events (#3858)
  • Update CloudfrontAccessLogAdapter to support newer 26 field format (#3816)
  • Leverage the x-forwarded-for field in CloudfrontLoader (#2859)
  • Handle comma-separated list of ips (#3771)
  • Bump SBT Bintray to 0.5.4 (#3840)
  • Bump SBT to 1.1.6 (#3839)

Stream Enrich

  • Bump to 0.19.0 (#3864)
  • Bump scala-common-enrich to 0.35.0 (#3863)
  • Allow overriding of the kinesis endpoint url in the configuration (#3775)
  • Decorrelate the need for a pii stream and the pii enrichment (#3828)
  • Bump SBT to 1.1.6 (#3841)
  • Fix configuration example (#3820)

Spark Enrich

  • Bump to 1.1.6.0 (#3874)
  • Bump scala-common-enrich to 0.35.0 (closes #3869)
  • Bump SBT Assembly to 0.14.7 (#3844)
  • Bump SBT to 1.1.6 (#3843)

EmrEtlRunner

  • Bump to 0.33.1 (#3867)
  • Replace recursive functions by their iterative versions (#3866)
  • Retrieve correct latest run id when using s3a (#3871)
  • Update spark_enrich version in config samples to 1.16.0 (#3859)

Common

  • Add pull request template (#3818)
  • Add issue template (#3819)
  • Update CONTRIBUTING.md (#3530)
  • Update to new logo in README (#3855)
  • Remove Vagrant setup (#3851)
  • Add Gitter badge (#3838)
  • Bump Travis Scala version to 2.11.12 (#3837)

Miscellaneous

  • Config: update example config for the UA parser enrichment to version 1-0-1 (#3868)
  • Event Manifest Populator: set visible_to_all_users flag to true (#3201)

r108-val-camonica

5 years ago

This release brings end-to-end encryption capabilities to the batch pipeline as well as update the Clojure Collector.

EmrEtlRunner

  • Bump to 0.33.0 (#3800)
  • Add ability to specify an EMR security configuration (#3798)
  • Handle SSE-S3 encrypted S3 buckets (#3456)
  • Replace Sluice by aws-sdk-s3 (#3524)
  • Add --ignore-lock-on-start option (#3537)
  • Check the processing folder for emptiness when resuming from enrich (#3803)
  • Make port in Snowplow monitoring configurable (#3236)
  • Make protocol in Snowplow monitoring configurable (#3791)

Clojure Collector

  • Bump to 2.1.0 (#3801)
  • Make cookie path configurable (#2739)
  • Do not allow dependencies requiring an HTTP repository (#3559)
  • Bump lein-ring to 0.12.4 (#3783)
  • Bump commons-codec to 1.11 (#3782)
  • Bump metrics-clojure to 2.10.0 (#3781)
  • Bump compojure to 1.6.1 (#3780)
  • Bump clojure to 1.9.0 (#3779)
  • Bump ring to 1.6.3 (#3778)
  • Remove lein-beanstalk (#3784)

r107-trypillia

5 years ago

This release brings a new enrichment to detect bots and spiders as well as two new webhooks integration (Vero and Marketo).

Scala Common Enrich

Add adapter to pre-process Marketo webhooks (#2616) Add adapter to pre-process Vero webhooks (#2757) Propagate the currency code to all the contexts which need it in the GA adapter (#3733) Add IAB Spiders & Robots Enrichment (#937) Bump to 0.34.0 (#3758)

Stream Enrich

Bump scala-common-enrich to 0.34.0 (#3730) Add support for the IAB enrichment (#3797) Force jackson-databind to 2.9.3 for all projects (#3767) Rename force-ip-lookups-download to force-cached-files-download (#3809) Bump to 0.18.0 (#3727)

Spark Enrich

Bump scala-common-enrich to 0.34.0 (#3729) Add support for the IAB Enrichment (#3772) Bump to 1.15.0 (#3728)

Misc

Fix travis deployment test (#3805) Update example config for PII to version 2-0-0 (#3812)

r106-acropolis

5 years ago

This release brings a new version of the PII enrichment for both the batch and real-time pipelines.

Scala Common Enrich

  • Add formats as ScalazJson4sUtils.extract as implicit parameter (#3668)
  • Add salt to PII Enrichment (#3648)
  • Bump to 0.33.0 (#3763)
  • Bump user-agent-utils to 1.21 (#3656)
  • Extend PII Enrichment to include identification events in EnrichedEvent (#3580)
  • Fix platform specific error checking in IpLookupsEnrichmentSpec (#3762)
  • Fix unnecessarily-created JSON object as a result of the PII Enrichment (#3636)

Spark Enrich

  • Apply automated code formatting (#3655)
  • Bump scala-common-enrich to 0.33.0 (#3764)
  • Bump to 1.14.0 (#3765)
  • Ignore PII identification events from Scala Common Enrich (#3582)
  • Use automated code formatting (#3654)

Stream Enrich

  • Add context for parent event when generating PII event (#3724)
  • Add end-to-end test using mock streaming (#3639)
  • Apply automated code formatting (#3651)
  • Bump scala-common-enrich to 0.33.0 (#3607)
  • Bump to 0.17.0 (#3608)
  • Extend PII Enrichment to output a stream of PII identification events (#3581)
  • Update config.hocon.sample to include a PII output stream (#3579)
  • Use automated code formatting (#3644)

EmrEtlRunner

  • Update rdb_shredder version in config samples to 0.13.1 (#3790)
  • Update spark_enrich version in config samples to 1.14.0 (#3804)

r105-pompeii

6 years ago

This release focuses on fixing a duplication issue in Stream Enrich.

Stream Enrich

  • Bump to 0.16.1 (#3748)
  • Ensure a one-to-one relationship between sink and record processor (#3745)
  • Force jackson-databind to 2.9.3 (#3744)

Scala Common Enrich

  • Update WeatherEnrichmentSpec (#3749)

r104-stoplesteinan

6 years ago

Fixes issues with the Stream Enrich mode introduced in EmrEtlRunner in R102.

EmrEtlRunner

  • Bump to 0.32.0 (#3723)
  • Fail fast when trying to skip staging or enrich in stream enrich mode (#3726)
  • Factor out steps-generating function (#3718)
  • Uncompress enriched files when copying to HDFS (#3719)
  • Fix srcPattern for copying stream enriched data to HDFS (#3722)
  • Check if whole enriched.good is non-empty in stream enrich mode (#3717)

Common

  • Remove trailing hyphen from CHANGELOG entry for R103 (#3731)

r103-paestum

6 years ago

Upgrading the IP Lookups Enrichment

Scala Common Enrich

  • Bump to 0.32.0 (#3673)
  • Bump scala-maxmind-iplookups to 0.4.0 (#3675)
  • Update IP Lookups Enrichment to support non-legacy database (#3672)
  • Support extraction of IP addresses in the Forwarded header (#3475)
  • Support IPv6 addresses in the IpAddressExtractor (#3474)
  • Bump mandrill event versions to 1-0-1 (#3372)

Stream Enrich

  • Bump to 0.16.0 (#3698)
  • Bump scala-common-enrich to 0.32.0 (#3676)
  • Force jackson-dataformat-cbor to 2.9.3 (#3701)

Spark Enrich

  • Bump to 1.13.0 (#3705)
  • Bump scala-common-enrich to 0.32.0 (#3674)
  • Downgrade geoip2 to 2.5.0 (#3702)

Clojure Collector

  • Bump to 2.0.0 (#3708)
  • Make Flash access domains and secure configurable (#2914)
  • Retrieve configuration only through JVM properties (#3709)
  • Allow HTTP repositories (#3707)
  • Add CI/CD (#3712)

Config

  • Update database value in config/enrichments/ip_lookups.json (#3671)

EmrEtlRunner

Update spark_enrich version in config.yml.sample to 1.13.0 (#3710)

Common

  • Rename Caravel to Superset (#3595)
  • Redirect support request to discourse in CONTRIBUTING.md (#3478)