StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
A complete list of changes in v3.5.0 can be found here
A complete list of changes in v3.4.1 can be found here
A complete list of changes in v3.4.0 can be found here
A complete list of changes in v3.3.0 can be found here
A complete list of changes in v3.2.1 can be found here
The previously implemented AWS Lambda output was unable to perform cross-account execution of Lambda functions. The V2 version of the Lambda output now supports performing an AssumeRole call prior to invoking the Lambda output, enabling this sort of functionality.
A complete list of changes in v3.2.0 can be found here
A complete list of changes in v3.1.2 can be found here
A complete list of changes in v3.1.1 can be found here
The concept of "stateful" alerting has always been a gap that StreamAlert has failed to bridge. We've introduced a feature we've dubbed Scheduled Queries as a way to help bridge that gap. Users can now write and deploy Athena queries that will run on a user-defined schedule. The results of these queries are then fed data back into StreamAlert's Rules Engine for further processing and alerting. See the documentation for more information on getting up and running with Scheduled Queries.
See also: #1209
It is now possible for rules to dynamically configure outputs based on information in a record. A new keyword argument of dynamic_outputs has been added to the @rule decorator to support this. For more information on how to leverage this for yourself, see the documentation. This is great addition that we've also wanted for a long time, so a huge thank you to @jack1902 for adding this!
Support has been added for sending alerts to AWS Simple Email Service (SES). This enables sending richly formatted emails to recipients, as opposed to the previous method of using AWS SNS for sending only very simple emails. A huge thanks to @jack1902 for contributing this!
Support has also been added for sending alerts to Microsoft Teams. A huge thanks (again!) to @jack1902 for contributing this!
The Publishers testing implementation has been updated to support configuring tests for publishers directly within a test event file. For more information on how to add tests for Publishers, see the documentation.
See also: #1185
One of our biggest pain points in the StreamAlert ecosystem has been the speed of searches. This release adds support for Parquet as the storage format of data sent to S3 for historical data retention, and we're already seeing vast improvements in comparison to JSON. In addition to this, Athena tables are also now created and managed via Terraform, removing the need for users to reason about them during deployment time.
See also: #1202
In addition to the updates to integration tests made as part of #1181, a larger update to the framework has migrated tests out of the tests/integration directory. Integration test files for rules should now live beside the rule being tested. The documentation for tests includes more details.
Thanks to @jack1902 for adding two new rules related to AWS Config!
Thanks to @chunyong-lin for open-sourcing a rule to alert on SSH login activity captured by osquery.
To view the complete list of all of the bugs fixed in v3.1.0, including many not mentioned above, see here.
To view the complete list of all changes included in v3.1.0, see here.
In StreamAlert version 2, we added undocumented support for utilizing files stored in S3 as a method to dynamically load information for use within rules. This release completely redesigns LookupTables and adds new support for DynamoDB as a backend storage mechanism. The big driver behind this decision was the need to alleviate the high memory consumption that came with reading and deserializing JSON files from S3, among other benefits. Additionally, the new implementation will enable easily extending LookupTables to support other, arbitrary backend types.
Given that StreamAlert utilizes a shared, remote state file that is stored in S3, it was effectively possible for multiple users to be reading and writing a state file at the same time. This could lead to state file corruption, and the potential to botch your entire deployment. In this release, we have added support for remote state file locking, via DynamoDB, to ensure that this type of corruption never occurs.
Perhaps the most important part of this release is the full conversion of the codebase to Python 3.7. This has been a long time coming and we know that many have been patiently awaiting this. The Getting Started instructions have been updated to be specific to Python 3, and there is no backward compatibility to Python 2.7. See the note below on migration. A huge thanks to @GarretReece and @btonic for their support in this endeavor.
See also: #986
The release of the new version of Terraform introduced support for proper for each loops and "dynamic" blocks, along with numerous other benefits. These features have already enabled us to fix longstanding bug in our S3 event notifications module, and we will continue to leverage the benefits of Terraform version 12 to improve our Terraform modules going forward. A huge thanks to @scoders for their support in making this possible.
This release introduces the ability to break up your logs.json file into multiple different files, nested within a schemas directory. It is now possible to have, for example, definition files of schemas/osquery.json and/or schemas/aws.json to independently house all of your osquery and aws related log definitions, respectively. This feature is currently backwards compatible, so if your logs.json is not unmanageably large, or you are not ready to make the change, then your current definitions will continue to function just fine.
Version 3 of StreamAlert includes the migration of the cluster source definitions to reside within the individual cluster configuration files. Be sure to check the new format in the StreamAlert docs on the Datasource Configuration page and update your deployment configs accordingly.
To view the complete list of all of the improvements in v3.0.0, including many not mentioned above, see here.
This release includes sweeping changes to correct any and all discrepancies related to prefixing or namespacing with the streamlert name. All IAM roles are now created under a valid namespace, each Kinesis Data Firehose is now prefixed properly, and even the pesky “stream_alert” naming convention has been thoroughly scrubbed in this update. We know that these changes may lead to many complications with upgrading, so please see the note below on migration.
See also: #1013, #1064
The alert merger function has been updated to better handle massive alert throughput, reducing the potential for out-of-memory exceptions in the function.
Various Terraform modules have also been refactored to reduce complexity or redundancy.
See also: #1041, #1069
The G Suite StreamAlert Apps have been updated to support newer versions of the Python client library for Google. Additionally, the Box StreamAlert App dependencies have been updated to an official, non-alpha release that supports JSON Web Tokens (JWT) for authentication.
To view the complete list of all of the bugs fixed in v3.0.0, including many not mentioned above, see here.
To view the complete list of all changes included in v3.0.0, see here.
There are many changes in this release that will make it prohibitively difficult to do a direct "upgrade" from version 2 to version 3. Therefore, we recommend updating your prefix setting and doing a complete redeploy of StreamAlert version 3. This should be possible alongside a current deployment, thanks in part to the changes that are included in release 3.
After deploying version 3, you must also migrate over your data sources/producers as you see fit. This step is beyond the scope of the advice we're willing to provide here, but please reach out to us if you are stuck and need assistance.
At a minimum, any rules will need to be updated for new import paths (stream_alert --> streamalert) along with any Python 2 vs Python 3 syntax changes. We recommend using the Python 2to3 tool on all custom rules to begin your conversion, which should address most of the necessary changes.
We would also like to give a special shout out to @jack1902 and @0xdabbad00 for their contributions to version 3.