• Terraform module to deploy lambda that sends notifications about AWS CloudTrail events to Slack
  • Why this module?
    • Example message
• Configurations
  • Slack App (Recommended)
  • Slack Webhook
  • AWS SNS
• Rules
  • Default rules:
  • User defined rules to match events
  • Events to track
  • Custom Separator for Rules
  • Ignore Rules
• About processing Cloudtrail events
• Terraform specs
  • Requirements
  • Providers
  • Modules
  • Resources
  • Inputs
  • Outputs
  • License

Terraform module to deploy lambda that sends notifications about AWS CloudTrail events to Slack

Why this module?

This module allows you to get notifications about:

• actions performed by root account (According to AWS best practices, you should use root account as little as possible and use SSO or IAM users)
• API calls that failed due to lack of permissions to do so (could be an indication of compromise or misconfiguration of your services/applications)
• console logins without MFA (Always use MFA for you IAM users or SSO)
• track a list of events that you might consider sensitive. Think IAM changes, network changes, data storage (S3, DBs) access changes. Though we recommend keeping that to a minimum to avoid alert fatigue
• define sophisticated rules to track user-defined conditions that are not covered by default rules (see examples below)
• send notifications to different Slack channels based on event account id

Example message

Configurations

The module has three variants of notification delivery:

Slack App (Recommended)

• Offers additional features, such as consolidating duplicate events into a single message thread. More features may be added in the future.
• The Slack app must have the chat:write permission.
• Terraform configuration example

Slack Webhook

• Provides all the basic functionality of the module, but does not offer additional features and is not recommended by Slack.
• Terraform configuration example

AWS SNS

• An optional feature that allows sending notifications to an AWS SNS topic. It can be used alongside either the Slack App or Slack Webhook.

All three variants of notification delivery support separating notifications into different Slack channels or SNS topics based on event account ID.

Rules

Rules are python strings that are evaluated in the runtime and should return the bool value, if rule returns True, then notification will be sent to Slack.

This module comes with a set of predefined rules (default rules) that users can take advantage of:

Default rules:

# Notify if someone logged in without MFA but skip notification for SSO logins
default_rules.append('event["eventName"] == "ConsoleLogin" '
                     'and event.get("additionalEventData.MFAUsed", "") != "Yes" '
                     'and "assumed-role/AWSReservedSSO" not in event.get("userIdentity.arn", "")')
# Notify if someone is trying to do something they not supposed to be doing but do not notify
# about not logged in actions since there are a lot of scans for open buckets that generate noise
default_rules.append('event.get("errorCode", "").endswith(("UnauthorizedOperation"))')
default_rules.append('event.get("errorCode", "").startswith(("AccessDenied"))'
                     'and (event.get("userIdentity.accountId", "") != "ANONYMOUS_PRINCIPAL")')
# Notify about all non-read actions done by root
default_rules.append('event.get("userIdentity.type", "") == "Root" '
                     'and not event["eventName"].startswith(("Get", "List", "Describe", "Head"))')

# Catch CloudTrail disable events
default_rules.append('event["eventSource"] == "cloudtrail.amazonaws.com" '
                     'and event["eventName"] == "StopLogging"')
default_rules.append('event["eventSource"] == "cloudtrail.amazonaws.com" '
                     'and event["eventName"] == "UpdateTrail"')
default_rules.append('event["eventSource"] == "cloudtrail.amazonaws.com" '
                     'and event["eventName"] == "DeleteTrail"')
# Catch cloudtrail to slack lambda changes
default_rules.append('event["eventSource"] == "lambda.amazonaws.com" '
                     'and "responseElements.functionName" in event '
                     f'and event["responseElements.functionName"] == "{function_name}" '
                     'and event["eventName"].startswith(("UpdateFunctionConfiguration"))')
default_rules.append('event["eventSource"] == "lambda.amazonaws.com" '
                     'and "responseElements.functionName" in event '
                     f'and event["responseElements.functionName"] == "{function_name}" '
                     'and event["eventName"].startswith(("UpdateFunctionCode"))')

User defined rules to match events

Rules must be provided as a list of strings, each separated by a comma or a custom separator. Each string is a Python expression that will be evaluated at runtime. By default, the module will send rule evaluation errors to Slack, but you can disable this by setting 'rule_evaluation_errors_to_slack' to 'false'.

Example of user-defined rules:

locals = {
  rules = [
    # Catch CloudTrail disable events
    "event['eventSource'] == 'cloudtrail.amazonaws.com' and event['eventName'] == 'StopLogging'"
    "event['eventSource'] == 'cloudtrail.amazonaws.com' and event['eventName'] == 'UpdateTrail'"
    "event['eventSource'] == 'cloudtrail.amazonaws.com' and event['eventName'] == 'DeleteTrail'"
  ]
    rules = join(",", local.rules)
}

Events to track

This is much simpler than rules. You just need a list of eventNames that you want to track. They will be evaluated as follows:

f'"eventName" in event and event["eventName"] in {json.dumps(events_list)}'

Terraform example:

local{
  # EC2 Instance connect and EC2 events
  ec2 = "SendSSHPublicKey"
  # Config
  config = "DeleteConfigRule,DeleteConfigurationRecorder,DeleteDeliveryChannel,DeleteEvaluationResults"
  # All events
  events_to_track = "${local.ec2},${local.config}"
}

events_to_track = local.events_to_track

Custom Separator for Rules

By default, the module expects rules to be separated by commas. However, if you have complex rules that contain commas, you can use a custom separator by providing the rules_separator variable. Here's how:

locals {
  cloudtrail_rules = [
      ...
    ]
  custom_separator = "%"
}

module "cloudtrail_to_slack" {
  ...
  rules = join(local.custom_separator, local.cloudtrail_rules)
  rules_separator = local.custom_separator
}

Ignore Rules

Note: We recommend addressing alerts rather than ignoring them. However, if it's impossible to resolve an alert, you can suppress events by providing ignore rules.

Ignore rules have the same format as the rules, but they are evaluated before them. So, if an ignore rule returns True, then the event will be ignored and no further processing will be done.

locals {
  ignore_rules = [
    # Ignore events from the account "111111111".
    "'userIdentity.accountId' in event and event['userIdentity.accountId'] == '11111111111'",
  ]
  ignore_rules = join(",", local.ignore_rules)
}

About processing Cloudtrail events

CloudTrail event (see format here, or find more examples in src/tests/test_events.json) is flattened before processing and should be referenced as event variable So, for instance, to access ARN from the event below, you should use the notation userIdentity.arn

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "XXXXXXXXXXX",
    "arn": "arn:aws:iam::XXXXXXXXXXX:user/xxxxxxxx",
    "accountId": "XXXXXXXXXXX",
    "userName": "xxxxxxxx"
  },
  "eventTime": "2019-07-03T16:14:51Z",
  "eventSource": "signin.amazonaws.com",
  "eventName": "ConsoleLogin",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "83.41.208.104",
  "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0",
  "requestParameters": null,
  "responseElements": {
    "ConsoleLogin": "Success"
  },
  "additionalEventData": {
    "LoginTo": "https://console.aws.amazon.com/ec2/v2/home?XXXXXXXXXXX",
    "MobileVersion": "No",
    "MFAUsed": "No"
  },
  "eventID": "0e4d136e-25d4-4d92-b2b2-8a9fe1e3f1af",
  "eventType": "AwsConsoleSignIn",
  "recipientAccountId": "XXXXXXXXXXX"
}

Slack App configuration:

1. Go to https://api.slack.com/
2. Click create an app
3. Click From an app manifest
4. Select workspace, click next
5. Choose yaml for app manifest format

display_information:
  name: CloudtrailToSlack
  description: Notifications about Cloudtrail events to Slack.
  background_color: "#3d3d0e"
features:
  bot_user:
    display_name: Cloudtrail to Slack
    always_online: false
oauth_config:
  scopes:
    bot:
      - chat:write
settings:
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false

6. Check permissions and click create
7. Click install to workspace
8. Copy Signing Secret # for slack_signing_secret module input
9. Copy Bot User OAuth Token # for slack_bot_token module input

Terraform specs

Requirements

Providers

Modules

Resources

Inputs

list(object({
    accounts       = list(string)
    slack_hook_url = string
  }))

list(object({
    accounts         = list(string)
    slack_channel_id = string
  }))

list(object({
    accounts      = list(string)
    sns_topic_arn = string
  }))

Outputs

License

Apache 2 Licensed. See LICENSE for full details.

Weekly review link

• Review
• Review branch-based review