Dsiem

Dsiem is a security event correlation engine for ELK stack, allowing the platform to be used as a dedicated and full-featured SIEM system.

Dsiem provides OSSIM-style correlation for normalized logs/events, perform lookup/query to threat intelligence and vulnerability information sources, and produces risk-adjusted alarms.

Features

• Runs in standalone or clustered mode with NATS as messaging bus between frontend and backend nodes. Along with ELK, this made the entire SIEM platform horizontally scalable.
• OSSIM-style correlation and directive rules, bridging easier transition from OSSIM.
• Alarms enrichment with data from threat intel and vulnerability information sources. Builtin support for Arkime Wise (which supports Alienvault OTX and others) and Nessus CSV exports. Support for other sources can easily be implemented as plugins.
• Instrumentation supported through Metricbeat and/or Elastic APM server. No need extra stack for this purpose.
• Builtin rate and back-pressure control, set the minimum and maximum events/second (EPS) received from Logstash depending on your hardware capacity and acceptable delays in event processing.
• Loosely coupled, designed to be composable with other infrastructure platform, and doesn't try to do everything. Loose coupling also means that it's possible to use Dsiem as an OSSIM-style correlation engine with non ELK stack if needed.
• Batteries included:
  • A directive conversion tool that reads OSSIM XML directive file and translate it to Dsiem JSON-style config.
  • A SIEM plugin creator tool that will read off an existing index pattern from Elasticsearch, and creates the necessary Logstash configuration to clone the relevant fields' content to Dsiem. The tool can also generate basic directive required by Dsiem to correlate received events and generate alarm.
  • A helper tool to serve Nessus CSV files over the network to Dsiem.
  • A light weight Angular web UI just for basic alarms management (closing, tagging), and easy pivoting to the relevant indices in Kibana to perform the actual analysis.
• Obviously a cloud-native, twelve-factor app, and all that jazz.

How It Works

flowchart TB

l1 --> |Normalized Logs/Events| e1
l1 --> |Alarms|e1
l1 --> |Normalized Logs/Events| d1

f1 --> |Logs| l1

s1 --> |Logs| f1
s2 --> |Logs| f1
s3 --> |Logs| f1

d1 --> |Alarms| d2
d2 --> |Alarms| l1
d1 --> |Query| d3

subgraph Elasticsearch
  e1[Elasticsearch]
end

subgraph Logstash
  l1[Logstash] 
end

subgraph Filebeat
 f1[Syslog/Filebeat]
end

subgraph Log Sources
  s1[IDS]
  s2[Firewall]
  s3[OS]
end

subgraph Dsiem
  d1[Dsiem]
  d2[Dsiem-filebeat]
  d3[Threat-Intel/Vulnerability-Info-Sources]
end

On the diagram above:

1. Log sources send their logs to Syslog/Filebeat, which then sends them to Logstash with a unique identifying field. Logstash then parses the logs using different filters based on the log sources type, and sends the results to Elasticsearch, typically creating a single index pattern for each log type (e.g. suricata-* for logs received from Suricata IDS, ssh-* for SSH logs, etc.).

2. Dsiem uses a special purpose logstash config file to clone incoming event from log sources, right after logstash has done parsing it. Through the same config file, the new cloned event is used (independently from the original event) to collect Dsiem required fields like Title, Source IP, Destination IP, and so on.

3. The output of the above step is called Normalized Event because it represent logs from multiple different sources in a single format that has a set of common fields. Those events are then sent to Dsiem through Logstash HTTP output plugin, and to Elasticsearch under index name pattern siem_events-*.

4. Dsiem correlates incoming normalized events based on the configured directive rules, perform threat intel and vulnerability lookups, and then generates an alarm if the rules conditions are met. The alarm is then written to a local log file, that is harvested by a local Filebeat configured to send its content to Logstash.

5. At the logstash end, there's another Dsiem special config file that reads those submitted alarms and push them to the final SIEM alarm index in Elasticsearch.

The final result of the above processes is that now we can watch for new alarms and updates to an existing one just by monitoring a single Elasticsearch index.

Installation

You can use Docker Compose or the release binaries to install Dsiem. Refer to the Installation Guide for details.

Alternatively, there's also a Docker Compose or virtual machine-based demo environment that you can use to evaluate all Dsiem integration from one simple web interface.

Documentation

Currently available docs are located here.

Reporting Bugs and Issues

Please submit bug and issue reports by opening a new Github issue. Security-sensitive information, like details of a potential security bug, may also be sent to [email protected]. The GPG public key for that address can be found here.

How to Contribute

Contributions are very welcome! Submit PR for bug fixes and additional tests, gist for Logstash config files to parse device events, SIEM directive rules, or a new threat intel/vulnerability lookup plugins.

If you're not sure on what to do on a particular matter, feel free to open an issue and discuss first.

License

The project is licensed under GPLv3. Contributors are not required to sign any form of CAA/CLA or a like: We consider their acceptance of this Github terms of service clause to be sufficient.