System for network traffic analysis and anomaly detection.
NEMEA (Network Measurements Analysis) system is a stream-wise, flow-based and modular detection system for network traffic analysis. It consists of many independent modules which are interconnected via communication interfaces and each of the modules has its own task. Communication between modules is done by message passing where the messages contain flow records, alerts, some statistics or preprocessed data.
The following picture shows all important parts of the system.
The project is divided into four repositories added as submodules:
Debian/Ubuntu:
apt-get install -y gawk bc autoconf automake gcc g++ libtool libxml2-dev make pkg-config libpcap-dev libidn11-dev bison flex
RHEL/CentOS/Fedora:
yum install -y bc autoconf automake gcc gcc-c++ libtool libxml2-devel make pkg-config libpcap-devel libidn-devel bison flex
Note: Latest systems (e.g. Fedora) use dnf
instead of yum
.
There are three different ways of installation of the NEMEA system covered in this document: vagrant, binary packages and source codes.
To try the system "out-of-box", you can use Vagrant. For more information see ./vagrant/.
Latest RPM packages can be found in COPR: https://copr.fedorainfracloud.org/groups/g/CESNET/coprs/ The NEMEA repository is at https://copr.fedorainfracloud.org/coprs/g/CESNET/NEMEA/
The repository can be added trough dnf copr enable
or by using the following commands (for CentOS/Fedora):
wget -O /etc/yum.repos.d/cesnet-nemea.repo https://copr.fedorainfracloud.org/coprs/g/CESNET/NEMEA/repo/epel-7/group_CESNET-NEMEA-epel-7.repo
rpm --import https://copr-be.cloud.fedoraproject.org/results/@CESNET/NEMEA/pubkey.gpg
After installation of the COPR repository, NEMEA can be installed as any other package (run as root/sudo):
yum install nemea
Note: Latest systems (e.g. Fedora) use dnf
instead of yum
.
For development purposes, there is nemea-framework-devel
package that installs
all needed development files and docs. To install this package, also add the NEMEA-testing
repository.
Currently, we do not have .deb packages (for Debian/Ubuntu/...) but we are working on it. Please follow installation from source codes
The whole system is based on GNU/Autotools build system that makes dependency checking and building process much more easier.
To clone the NEMEA repositories, use:
git clone --recursive https://github.com/CESNET/nemea
After successful clone and dependencies installation (!), use:
./bootstrap.sh
that will create configure
scripts and other needed files.
The configure
script supplies various possibilities of
configuration and it uses some environmental variables that influence the build
and compilation process. For more information see:
./configure --help
We recommend to set paths according to the used operating system, e.g.:
./configure --enable-repobuild --prefix=/usr --bindir=/usr/bin/nemea --sysconfdir=/etc/nemea --libdir=/usr/lib64
After finishing ./configure
, build process can be started by:
make
The make(1) tool has various parameters, to build the NEMEA package faster on multicore systems, we recommend to use parameter -j with the number of jobs that should be run in parallel.
When the compilation process ends without any error, the package can be installed
into paths that were set by configure
. It is recommended NOT to change
target paths by passing variables directly to make(1).
The installation can be done by (usually it requires root / sudo):
make install
Congratulations, the whole NEMEA system should be installed right now... :-)
NEMEA modules using have two implicit arguments. module -h
for help
(optional) and module -i IFC_SPEC
for communication interface (IFC)
specification. The -i
parameter is mandatory for all NEMEA modules.
Module help -h
The example below shows part of help output of logger. It contains module's name, description, number of input and output IFC, modules parameters and TRAP library parameters (common for all modules).
TRAP module, libtrap version: 0.7.6 b258bb4
===========================================
Name: Logger
Inputs: variable
Outputs: 0
Description:
This module logs all incoming UniRec records to standard output or into a specified file. Each record
is written as one line containing values of its fields in human-readable format separated by chosen
delimiters (CSV format). If you use more than one input interface you have to specify output format by
parameter "-o".
Usage: logger [COMMON]... [OPTIONS]...
Parameters of module [OPTIONS]:
-------------------------------
-w --write <string> Write output to FILE instead of stdout (rewrite the file).
-a --append <string> Write output to FILE instead of stdout (append to the end).
-t --title Write names of fields on the first line.
-c --cut <uint32> Quit after N records are received, 0 can be useful in combination
with -t to print UniRec.
Common TRAP parameters [COMMON]:
--------------------------------
-h [trap,1] If no argument, print this message. If "trap" or 1 is given, print
TRAP help.
-i IFC_SPEC Specification of interface types and their parameters, see "-h trap"
(mandatory parameter).
-v Be verbose.
Environment variables that affects output:
------------------------------------------
LIBTRAP_OUTPUT_FORMAT If set to "json", information about module is printed in JSON format.
PAGER Show the help output in the set PAGER.
Interface specifier -i
The -i
parameter with the interface specifier IFC_SPEC (module -i IFC_SPEC
)
specifies modules interfaces - their types and parameters. The interface
specifier has the following format:
<IFC 1>,<IFC 2>,...,<IFC N>
where <IFC x>
looks like
<type>:<par1>:<par2>:...:<parN>
.
<type>
can be one of the following: t
- TCP socket (for remote
communication), u
- UNIX socket (for local communication), b
- blackhole to
drop all messages during sending, f
- File IFC.
Interfaces are separated by ,
and their parameters are separated by :
.
Input IFCs must be specified at first, output IFCs follow. Examples below show
Example:
module1 -i t:address:port1,t:port2
module1 uses TCP interfaces (for machine to machine communication). Let's
assume it has one input and one output interface (number of input and output
IFCs is given by programmer of the module). Therefore, input IFC will connect
to address:port1
and output IFC will listen on port2
.
TCP type of IFC expects mandatory parameter port
and optionally, additional
parameter address
(as it is used in example). Default value of address
is
localhost
.
Example:
module2 -i u:sock1,u:sock2
UNIX type of IFC expects unique identifier of the socket. For compatibility
with TCP IFC, address
can be specified but it has no effect!
Important findings:
Detailed information and another examples of IFC_SPEC can be found in libtrap/README.ifcspec.md.
Let´s try to interconnect logreplay and logger modules to see them communicate. Logreplay module has one output IFC. It reads CSV file created by logger module and sends it in UniRec format. Logger has one input interface and it logs all incoming UniRec records to standard output or into specified file in CSV format. These two modules can be interconnected using one input IFC and one output IFC.
use-cases/logger-repeater.sh can be used for the demonstration. With no parameter, it prints help with description. With generate parameter, the script creates a CSV file with header and 3 flow records (see use-cases/logger-repeater.sh#L53). Thereafter it executes logreplay and logger modules
logreplay -i "u:my_socket"` -f CSV_file
and
logger -i "u:my_socket" -t
Logreplay has one UNIX output IFC listening on my_socket and logger has one UNIX input IFC which connects to my_socket.
To see the effect, go to use-cases/
and execute the script:
./logger-repeater.sh generate
It should print exactly the same output as generated CSV tmp input (header and
3 records). In
use-cases there are
more examples with basic modules.
logreplay
is one of possible ways of getting data into the NEMEA system.
Other data sources are discussed later in Get flows to your system.
This section shows how to deploy NEMEA in four steps.
It only covers the basics needed to run the system in its default configuration. Keep in mind that NEMEA was designed as a very flexible framework, so every user can (and should) adjust the configuration of NEMEA modules to their own purposes.
First of all, the whole system (NEMEA Framework, Modules, Detectors and Supervisor) has to be installed. Follow installation instructions to install the system from RPM or from source codes.
To avoid manual control of the system, there is NEMEA Supervisor. It is a central management and monitoring tool of the system and it takes care of running modules according to a specified XML configuration.
We need to prepare XML configuration file for Supervisor. Fortunately, almost everything is already done.
After installation (from RPM or from source codes with recommended configure
parameters), there are 2 important paths with configurations:
/ush/share/nemea-supervisor/
- contains default prepared XML configuraions of all NEMEA modules (like nemea-supervisor/configs/)/etc/nemea/
- contains XML configuration file for Supervisor and directories with used modules configurations (they are empty after installation)Note: these two paths depend on datarootdir and sysconfdir parameters of the configure
script during the installation.
The only thing we have to do is this (probably with sudo / root):
cp -r /usr/share/nemea-supervisor/*/ /etc/nemea
After this command, supervisor will use default configurations of the modules.
It is shown in nemea-supervisor/configs/supervisor_config_template.xml.in#L8
that the paths from sysconfdir
(/etc/nemea/
in our case) are included in the
configuration file. For detailed information about supervisor configuration
see README of Supervisor.
Once the configurations are prepared, modules can be managed by Supervisor. It can be easily started as a systemd service with
service nemea-supervisor start
(recommended, probably with root / sudo)
or manually
/usr/bin/nemea/supervisor --daemon -T /etc/nemea/supervisor_config_template.xml -L /var/log/nemea-supervisor
Note: manual approach does not change UID that supervisor runs with.
Contrary, using service
, NEMEA runs as nemead
UID and nemead
GID.
See all service commands in
README and all
program parameters with /usr/bin/nemea/supervisor -h
. You can also check
whether the process is running or not with ps -ef | grep supervisor
.
If Supervisor has not started successfully, it should print error info directly
to system log (in case of service), which can be browsed with journalctl -xe
,
or to stdout (in case of manual start). Runtime errors and events can be
found in supervisor_log
file located in the -L directory
(/var/log/nemea-supervisor
by default).
Now we can connect to running supervisor with supervisor client simply with
command supcli
. The menu with options is described in detail
in README.
After pressing number 4 and enter, it prints current status of the system.
By default, all detectors and loggers (except flow_meter logger) should be
enabled and running.
The modules are running, but they don't receive any data yet. We need to send some flow data to the system...
IPFIXcol
(recommended) Use IPFIXcol to collect NetFlow/IPFIX data from routers/probes and an IPFIXcol unirec plugin to re-send the data to NEMEA.
FlowMeter
Use NEMEA internal flow exporter (flow_meter module).
NfReader
NfReader reads nfdump files and sends flow records in UniRec format on its output TRAP interface.
LogReplay
LogReplay converts CSV format of data, from logger module to UniRec format and sends it to the output interface.
Important: Nemea-Framework has to be installed in advance. Follow installation instructions
Let ~/mighty-module/
be the directory we want to develop our module in (replace path ~/mighty-module/
in all commands with another directory if needed) and mighty_module the name of our module. We will use example module as a template - copy the directory nemea-framework/examples/c/module/ to ~/mighty-module/
.
In ~/mighty-module/configure.ac
update the following lines
AC_INIT([example_module], [1.0.0], [[email protected]])
AC_CONFIG_SRCDIR([example_module.c])
with
AC_INIT([migty_module], [1.0.0], [YOUR EMAIL ADDRESS])
AC_CONFIG_SRCDIR([mighty_module.c])
In ~/mighty-module/Makefile.am
update the following lines
bin_PROGRAMS=example_module
example_module_SOURCES=example_module.c fields.c fields.h
example_module_LDADD=-lunirec -ltrap
with
bin_PROGRAMS=mighty_module
mighty_module_SOURCES=mighty_module.c fields.c fields.h
mighty_module_LDADD=-lunirec -ltrap
Finally, execute
mv ~/mighty-module/example_module.c ~/mighty-module/mighty_module.c
to rename the source file.
Execute the following commands in ~/might-module/
:
autoreconf -i
./configure
make
make install
The example module already links TRAP (libtrap) and UniRec libraries. It is a simple module with one input and one output interface which receives on input inteface a message in UniRec format with two numbers and sends them together with their sum to output interface.
The code contains comments but here is the list of important operations:
Generated doxygen doc for module developers: https://rawgit.com/CESNET/Nemea-Framework/master/libtrap/doc/doxygen/html/index.html
Generated doxygen doc for libtrap developers: https://rawgit.com/CESNET/Nemea-Framework/master/libtrap/doc/devel/html/index.html
Generated doxygen doc: https://rawgit.com/CESNET/Nemea-Framework/master/unirec/doc/html/index.html
After executing ~/mighty-module/mighty_module -h
, program prints help which contains information from module info structure:
Now just modify the algorithm in the main loop and the job is done :-)
This section is for those who has already deployed the system (Deploy NEMEA section) and wants to add their module to the running configuration. It can be done in 3 steps:
configure
script during the installation, both the configuration file and the directories should be located in /etc/nemea
, otherwise check the paths in the configuration file the Supervisor is running with. Than copy the file to one of the directories you want e.g. cp ./your_module.sup /etc/nemea/others
.supcli
command and select option 6 reload configuration. New module should be added and if the enabled flag is set to true, it should be also running.For detailed information about Supervisor configuration see its README.
We kindly ask anyone using NEMEA for research and write an academic paper about it to cite NEMEA as follows:
@inproceedings{nemea16,
author = {Tomas Cejka, Vaclav Bartos, Marek Svepes, Zdenek Rosa, Hana Kubatova},
title = {NEMEA: A Framework for Network Traffic Analysis},
booktitle = {12th International Conference on Network and Service Management (CNSM 2016)},
year = 2016
}
Tomas Cejka, Václav Bartoš, Marek Svepes, Zdenek Rosa, Hana Kubatova. NEMEA: A Framework for Network Traffic Analysis. In 12th International Conference on Network and Service Management (CNSM 2016), Montreal, Canada, 2016.
Zdenek Rosa, Tomas Cejka, Martin Zadnik, Viktor Puš. Building a Feedback Loop to Capture Evidence of Network Incidents. In 12th International Conference on Network and Service Management (CNSM 2016), Montreal, Canada, 2016.
Tomas Cejka, Marek Svepes. Analysis of Vertical Scans Discovered by Naive Detection. Management and Security in the Age of Hyperconnectivity: 10th IFIP WG 6.6 International Conference on Autonomous Infrastructure, Management, and Security, AIMS 2016.
Tomáš Čejka, Radoslav Bodó, Hana Kubátová: Nemea: Searching for Botnet Footprints. In: Proceedings of the 3rd Prague Embedded Systems Workshop (PESW), Prague, CZ, 2015.
Tomáš Čejka, Václav Bartoš, Lukáš Truxa, Hana Kubátová: Using Application-Aware Flow Monitoring for SIP Fraud Detection. In: Proc. of 9th International Conference on Autonomous Infrastructure, Management and Security (AIMS15), 2015.
Tomáš Čejka, Zdeněk Rosa and Hana Kubátová: Stream-wise Detection of Surreptitious Traffic over DNS. In: Proc. of 19th IEEE International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD 2014). Athens, 2014.
Václav Bartoš, Martin Žádník, Tomáš Čejka: Nemea: Framework for stream-wise analysis of network traffic, CESNET technical report 6/2013.