Intelmq Versions Save

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.

3.3.0

2 months ago

3.3.0 (2024-03-01)

Configuration

  • Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
    • auth_by_ssl_client_certificate (Boolean, default: true; if false then ssl_client_certificate and ssl_client_certificate_key will be ignored);
    • username (STOMP authentication login, default: "guest"; to be used only if auth_by_ssl_client_certificate is false);
    • password (STOMP authentication passcode, default: "guest"; to be used only if auth_by_ssl_client_certificate is false).
  • Add the possibility to set the ssl_ca_certificate configuration parameter for intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output to an empty string - which means that the SSL machinery used for STOMP communication will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).

Core

  • intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
  • intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
  • Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
  • intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp), which provides certain common STOMP-bot-specific operations, factored out from intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski).
  • intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
  • intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
  • AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
  • Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).

Development

  • Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).

Data Format

Bots

Collectors

  • intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
    • Drop support for versions of stomp.py older than 4.1.12.
    • Update the code to support new versions of stomp.py, including the latest (8.1.0); fixes #2342.
    • Add support for authentication based on STOMP login and passcode, introducing three new configuration parameters (see above: Configuration).
    • Add support for loading the system’s default CA certificates, as an alternative to specifying the CA certificate(s) file path explicitly (see above: Configuration).
    • Fix (by carefully targeted monkey patching) certain security problems caused by SSL-related weaknesses that some versions of stomp.py suffer from.
    • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also, never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it did not work properly anyway).
    • Add coercion of the port config parameter to int.
    • Add implementation of the check hook (verifying, in particular, accessibility of necessary file(s)).
    • Remove undocumented and unused attributes of StompCollectorBot instances: ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
      • Minor fixes/improvements and some refactoring (see also above: Core...).
  • intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
  • intelmq.bots.collectors.shadowserver.collector_reports_api:
    • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
  • intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
    • Fix modified_pulses_only is always False.

Parsers

  • intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.shadowserver._config:
    • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
  • intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)

Experts

  • intelmq.bots.experts.jinja (PR#2417 by Mikk Margus Möll):
    • Add optional socket_perms and socket_group parameters to change file permissions on socket file, if it is in use.
  • intelmq.bots.experts.ripe (PR#2461 by Mikk Margus Möll):
    • Handle "No abuse contact found for" messages for non-ASN resources

Outputs

  • intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
    • Drop support for versions of stomp.py older than 4.1.12.
    • Update the code to support new versions of stomp.py, including the latest (8.1.0).
    • Add support for authentication based on STOMP login and passcode, introducing three new configuration parameters (see above: Configuration).
    • Add support for loading the system’s default CA certificates, as an alternative to specifying the CA certificate(s) file path explicitly (see above: Configuration).
    • Fix (by carefully targeted monkey patching) certain security problems caused by SSL-related weaknesses that some versions of stomp.py suffer from.
    • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot (ssl_ca_cert et consortes).
    • Add coercion of the port config parameter to int.
    • Add implementation of the check hook (verifying, in particular, accessibility of necessary file(s)).
    • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
    • Minor fixes/improvements and some refactoring (see also above: Core...).
  • intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
    • Try to reconnect on NotConnectedException.
  • intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
    • Fix ability to send with the default bcc
  • intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).

Documentation

  • Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
  • Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
  • Update/fix/improve the stuff related to the STOMP bots and integration with the n6's Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
  • Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
  • Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
  • Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).

Packaging

  • Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
  • debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).

Tests

Tools

  • intelmq_psql_initdb:
    • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
    • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).

Contrib

Known issues

3.2.1

8 months ago

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

All Bots

Fixes an issue which prevented bots from stopping gracefully after reloading. As logrotate reloads all bots regularly, this bug affects most IntelMQ installations.

Reverse DNS Expert

Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses. If the bot retrieved the PTR for 192.0.43.7, it was cached for 192.0.43.0/24 and used for all IP addresses in this range, for example for 192.0.43.8. IntelMQ version 3.2.1 fixes this issue.

The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.

3.2.0

9 months ago

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi. Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

  • intelmq.lib.utils:
    • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
  • Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
  • Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
  • intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
  • intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
  • intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
  • intelmq.lib.harmonization:
    • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
    • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
  • intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

  • CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

Bots

Collectors

  • intelmq.bots.collector.rt:
    • restrict python-rt to be below version 3.0 due to introduced breaking changes,
    • added support for Subject NOT LIKE queries,
    • added support for multiple values in ticket subject queries.
  • intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

  • intelmq.bots.parsers.shadowserver._config:
    • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
  • intelmq.bots.parsers.shadowserver._config:
    • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
    • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
    • Added 'Accessible-SIP' report. (PR#2348)
    • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
    • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
  • intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
  • intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
  • intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).

Experts

  • intelmq.bots.experts.sieve:
    • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
  • intelmq.bots.experts.cymru_whois:
    • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
    • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
  • intelmq.bots.experts.sieve:
    • Added :before and :after keywords (PR#2374)

Outputs

  • intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
  • intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
  • intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

  • API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

  • New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)

Tools

  • intelmqsetup:
    • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
  • contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
  • intelmq_psql_initdb:
    • Added parameter -o to set the output file destination. (by Sebastian Kufner)
  • intelmqctl:
    • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

  • intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
  • Breaking changes in 'rt' library (#2367).
  • Stomp collector failed (#2342).
  • Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
  • intelmq_psql_initdb does not work for SQLite (#2202).
  • intelmqsetup: should install a default state file (#2175).
  • Misp Expert - Crash if misp event already exist (#2170).
  • Turris greylist has been updated (#2167).
  • Spamhaus CERT parser uses wrong field (#2165).
  • Custom headers ignored in HTTPCollectorBot (#2150).
  • intelmqctl log: parsing syslog does not work (#2097).
  • Bash completion scripts depend on old JSON-based configuration files (#2094).
  • Bot configuration examples use JSON instead of YAML (#2066).
  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
  • Corrupt dump files when interrupted during writing (#870).

3.1.0

1 year ago

Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

  • Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).

Core

  • intelmq.lib.upgrades:
    • Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
    • Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
  • intelmq.lib.message:
    • Fix and pre-compile the regular expression for harmonization key names and also check keys in the extra. namespace (PR#2059 by Sebastian Wagner, fixes #1807).
  • intelmq.lib.bot.SQLBot was replaced by an SQLMixin in intelmq.lib.mixins.SQLMixin. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
    • Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
    • Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
  • Added an ExpertBot class - it should be used by all expert bots as a parent class
  • Introduced a module for IntelMQ related datatypes intelmq.lib.datatypes which for now only contains an Enum listing the four bot types
  • Added a bottype attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
  • Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script. They now reside in intelmq.lib.processmanager which also contains an interface definition the processmanager implementations must adhere to. Both the processmanagers and the intelmqctl script were cleaned up a bit. The LogLevel and ReturnType Enums were added to intelmq.lib.datatypes.
  • intelmq.lib.bot:
    • Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
    • Fix line recovery and message dumping of the ParserBot (PR#2192 by Sebastian Wagner).
      • Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
    • Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
    • Added default_fields parameter to ParserBot (PR#2293 by Filip Pokorný)
  • intelmq.lib.pipeline:
    • Changed BRPOPLPUSH to BLMOVE, because BRPOPLPUSH has been marked as deprecated by redis in favor of BLMOVE (PR#2149 and PR#2240 by Sebastian Waldbauer and Sebastian Wagner, fixes #1827, #2233).
  • intelmq.lib.utils:
    • Added wrapper resolve_dns for querying DNS, with the support for recommended methods from dnspython package in versions 1 and 2.
    • Moved line filtering inside RewindableFileHandle for easier handling and limiting number of temporary objects.
  • intelmq.lib.harmonization:
    • Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorný, fixes #2278)
    • Removes tzone argument from DateTime.from_timestamp and DateTime.from_epoch_millis
    • DateTime.from_timstamp now also allows string argument
  • Removes pytz global dependency
  • Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)

Development

  • Removed Python 3.6 from CI.
  • Enabled tests against Python 3.11.

Bots

  • Set the parent class of all bots to the correct bot class

Collectors

  • intelmq.bots.collectors.mail._lib:
    • Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
    • Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
  • intelmq.bots.collectors.blueliv: Fix Blueliv collector requirements (PR#2161 by Gethvi).
  • intelmq.bots.collectors.github_api._collector_github_api: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).
  • intelmq.bots.collectors.file.collector_file: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)
  • intelmq.bots.collectors.shadowserver.collector_reports_api.py: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)

Parsers

  • intelmq.bots.parsers.alienvault.parser_otx: Save CVE data in extra.cve instead of extra.CVE due to the field name restriction on lower-case characters (PR#2059 by Sebastian Wagner).

  • intelmq.bots.parsers.anubisnetworks.parser: Changed field name format from extra.communication.http.x_forwarded_for_#1 to extra.communication.http.x_forwarded_for_1 due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner).

  • intelmq.bots.parsers.dataplane.parser:

    • Add support for additional feeds (PR#2102 by Mikk Margus Möll).
      • DNS Recursion Desired
      • DNS Recursion Desired ANY
      • DNS Version
      • Protocol 41
      • SMTP Greet
      • SMTP Data
      • Telnet Login
      • VNC/RFB Login
    • Fix event object creation (PR#2298 by DigitalTrustCenter).

  • Removed intelmq.bots.parsers.malc0de: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178).

  • intelmq.bots.parsers.microsoft.parser_ctip:

    • New parameter overwrite (PR#2112 by Sebastian Wagner, fixes #2022).
    • Fix handling of field Payload.domain if it contains the same IP address as Payload.serverIp (PR#2144 by Mikk Margus Möll and Sebastian Wagner).
    • Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)

  • intelmq.bots.parsers.shodan.parser (PR#2117 by Mikk Margus Möll):

    • Instead of keeping track of extra.ftp.<something>.parameters, FTP parameters are collected together into extra.ftp.features as a list of said features, reducing field count.
    • Shodan field rsync.modules is collected.
    • Conversion functions can raise NoValueException with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into source.reverse_dns and fail to validate as a FQDN.
    • Variable _common_keys is moved out of the class.
    • _dict_dict_to_obj_list is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g vulns.CVE-2010-0001.cvss, CVE-2010-0002.cvss etc.
    • _get_first to get the first item from a list, with NoValueException raised on empty lists.
    • _get_first_hostname to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives NoValueException otherwise.
    • ssl.cert.serial and ssl.dhparams.generator, which may return both integers and strings, are converted to strings.
    • Changes to method apply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding the NoValueException handling described above.
    • Stops falsy values (False, 0) besides None from being filtered out.

  • intelmq.bots.parsers.shadowserver._config:

    • Added support for Accessible AMQP, Device Identification Report (IPv4 and IPv6) (PR#2134 by Mateo Durante).
    • Added file name mapping for SSL-POODLE-Vulnerable-Servers IPv6 (file name scan6_ssl_poodle) (PR#2134 by Mateo Durante).
    • Added Malware-URL, Sandbox-Connection, Sandbox-DNS, Accessible-AMQP, Open-AnonymouIs-MQTT, Accessible-QUIC, Accessible-SSH, SYNful-Knock, and Special (PR#2227 by elsif2)
    • Removed legacy reports Amplification-DDoS-Victim, CAIDA-IP-Spoofer, Darknet, Drone, Drone-Brute-Force, IPv6-Sinkhole-HTTP-Drone, Microsoft-Sinkhole, and Sinkhole-HTTP-Drone (PR#2227 by elsif2).
    • Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
    • Corrected "Accessible-AMQP" message_length type (int) and added "STUN" support (PR#2235 by elsif2).
    • Added amplification factor to UDP scan reports (PR#2238 by elsif2).
    • Added version and build_date to "Vulnerable-HTTP" report (PR#2238 by elsif2).
    • The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2): destination.fqdn (validate_fqdn) destination.url (convert_http_host_and_url) extra.browser_trusted (convert_bool) extra.duration (convert_int) extra.end_time (convert_date_utc) extra.freak_vulnerable (convert_bool) extra.ok (convert_bool) extra.password (validate_to_none) extra.ssl_poodle (convert_bool) extra.status (convert_int) extra.uptime (convert_int) extra.version (convert_to_none) source.network (validate_network)
    • The following report field names have changed to better represent their values: scan_rsync:extra.password renamed to extra.has_password scan_elasticsearch:status renamed to http_code
    • Added Accessible-HTTP-proxy and Open-HTTP-proxy (PR#2246 by elsif2).
    • Added http_agent to the Honeypot-DDoS report and added the DDoS-Participant report (PR#2303 by elsif2)
    • Added Accessible-SLP, IPv6 Accesssible-SLP, IPv6-DNS-Open-Resolvers, and IPv6-Open-LDAP-TCP reports (PR#2311 by elsif2)
    • Standardized response_length to response_size in Accessible-ICS and Open-MSSQL (PR#2311 by elsif2)

  • intelmq.bots.parsers.cymru.parser_cap_program: The parser mapped the hostname into source.fqdn which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169)

  • intelmq.bots.parsers.generic.parser_csv:

    • Use RewindableFileHandle to use the original current line for line recovery (PR#2192 by Sebastian Wagner).
    • Recovering CSV lines preserves the original line ending (PR#2280 by Kamil Mankowski, fixes #1597)

  • intelmq.bots.parsers.autoshun.parser: Removed, as the feed is discontinued (PR#2214 by Sebastian Waldbauer, fixes #2162).

  • intelmq.bots.parsers.openphish.parser_commercial: Refactored complete code (PR#2160 by Filip Pokorný).

    • Fixes wrong mapping of host field to source.fqdn when the content was an IP address.
    • Adds newly added fields in the feed.

  • intelmq.bots.parsers.phishtank.parser: Refactored code (PR#2270 by Filip Pokorný)

    • Changes feed URL to JSON format (contains more information). The URL needs to by manually updated in the configuration!
    • Adds fields from the JSON feed.

  • intelmq.bots.parsers.dshield.parser_domain: Has been removed, due to the feed is discontinued. (PR#2276 by Sebastian Waldbauer)

  • intelmq.bots.parsers.abusech.parser_ip: Removed (PR#2268 by Filip Pokorný).

  • intelmq.bots.parsers.abusech.parser_domain: Removed (PR#2268 by Filip Pokorný).

  • intelmq.bots.parsers.abusech.parser_feodotracker: Added new parser bot (PR#2268 by Filip Pokorný)

    • Changes feed URL to JSON format (contains more information).
    • Adds fields from the JSON feed.

  • intelmq.bots.parsers.generic.parser_csv: Parameter type is deprecated, default_fields should be used. (PR#2293 by Filip Pokorný)

  • intelmq.bots.parsers.generic.parser_csv: Parameter skip_header now allows also integer as a fixed number of lines to skip. (PR#2313 by Filip Pokorný)

  • intelmq.bots.parsers.taichung.parser: Removed (PR#2266 by Filip Pokorný)

Experts

  • intelmq.bots.experts.domain_valid: New bot for checking domain's validity (PR#1966 by Marius Karotkis).
  • intelmq.bots.experts.truncate_by_delimiter.expert: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
  • intelmq.bots.experts.remove_affix: Remove prefix or postfix strings from a field (PR#1965 by Marius Karotkis).
  • intelmq.bots.experts.asn_lookup.expert: Fixes update-database script on the last few days of a month (PR#2121 by Filip Pokorný, fixes #2088).
  • intelmq.bots.experts.threshold.expert: Correctly use the standard parameter redis_cache_ttl instead of the previously used parameter timeout (PR#2155 by Karl-Johan Karlsson).
  • intelmq.bots.experts.jinja2.expert: Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
  • intelmq.bots.experts.asn_lookup.expert, intelmq.bots.experts.domain_suffix.expert, intelmq.bots.experts.maxmind_geoip.expert, intelmq.bots.experts.recordedfuture_iprisk.expert, intelmq.bots.experts.tor_nodes.expert: New parameter autoupdate_cached_database to disable automatic updates (downloads) of cached databases (PR#2180 by Sebastian Wagner).
  • intelmq.bots.experts.url.expert: New bot for extracting additional information from source.url and/or destination.url (PR#2315 by Filip Pokorný).

Outputs

  • Removed intelmq.bots.outputs.postgresql: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR#2045 by Birger Schacht).
  • Added intelmq.bots.outputs.rpz_file.output to create RPZ files (PR#1962 by Marius Karotkis).
  • Added intelmq.bots.outputs.bro_file.output to create Bro intel formatted files (PR#1963 by Marius Karotkis).
  • intelmq.bots.outputs.templated_smtp.output:
    • Add new function from_json() (which just calls json.loads() in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
    • Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
  • intelmq.bots.outputs.sql:
    • For PostgreSQL, escape Nullbytes in text to prevent "unsupported Unicode escape sequence" issues (PR#2223 by Sebastian Wagner, fixes #2203).

Documentation

  • Feeds: Add documentation for newly supported dataplane feeds, see above (PR#2102 by Mikk Margus Möll).
  • Installation: Restructured the whole document to make it clearer and straight-forward (PR#2113 by Sebastian Wagner).
  • Add workaround for https://github.com/sphinx-doc/sphinx/issues/10701 (PR#2225 by Sebastian Wagner, kudos @yarikoptic, fixes #2224).
  • Fix wrong operator for list-contains-value operation in sieve expert documentation (PR#2256 by Filip Pokorný).
  • Added documentation on default_fields parameter (PR#2293 by Filip Pokorný).
  • Updated documentation on skip_header parameter (PR#2313 by Filip Pokorný).
  • Viriback Unsafe Sites feed replaced with Viriback C2 Tracker. (PR#2266 by Filip Pokorný)
  • Netlab 360 Mirai Scanner feed removed as it is discontinued. (PR#2266 by Filip Pokorný)
  • Benkow Malware Panels Tracker feed changed parser configuration. (PR#2266 by Filip Pokorný)
  • Taichung feed removed as it is discontinued. (PR#2266 by Filip Pokorný)
  • Added new URL Expert bot. (PR#2315 by Filip Pokorný)

Packaging

  • Remove deleted intelmq.bots.experts.sieve.validator from executables in setup.py (PR#2256 by Filip Pokorný).
  • Run the geoip database cron-job twice a week (PR#2285 by Filip Pokorný).

Tests

  • Add GitHub Action to run regexploit on all Python, JSON and YAML files (PR#2059 by Sebastian Wagner).
  • intelmq.lib.test:
    • Decorator skip_ci also detects dpkg-buildpackage environments by checking the environment variable DEB_BUILD_ARCH (PR#2123 by Sebastian Wagner).
    • Fixing regex to catchall after python version and process ID, add tests for it (PR#2216 by Sebastian Waldbauer and Sebastian Wagner, fixes #2185)
  • Also test on Python 3.10 (PR#2140 by Sebastian Wagner).
  • Switch from nosetests to pytest, as the former does not support Python 3.10 (PR#2140 by Sebastian Wagner).
  • CodeQL Github Actions exponential backtracking on strings fixed. (PR#2148 by Sebastian Waldbauer, fixes #2138)
  • Reverse DNS expert tests: remove outdated failing test test_invalid_ptr (PR#2208 by Sebastian Wagner, fixes #2206).
  • Add test dependency requests_mock to the development extra requirements in setup.py (PR#2210 by Sebastian Wagner).
  • Threshold Expert tests: Use environment variable INTELMQ_PIPELINE_HOST as redis host, analogous to other tests (PR#2209 by Sebastian Wagner, fixes #2207).
  • Remove codecov action as it failed regularly (PR#2237 by Sebastian Wagner, fixes #2229).
  • intelmq.lib.test.BotTestCase: Adds skip_checks variable to not fail on non-empty messages from calling check function (PR#2315 by Filip Pokorný).

Tools

  • intelmqctl:
    • fix process manager initialization if run non-interactively, as intelmqdump does it (PR#2189 by Sebastian Wagner, fixes 2188).
    • check: handle SyntaxError in bot modules and report it without breaking execution (fixes #2177)
    • Privilege drop before logfile creation (PR#2277 by Sebastian Waldbauer, fixes 2176)
  • intelmqsetup: Revised installation of manager by building the static files at setup, not build time, making it behave more meaningful. Requires intelmq-manager >= 3.1.0 (PR#2198 by Sebastian Wagner, fixes #2197).
  • intelmqdump: Respected global and per-bot custom settings of logging_path (fix #1605).

Contrib

  • logrotate: Move compress and ownership rules to the IntelMQ-blocks to prevent that they apply to other files (PR#2111 by Sebastian Wagner, fixes #2110).

Known issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

  • intelmq_psql_initdb does not work for SQLite (#2202).
  • intelmqsetup: should install a default state file (#2175).
  • Misp Expert - Crash if misp event already exist (#2170).
  • Turris greylist has been updated (#2167).
  • Spamhaus CERT parser uses wrong field (#2165).
  • Custom headers ignored in HTTPCollectorBot (#2150).
  • Missing commas in SQL query for separate Events table (#2125).
  • intelmqctl log: parsing syslog does not work (#2097).
  • Bash completion scripts depend on old JSON-based configuration files (#2094).
  • Bot configuration examples use JSON instead of YAML (#2066).
  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
  • Corrupt dump files when interrupted during writing (#870).

3.1.0-rc1

1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/certtools/intelmq/compare/3.0.2...3.1.0-rc1

3.0.2

2 years ago

Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

  • intelmq.lib.bot.CollectorBot: Fixed an issue with within the new_report function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
  • intelmq.lib.bot.Bot: Make private members __is_multithreadable and __collector_empty_process protected members _is_multithreadable and _collector_empty_process to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108). Also affected and adapted bots by this change are:
    • intelmq.bots.collectors.api.collector_api
    • intelmq.bots.collectors.stomp.collector
    • intelmq.bots.experts.splunk_saved_search.expert
    • intelmq.bots.experts.threshold.expert
    • intelmq.bots.outputs.file.output
    • intelmq.bots.outputs.misp.output_api
    • intelmq.bots.outputs.misp.output_feed
    • intelmq.bots.outputs.tcp.output
    • intelmq.bots.outputs.udp.output
  • intelmq.lib.cache: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).

Bots

Experts

  • intelmq.bots.experts.domain_suffix.expert: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).

Documentation

  • Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).

Known issues

See open bug reports for a more detailed list.

  • ParserBot: erroneous raw line recovery in error handling (#1850).

3.0.1

2 years ago

Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

  • intelmq.lib.bot_debugger: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus Möll).
  • intelmq.lib.pipeline: Fix handling of load_balance parameter (PR#2027 by Mikk Margus Möll).
  • intelmq.lib.bot: Fix handling of parameter destination_queues if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).

Bots

Collectors

  • intelmq.bots.collectors.shodan.collector_stream: Fix access to parameters, the bot wrongly used self.parameters (PR#2020 by Mikk Margus Möll).
  • intelmq.bots.collectors.mail.collector_mail_attach: Add attachment file name as extra.file_name also if the attachment is not compressed (PR#2021 by Alex Kaplan).
  • intelmq.bots.collectors.http.collector_http_stream: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).

Parsers

  • intelmq.bots.parsers.microsoft.parser_ctip: Map Payload.domain to destination.fqdn instead of extra.payload.domain as it matches to destination.ip from DestinationIp (PR#2023 by Sebastian Wagner).
  • Removed intelmq.bots.parsers.malwaredomains because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
  • intelmq.bots.parsers.shadowserver.config:
    • Add support for feed "Vulnerable SMTP Server" (PR#2037 by Mikk Margus Möll).
    • Fix differentiation between feeds "Accessible HTTP" and "Vulnerable HTTP" (PR#2037 by Mikk Margus Möll, fixes #1984).
    • Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
    • Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus Möll and Sebastian Wagner).
    • Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
    • Add support for feed "Honeypot HTTP Scan" (PR#2047 by Mikk Margus Möll).
    • Update filename mapping for changed filename of feed "Accessible-MSRDPUDP" (PR#2060 by abr4xc).

Experts

  • intelmq.bots.experts.gethostbyname.expert: Handle numeric values for the gaierrors_to_ignore parameter (PR#2073 by Sebastian Wagner, fixes #2072).
  • intelmq.bots.experts.filter.expert: Fix handling of empty-string parameters not_after and not_before (PR#2075 by Sebastian Wagner, fixes #2074).

Outputs

  • intelmq.bots.outputs.mcafee.output_esm_ip: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
  • intelmq.bots.outputs.misp.output_api: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
  • intelmq.bots.outputs.smtp.output: Add Content-Disposition-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).

Documentation

  • Various formatting fixes (by Sebastian Wagner).
  • Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
  • Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).

Packaging

  • intelmq-update-database crontab: Add missing recordedfuture_iprisk update call (by Sebastian Wagner).

Tests

  • Replace calls to deprecated/undocumented logging.warn with logging.warning (by Sebastian Wagner, fixes #2013).
  • intelmq.tests.bots.experts.rdap.test_expert: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).
  • intelmq.tests.bots.collectors.mail.test_collector_attach: Test text attachment (by Sebastian Wagner).

Tools

  • intelmqctl:
    • Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
    • Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
    • Do not use hardcoded logging path in /opt/intelmq, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).

Known issues

See open bug reports for a more detailed list.

  • ParserBot: erroneous raw line recovery in error handling (#1850).

3.0.0.rc1

2 years ago

This is just an intermediate unstable release towards 3.0.0. Please do not use it in production.

https://intelmq.readthedocs.io/

3.0.0

2 years ago

Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Configuration

  • The BOTS file is no longer used and has been removed (by Sebastian Wagner).
  • The defaults.conf file is no longer used and has been removed (PR#1814 by Birger Schacht).
  • The pipeline.conf file is no longer used and has been removed (PR#1849 by Birger Schacht).
  • The runtime.conf was renamed to runtime.yaml and is now in YAML format (PR#1812 by Birger Schacht).

Core

  • intelmq.lib.harmonization:
    • New class ClassificationTaxonomy with fixed list of taxonomies and sanitiation
  • intelmq.lib.bot:
    • Handle InvalidValue exceptions upon message retrieval by dumping the message instead of repeating endlessly (#1765, PR#1766 by Filip Pokorný).
    • Rewrite of the parameter loading and handling, getting rid of the parameters member (PR#1729 by Birger Schacht).
    • The pipeline is now initialized before the call of init to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR#1982 by Sebastian Wagner).
  • intelmq.lib.exceptions:
    • InvalidValue: Add optional parameter object (PR#1766 by Filip Pokorný).
  • intelmq.lib.utils:
    • New function list_all_bots to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer).
    • New function get_bots_settings to return the effective bot parameters, with global parameters applied.
    • Removed deprecated function create_request_session_from_bot (PR#1997 by Sebastian Wagner, #1404).
    • parse_relative: Add support for parsing minutes and seconds (PR#1857 by Sebastian Wagner).
  • intelmq.lib.bot_debugger:
    • Set bot's logging_level directly in __init__ before the bot's initialization by changing the default value (by Sebastian Wagner).
    • Rewrite load_configuration_patch by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
    • Do not rely on the runtime configuration's group setting of bots to determine the required message type of messages given on the command line (PR#1949 by Sebastian Wagner).

Development

  • rewrite_config_files.py: Removed obsolete BOTS-file-related rewriting functionality.
  • A Github action that checks for reuse compliance of all the license and copyright headers was added.
  • PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).

Data Format

The IntelMQ Data Harmonization ("DHO") is renamed to IntelMQ Data Format ("IDF"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810). Update allowed classification fields to version 1.3 (2021-05-18) (fixes #1409, #1476).

  • The taxonomy abusive content has been renamed to abusive-content.
  • The taxonomy information content security has been renamed to information-content-security.
    • The validation of type unauthorised-information-access has been fixed, a bug prevented the use of it.
    • The validation of type unauthorised-information-modification has been fixed, a bug prevented the use of it.
    • The type leak has been renamed to data-leak.
    • The type dropzone has been removed. Taxonomy other with type other and identifier dropzone can be used instead. Ongoing discussion in the RSIT WG.
  • The taxonomy intrusion attempts has been renamed to intrusion-attempts.
  • For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409):
    • The type compromised has been renamed to system-compromise.
    • The type unauthorized-command has been merged into system-compromise.
    • The type unauthorized-login has been merged into system-compromise.
    • The type backdoor has been merged into system-compromise (PR#1995 by Sebastian Wagner, addresses #1409).
    • The type defacement has been merged into taxonomy information-content-security, type unauthorised-information-modification (PR#1994 by Sebastian Wagner, addresses #1409).
  • The taxonomy information gathering has been rename to information-gathering.
  • The taxonomy malicious code has been renamed to malicious-code.
    • The type c2server has been renamed to c2-server.
    • The type malware has been integrated into infected-system and malware-distribution, respectively (PR#1917 by Sebastian Wagner addresses #1409).
    • The type ransomware has been integrated into infected-system.
    • The type dga domain has been moved to the taxonomy other renamed dga-domain (PR#1992 by Sebastian Wagner fixes #1613).
  • For the taxonomy 'availability', the type misconfiguration is new.
  • For the taxonomy 'other', the type unknown has been renamed to undetermined.
  • For the taxonomy 'vulnerable':
    • The type vulnerable client has been renamed to vulnerable-system.
    • The type vulnerable service has been renamed to vulnerable-system.

Bots

  • The parameters handling of numerous bots has been refactored (PR#1751, PR#1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).

Collectors

  • Remove intelmq.bots.collectors.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761, closes #1614)
  • intelmq.bots.collectors.mail._lib: Added parameter mail_starttls for STARTTLS in all mail collector bots (PR#1831 by Marius Karotkis, fixes #1128).
  • Added intelmq.bots.collectors.fireeye: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein).
  • intelmq.bots.collectors.api.collector_api (PR#1987 by Mikk Margus Möll, fixes #1986):
    • Added UNIX socket capability.
    • Correctly close the IOLoop in the shutdown method to fix reload.
  • intelmq.bots.collectors.rt.collector_rt (PR#1997 by Sebastian Wagner, #1404):
    • compatibility with the deprecated parameter unzip_attachment (removed in 2.1.0) was removed.

Parsers

  • Added intelmq.bots.parsers.fireeye: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
  • intelmq.bots.parsers.shadowserver._config:
    • Improved the feed-mapping and all conversion functions (PR#1971 by Mikk Margus Möll).
  • intelmq.bots.parsers.generic.parser_csv:
    • Fix handling of empty string values for parameter time_format (by Sebastian Wagner).

Experts

  • intelmq.bots.experts.domain_suffix.expert:
    • Added --update-database option to update domain suffix database (by Sebastian Wagner).
    • Fix check method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
  • Added intelmq.bots.experts.http.expert_status: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly).
  • Added intelmq.bots.experts.http.expert_content: A bot that fetches an HTTP resource and checks if it contains a specific string.
  • Added intelmq.bots.experts.lookyloo.expert: A bot that sends requests to a lookyloo instance & adds screenshot_url to the event (PR#1844 by Sebastian Waldbauer, fixes #1048).
  • Added intelmq.bots.experts.rdap.expert: A bot that checks the rdap protocol for an abuse contact for a given domain.
  • intelmq.bots.experts.sieve.expert:
    • Add operators for comparing lists and sets (PR#1895 by Mikk Margus Möll):
      • :equals
      • :overlaps
      • :supersetof
      • :subsetof
      • :equals
    • Add support for comparing boolean values (PR#1895 by Mikk Margus Möll).
    • Add support for rule negation with ! (PR#1895, PR#1923 by Mikk Margus Möll).
    • Add support for values types float, int, bool and string for all lists items (PR#1895 by Mikk Margus Möll).
    • Add actions for lists (PR#1895 by Mikk Margus Möll).
      • append
      • append! (forced/overwriting)
    • Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR#1895, PR#1923 by Mikk Margus Möll).
    • Nested if statements, plus mixed actions and actions in the same scope (PR #1923 by Mikk Margus Möll).
    • The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR #1923 by Mikk Margus Möll).
    • Drop the :notcontains operator, as it made is redundant by generic negation: ! foo :contains 'x' instead of foo :notcontains 'x' (PR#1957 by Mikk Margus Möll).
    • Split string and numeric matches into single- and multivalued variants, with the relevant new operators :in, :containsany and :regexin for string lists, and :in for numeric value lists (PR#1957 by Mikk Margus Möll).
      • Removed the == operator for lists, with the previous meaning of :in. Have a look at the NEWS.md for more information.
  • Added intelmq.bots.experts.uwhoisd: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot).
  • Removed deprecated intelmq.bots.experts.ripencc_abuse_contact.expert. It was replaced by intelmq.bots.experts.ripe.expert and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404).
  • intelmq.bots.experts.modify.expert:
    • Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
  • Added intelmq.bots.experts.aggregate: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer)
  • Added intelmq.bots.experts.tuency: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).

Outputs

  • Remove intelmq.bots.outputs.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761, closes #1614)
  • intelmq.bots.outputs.smtp: Add more debug logging (PR#1949 by Sebastian Wagner).
  • Added new bot intelmq.bots.outputs.templated_smtp (PR#1901 by Karl-Johan Karlsson).

Documentation

  • Updated user and developer documentation to reflect the removal of the BOTS file (PR#1780 by Birger Schacht).
  • Bots documentation:
    • Added anchors to all bot sections derived from the module names for easier linking (PR#1943 by Sebastian Wagner fixes part of certtools/intelmq-api#4).
  • License and copyright information was added to all the bots.
  • Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner).
  • Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer).
  • Improved n6 interoperability documentation by adding more graphs and illustrations (PR#1991 by Sebastian Wagner).
  • Feed documentation generation: fix and simplify formatting of parameters of types lists, non-string values have been ill-treated (by Sebastian Wagner).
  • Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner).

Packaging

  • Docker images tagged with certat/intelmq-full:develop are built and published on every push to the develop branch (PR#1753 by Sebastian Waldbauer).
  • Adapt packaging to IntelMQ 3.0 changes: ruamel.yaml dependency, changed configuration, updated database-update scripts (by Birger Schacht and Sebastian Wagner).

Tests

  • intelmq.tests.lib.test_bot:
    • Add test case for a raised InvalidValue exception upon message retrieval (#1765, PR#1766 by Filip Pokorný and Sebastian Wagner).
  • intelmq.lib.test:
    • Compare content of the output field as dictionaries, not as string in assertMessageEqual (PR#1975 by Karl-Johan Karlsson).
    • Support multiple calls to run_bot from test cases (PR#1989 by Sebastian Wagner).
      • Split prepare_source_queue out of prepare_bot.
      • Added new optional parameter stop_bot to run_bot.

Tools

  • intelmqdump (PR#1997 by Sebastian Wagner, #1404):
    • The command e for deleting single entries by given IDs has been merged into the command d ("delete"), which can now delete either entries by ID or the whole file.
    • The command v for editing entries has been renamed to e ("edit").

Contrib

  • eventdb:
    • Added separate-raws-table.sql (PR#1985 by Sebastian Wagner).
  • cron-jobs: Removed the deprecated update scripts (PR#1997 by Sebastian Wagner, #1404):
    • update-asn-data
    • update-geoip-data
    • update-tor-nodes
    • update-rfiprisk-data in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in contrib/cron-jobs/intelmq-update-database.

Known issues

  • ParserBot: erroneous raw line recovery in error handling (#1850).
  • ruamel.yaml loader and dumper: human readability bug / support for comments (#2003).

3.0.0.beta1

2 years ago

This is just an intermediate unstable release towards 3.0.0. Please do not use it in production.

https://intelmq.readthedocs.io/