Malcolm Versions Save

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.

v23.09.0

8 months ago

Malcolm v23.09.0 is a release containing enhancements and bug fixes.

https://github.com/cisagov/Malcolm/compare/v23.08.1...v23.09.0

  • Features and enhancements
    • enable/disable Zeek's ICS parsers via environment variable (idaholab/Malcolm#256)
    • fully automated configuration and installation (idaholab/Malcolm#237) via command-line arguments
    • improvements to several dashboards
    • improvements to field normalization for BACnet and Modbus
    • improvements to the install.py and control.py scripts
  • Component version updates
  • Bug fixes
    • filtering in Arkime sessions view returned zero rows for some reason (idaholab/Malcolm#212)
    • Hedgehog - logrotate service not starting (idaholab/Malcolm#243)
    • Documentation issue (idaholab/Malcolm#245)
    • Error with configure-interfaces.py on both new server images (23.08.1) when setting ntp to 0.pool.ntp.org (idaholab/Malcolm#247)
    • installer script not loading prepackaged tarball correctly (idaholab/Malcolm#257)
    • logs inserted before template gets created cause field conflicts (idaholab/Malcolm#261)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.08.1

9 months ago

Malcolm v23.08.1 is a patch release fixing a regression in Hedgehog Linux which would cause disks to not be detected and used for artifact storage.

https://github.com/cisagov/Malcolm/compare/v23.08.0...v23.08.1

  • Bug fixes
    • sensor-capture-disk-config.py not detecting disks correctly (idaholab/malcolm#239)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.08.0

9 months ago

Malcolm v23.08.0 is a minor release with a few improvements, bug fixes and component updates.

EDIT: I've discovered a regression in the Hedgehog Linux startup script that formats drives to make them available for artifact capture. I'm investigating now. If this affects you, you might want to avoid this release until I put out a patch.

https://github.com/cisagov/Malcolm/compare/v23.07.1...v23.08.0

  • Features and enhancements

    • Rewrote the Network Traffic Artifact Upload interface and backend, replacing the defunct jQuery-File-Upload with FilePond. This was mainly due to jQuery-File-Upload no longer receiving security fixes and having some known vulnerabilities. see idaholab/Malcolm#235
    • Use netbox-initializers plugin, adding the ability to drop YAML files for various NetBox obects to be preloaded at startup. see idaholab/Malcolm#228
    • handle changes to ICSNPP parsers with source_ip/destination_ip fields (idaholab/Malcolm#233 and idaholab/Malcolm#226)

  • Bug fixes

    • Fixed extracting Malcolm version during ISO build
    • Workaround for wireshark no longer publishing raw manuf (OUI) list (idaholab/Malcolm#230)
    • Remove news feed from default NetBox dashboard (as it would try to reach out to the web for RSS updates)

  • Component version updates

    • Rebased Docker and ISO images to Debian 12 (bookworm)
    • live-build tool for building ISO images to debian/1%20230131
    • Arkime to v4.4.0
    • supercronic to v0.2.26
    • FileBeat to v8.9.0
    • LogStash to v8.9.0 (idaholab/Malcolm#234)
    • NetBox to v3.5.7
    • PostgreSQL (used by NetBox) to v15
    • opensearch-py to v2.3.0
    • PHP (as used by Upload interface) to v8.2
    • Fluent Bit to v2.1.8
    • certifi to v2023.7.22 (idaholab/Malcolm#229)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.07.1

10 months ago

Malcolm v23.07.1 is a patch release fixing a single bug.

https://github.com/cisagov/Malcolm/compare/v23.07.0...v23.07.1

  • Bug fixes
    • Fix issue parsing modbus.log (idaholab/Malcolm#225)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.07.0

10 months ago

EDIT - A bug in how Modbus traffic was parsed was discovered shortly after this release. A v23.07.1 release will be put out in the next day or so, you may want to wait for that.

Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

https://github.com/cisagov/Malcolm/compare/v23.05.1...v23.07.0

  • New features

    • scan docker images built via GitHub actions for vulnerabilities using Trivy (idaholab/Malcolm#218)
    • document building and deplolying Malcolm with an AWS AMI image (idaholab/Malcolm#205)
    • handle Arkime field actions (idaholab/Malcolm#200)
    • kubernetes: document how to get running on Amazon EKS (idaholab/Malcolm#194)
    • Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (idaholab/Malcolm#135)

  • Enhancements

    • use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    • Malcolm documentation edits (idaholab/Malcolm#204)
    • add option to enable SSH via password in hedgehog's configure-interfaces.py script (idaholab/Malcolm#158)
    • updated "Network Traffic Analysis with Malcolm" slides
    • use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    • improvements to identifying source of third-party logs sent via fluent bit
    • don't do unnecessary clone of Zeek plugins, just install using URL
    • parse bacnet_device_control.log produced by the icsnpp-bacnet parser for Zeek

  • Bug fixes

    • maxlogins value includes tmux sessions, can lock user out of SSH (idaholab/Malcolm#214)
    • curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (idaholab/Malcolm#209)
    • failure to parse some suricata alerts due to integer type which should be indexed as long (idaholab/Malcolm#206)
    • netbox-restore doesn't work in Kubernetes (idaholab/Malcolm#202)
    • PCAP File with no - in pcapng Fails to Upload (cisagov/Malcolm#265)
    • disable NetBox telemetry

  • Component version updates

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.05.1

1 year ago

Malcolm v23.05.1 is a minor release with a few component version updates and bug fixes, particularly to fix an issue with install.py where the ownership of .env files in the config directory may get incorrectly set to root rather than the unprivileged user.

https://github.com/cisagov/Malcolm/compare/v23.05.0...v23.05.1

  • Enhancements and bug fixes

    • install.py can create .env files 0:0 ownership instead of unprivileged user ownership (cisagov/Malcolm#253)
    • both zeek and zeek-live containers are trying to pull intel feeds on startup (idaholab/Malcolm#196)
    • Make sure a few Arkime fields (http.xff*) get created in the index template with the right field types to avoid aggregation query issues
    • Tweaks to convenience scripts (malcolmmonitor and sensormonitor) in ISO-installed Malcolm and Hedgehog Linux environments
    • Added some .service files for the ISO-installed version of Malcolm to be able to feed itself resource statistics via Fluent Bit
    • Documentation updates

  • Component version updates

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.05.0

1 year ago

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the docker-compose.yml file, it uses environment variables in .env files inside of the config directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using ./scripts/configure and ./scripts/auth_setup.

https://github.com/cisagov/Malcolm/compare/v23.04.0...v23.05.0

  • New features

    • integrate ICSNPP-Synchrophasor parser (idaholab/Malcolm#190)
    • End-to-end Malcolm and Hedgehog Linux ISO Installation document (idaholab/Malcolm#181)
    • support Malcolm deployment with Kubernetes (idaholab/Malcolm#149)
      • see Deploying Malcolm with Kubernetes
      • This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still some work to be done in this area. Please let us know what issues or suggestions you have via the issue tracker or via email to [email protected].
      • contributing issues:
        • inotify issue (idaholab/Malcolm#168)
        • htadmin/nginx and htpasswd (idaholab/Malcolm#169)
        • opensearch (idaholab/Malcolm#170)
        • uploading large PCAP files (idaholab/Malcolm#171)
        • script consolidation (idaholab/Malcolm#172)
        • documentation (idaholab/Malcolm#173)
        • user-defined persistent volumes (idaholab/Malcolm#174)
        • opensearch keystore (idaholab/Malcolm#176)
        • expose other TCP services (idaholab/Malcolm#183)
        • provide with filebeat access to nginx access and error logs (idaholab/Malcolm#186)
        • use Secrets for some environment variables instead of ConfigMaps (idaholab/Malcolm#189)

  • Enhancements and fixes

    • remove name-map-ui container (idaholab/Malcolm#165) in favor of using NetBox for asset identification
    • Python script refactoring, consolidation and cleanup
    • standardization of Docker container entrypoints
    • create ./scripts/configure alias for ./scripts/install.py --configure

  • Component version updates

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.04.0

1 year ago

Malcolm v23.04.0 is a release with enhancements, component version updates and bug fixes.

IMPORTANT NOTE: In March 2023 Docker Inc. announced its decision to sunset the "Docker Free Team" plan, which prompted us to decide to migrate away from Docker Hub to the Github Container Registry or "ghcr" (see idaholab/Malcolm#163). Due to public backlash, Docker Inc. reversed its decision. However, the Malcolm project will continue with the decision to use GHCR beginning with this release (Malcolm v23.04.0) and moving forward. If you're updating an existing instance of Malcolm, it's recommended that you back up your docker-compose.yml and docker-compose-standalone.yml files, replace them with the ones from this release and re-run ./scripts/install.py --configure to ensure that you're pointing at the latest images (this is actually always good practice when moving to a new release of Malcolm).

https://github.com/cisagov/Malcolm/compare/v23.03.0...v23.04.0

  • Enhancements

    • autostart install.py --configure on Malcolm ISO first boot (idaholab/Malcolm#157)
    • clarify information about auth_setup's use of external OpenSearch connections (idaholab/Malcolm#160)
    • migrate away from DockerHub container registry (idaholab/Malcolm#163)
    • give easier option for transferring SSL client files from Malcolm to forwarder (idaholab/Malcolm#177)
      • added tx-rx-secure.sh script as wrapper around croc automatically creating and using a local-only relay

  • Component version updates

  • Fixes

    • XFCE4's "save session on exit" causes conflict with Hedgehog kiosk mode if firefox instance is started upon session restore (idaholab/Malcolm#164)
    • docker-compose move from go-yaml/v3 breaks Malcolm's docker-compose YAML files (idaholab/Malcolm#178, docker/compose#10411)
    • increase index.mapping.nested_fields.limit in opensearch index template (idaholab/Malcolm#180)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.03.0

1 year ago

Malcolm v23.03.0 is a release with enhancements, component version updates and bug fixes.

https://github.com/cisagov/Malcolm/compare/v23.02.0...v23.03.0

  • Enhancements

    • Replace Zeek's misc/scan.zeek with ncsa/bro-simple-scan
    • terminate start and restart scripts once Malcolm has started properly (cisagov/Malcolm#240 and cisagov/Malcolm#241, thanks @Njinx)
    • minor usability improvements for ISO-installed Malcolm and Hedgehog (idaholab/Malcolm#155)
      • Added a "Configure Malcolm" menu item (under the "Internet" GTK menu with the other Malcolm stuff) and launcher on the top panel of icons in Malcolm. This runs ./scripts/install.py --configure in full screen. May look at starting this automatically on first boot in the future. (Malcolm)
      • Added Malcolm shortcut to gtk-3.0/bookmarks so it shows up in Thunar sidebar (Malcolm)
      • Added /opt/sensor/sensor_ctl shortcut to gtk-3.0/bookmarks so it shows up in Thunar sidebar (Hedgehog)
      • Have tilix from launcher panel start in /opt/sensor/sensor_ctl (Hedgehog)
    • minor tweaks to defaults for install.py --configure (enable offline-capable file scanners by default)
    • interrupt NetBox startup import script when netbox-restore is run
    • added NetBox restore logic to reset_and_auto_populate.sh script (used mostly for demos and presentations)

  • Component version updates

  • Fixes

    • last few seconds' Zeek logs prior to log rotation may be lost (idaholab/Malcolm#151)
    • in ISO-packaged Malcolm installation scripts directory, symlink netbox-backup and netbox-restore to control.py
    • improve opensearchpy connect/health check logig in pcap_watcher.py in pcap-monitor container

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

v23.02.0

1 year ago

Malcolm v23.02.0 is a feature release with new features and enhancements, component version updates and bug fixes.

https://github.com/cisagov/Malcolm/compare/v23.01.0...v23.02.0

  • New features

    • Compare and highlight discrepancies between NetBox inventory and observed network traffic (idaholab/Malcolm#133)
      • Added Zeek Known Summary and Asset Interaction Analysis dashboards which include visualizations about uninventoried devices and services
      • Added Uninventoried Internal Assets and Uninventoried Observed Services views to Arkime
      • Documentation updates related to NetBox
    • Added default device roles and service templates for initial NetBox population
    • Added netbox-backup/netbox-restore scripts to control.py for NetBox database and media
    • Added zeek_script_to_malcolm_boilerplate.py script for automating some of the tasks involved with adding new Zeek logs to Malcolm

  • Enhancements

    • configurable dark mode for OpenSearch Dashboards (idaholab/Malcolm#145)
    • added third-party OpenSearch Dashboards custom visualization component lguillaud/osd_transform_vis
    • modbus and modbus_detailed logs should be better normalized for event.action and event.result (idaholab/Malcolm#146)
    • Added -n argument to script/logs akin to tail -n (cisagov/Malcolm#234, thanks @Njinx)
    • Accounted for major additions to the OPCUA-Binary parser in both parsing and the corresponding dashboard
    • Set state:storeInSessionStorage to true for OpenSearch dashboards: this allows some complicated visualizations to be built with the Vega and Transform plugins, at the cost of having some URL bookmarks not contain every possible state the current dashboard has
    • Added related.device_name for normalization and pivoting
    • Removed related.segment in favor of ECS network.name
    • allow NetBox in Malcolm's "read-only" configuration

  • Component version updates

  • Fixes

    • failure to build logstash container due to illformed gem requirement (idaholab/Malcolm#144)
    • when running as UID/GID other than 1000, chown on dashboards and logstash containers takes a LONG time (idaholab/Malcolm#148)
    • Logs are being spammed with Suricata warnings pertaining to duplicate rules (cisagov/Malcolm#233)
    • Opensearch statistics are now parsed correctly when only a one node is present (cisagov/Malcolm#232, thanks @Njinx)
    • Explicitly check /usr/bin for docker-compose in case for some reason that's not in PATH (?) (cisagov/Malcolm#226)
    • Some refactoring of the Zeek pipeline in Logstash

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.