Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.03.1...v24.04.0
Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is recommended you re-run ./scripts/configure for this release.
Active rather than Stage, and uses tags instead to indicated that they were created through autopopulation.export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel to /opt/sensor/sensor_ctl/control_vars.conf and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.zeek-live containers (idaholab/Malcolm#456). See this comment for more details.docker-compose.yml files as it is now obsolete and caused a warning message that sometimes was not handled correctly.zeekctl deploy instead of zeekctl restart../config/)
ARKIME_QUERY_ALL_INDICES in arkime.env can be set to control the queryAllIndices setting in Arkime's config.ini.DASHBOARDS_PREFIX in dashboards-helper.env has been added for idaholab/Malcolm#455 (see above in Features and Enhancements).LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env has been changed to include zeek.dhcp, zeek.dns, and zeek.ntlm to support idaholab/Malcolm#415 (see above in Features and Enhancements).LOGSTASH_ZEEK_IGNORED_LOGS in logstash.env has been changed to remove capture_loss and stats so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.ZEEK_CRON has been removed from zeek-live.env and ZEEK_INTEL_REFRESH_CRON_EXPRESSION was removed from zeek.env and moved to the "offline" version of the container in zeek-offline.env for idaholab/Malcolm#456.EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE, EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, and EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS were added to zeek.env for idaholab/Malcolm#453. See a new section of documentation on Managing disk usage for more information about these and similar settings.Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.03.1 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.03.0...v24.03.1
Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is strongly recommended you re-run ./scripts/configure for this release.
NETBOX_PRELOAD_PREFIXES variable, which no longer has any effect) in which three giant buckets (one for each block in the RFC1918 private address definitions) could be created once at startup. (idaholab/Malcolm#436). So, for example:
10./16 (255.255.0.0)
10.9.0.215 would cause us to create and assign it to a 10.9.0.0/16 subnet192.168./24 (255.255.255.0)
192.168.100.123 would cause us to create and assign it to a 192.168.100.0/24 subnet172.16./20 (255.255.240.0)
172.16.29.10 would cause us to create and assign it to a 172.16.29.10/20 subnetSURICATA_EVE_THREADED - controls threaded file output (default false)SURICATA_EVE_ROTATE_INTERVAL - controls eve.json file rotation (default 1h)100 and otherBucket: true has been set for all of these table visualizations to ensure that the end user knows that Other rows may also exist outside of the rows shown. (idaholab/Malcolm#447)bro_logs were renamed to zeek_logs on Hedgehog Linuxnotice.log
install.py instead of just failingzeek-live container. This did not affect Suricata, Arkime capture, or any of the Hedgehog Linux processes. (idaholab/Malcolm#437)MALCOLM_OTHER_INDEX_PATTERN variable had been set to something other than the default../config/)
ARKIME_DEBUG_LEVEL=0 has been added to arkime.env to control the debug level for Arkime's config.ini.netbox-common.env (also, see below for some existing variables that were moved from logstash.env):
NETBOX_PRELOAD_PREFIXES has been removed and replaced with NETBOX_AUTO_CREATE_PREFIX for idaholab/Malcolm#436NETBOX_ENRICHMENT_LOOKUP_SERVICE=true has been added to indicate whether or not services (i.e., destination IP/port) should be looked up during NetBox enrichmentopensearch.env to give examples of how to set the time bucketing for OpenSearch/Elasticsearch indexesLOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES are now in filebeat.env, moved from upload-common.env
logstash.env to netbox-common.env and renamed:
LOGSTASH_NETBOX_ENRICHMENT is now NETBOX_ENRICHMENT
LOGSTASH_NETBOX_AUTO_POPULATE is now NETBOX_AUTO_POPULATE
LOGSTASH_NETBOX_CACHE_SIZE is now NETBOX_CACHE_SIZE
LOGSTASH_NETBOX_CACHE_TTL is now NETBOX_CACHE_TTL
Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.03.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.02.0...v24.03.0
./config/)
zeek.env
# Set to true to indicate that Zeek should output logs in JSON format
ZEEK_JSON=
# Whether or not to require SSL certificate verification when querying a TAXII or MISP feed
ZEEK_INTEL_FEED_SSL_CERTIFICATE_VERIFICATION=false
# Whether or not to disable the ICSNPP Profinet IO CM parser
ZEEK_DISABLE_ICS_PROFINET_IO_CM=
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v24.02.1 is identical to v24.02.0 except for a minor fix to the code that builds the Hedgehog Linux Raspberry Pi image.
The usual build artifacts are not included in this release, and new docker images for Malcolm have not been published.
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.01.0...v24.02.0
$) to instance (@) (see "avoiding concurrency issues")./config/)
arkime.env to allow configuration of Arkime's ILM/ISM settings (idaholab/Malcolm#300)# These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
# Whether or not Arkime should perform index management
INDEX_MANAGEMENT_ENABLED=false
# Time in hours/days before moving to warm and force merge (number followed by h or d)
INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
# Time in hours/days before deleting index (number followed by h or d)
INDEX_MANAGEMENT_RETENTION_TIME=90d
# Number of replicas for older sessions indices
INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
# Number of weeks of history to retain
INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
# Number of segments to optimize sessions for
INDEX_MANAGEMENT_SEGMENTS=1
# Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
dashboards.env to override the values automatically configured for pivot links (idaholab/Malcolm#335) and /dashboard/ redirect (idaholab/Malcolm#403) for Elasticsearch backend# These values are used to handle the Arkime value actions to pivot from Arkime
# to Dashboards. The nginx-proxy container's entrypoint will try to formulate
# them automatically, but they may be specified explicitly here.
NGINX_DASHBOARDS_PREFIX=
NGINX_DASHBOARDS_PROXY_PASS=
logstash.env for customizing which log types get NetBox enrichment (idaholab/Malcolm#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash# Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
# Zeek log types that will be ignored (dropped) by LogStash
LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
netbox-common.env for adjusting matching device manufacturers to OUIs in NetBox autopopulation# Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
# Whether or not enable capture statistics and include them in eve.json
SURICATA_STATS_ENABLED=false
SURICATA_STATS_EVE_ENABLED=false
SURICATA_STATS_INTERVAL=30
SURICATA_STATS_DECODER_EVENTS=false
# Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
ZEEK_DISABLE_STATS=true
# Whether or not to use libmagic to show MIME types for Zeek-extracted files served
EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v23.12.1...v24.01.0
./config/)
ARKIME_ROTATE_INDEX to arkime.env with default value of daily (see Arkime docs on rotateIndex)opensearch.env:# OpenSearch index patterns and timestamp fields
# Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field to use for network traffic logs in Logstash and Dashboards
MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
# Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
# Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
# Default time field to use for other logs in Logstash and Dashboards
MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
# Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
# Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field used by for sessions in Arkime viewer
ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
EXTRACTED_FILE_HTTP_SERVER_KEY to infected in zeek-secret.env
EXTRACTED_FILE_HTTP_SERVER_ZIP with default value of false in zeek.env, see (idaholab/Malcolm#288)Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v23.12.0 is a minor release with a few updates and bug fixes
https://github.com/cisagov/Malcolm/compare/v23.12.0...v23.12.1
install.py offer to pull the docker images (idaholab/Malcolm#310)config.ini with config.orig.ini if config.ini doesn't already exist (idaholab/Malcolm#311)capture to listen on the interface directly rather than post-processing PCAPs (idaholab/Malcolm#281)SURICATA_DISABLE_ICS_ALL environment variable to disable OT/ICS analysis in SuricataZEEK_INTEL_REFRESH_THREADS to allow setting the number of threads for intel feed pullshedgehog vs. malcolm profiles) and generally improved documentation of live capture options/mapi/opensearch/, /mapi/logstash/ and /mapi/netbox/ from the Malcolm API endpoint to their respective component APIsMalcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v23.12.0 is a feature release with many improvements, updates and fixes
https://github.com/cisagov/Malcolm/compare/v23.10.0...v23.12.0
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v23.10.0 is a feature release.
https://github.com/cisagov/Malcolm/compare/v23.09.0...v23.10.0
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.
Malcolm v23.09.0 is a release containing enhancements and bug fixes.
https://github.com/cisagov/Malcolm/compare/v23.08.1...v23.09.0
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.