Releases · idaholab/Malcolm

27 Mar 19:46

mmguero

v24.03.1

99f68ec

Malcolm v24.03.1

Malcolm v24.03.1 contains new features, improvements, bug fixes and component version updates.

v24.03.0...v24.03.1

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is strongly recommended you re-run ./scripts/configure for this release.

Features and enhancements
- Malcolm instances created using the installer ISO will now detect and format any large (>100GB) storage devices and automatically set them up for use for storing the OpenSearch data store, PCAP files, and/or log storage, similar to what Hedgehog Linux does. (#266)
- Since v24.01.0, Malcolm has allowed users to specify custom index patterns for Zeek and Suricata logs (see issue 313). This release now also provides the capability for Arkime to know about those indexes so that those documents also appear in Arkime search results. (#313, arkime/arkime#2705) As this is not released in Arkime yet, Malcolm is using a local patch with these changes, to be released upstream in Arkime v5.0.2.
- A new setting for Logstash has been added to allow autocreation and assignment of NetBox subnets during enrichment. If "Should Malcolm automatically create missing NetBox subnet prefixes based on observed network traffic?" is answered to the affirmative during configuration, observed traffic that does not fall into any existing NetBox prefix will cause one to automatically be created, creating them one level down (e.g., 8 additional masked bits) from the RFC1918 address space definitions. This replaces an earlier feature (controlled by the NETBOX_PRELOAD_PREFIXES variable, which no longer has any effect) in which three giant buckets (one for each block in the RFC1918 private address definitions) could be created once at startup. (#436). So, for example:
  - 10./16 (255.255.0.0)
    - the IP address 10.9.0.215 would cause us to create and assign it to a 10.9.0.0/16 subnet
  - 192.168./24 (255.255.255.0)
    - the IP address 192.168.100.123 would cause us to create and assign it to a 192.168.100.0/24 subnet
  - 172.16./20 (255.255.240.0)
    - the IP address 172.16.29.10 would cause us to create and assign it to a 172.16.29.10/20 subnet
- New configuration settings have been added to specify creation and rotation of Suricata's EVE JSON log files, including controls for threaded file output and file rotation. See this comment for a full description of the changes (#445). Most noteworthy are:
  - SURICATA_EVE_THREADED - controls threaded file output (default false)
  - SURICATA_EVE_ROTATE_INTERVAL - controls eve.json file rotation (default 1h)
- Table visualizations in Malcolm's prebuilt OpenSearch Dashboards were not consistent in the number of rows returned. This has been standardized to 100 and otherBucket: true has been set for all of these table visualizations to ensure that the end user knows that Other rows may also exist outside of the rows shown. (#447)
- Some some field mappings were moved from malcolm_template.json to the composable template malcolm_common.json
- Documentation improvements
- Minor update to slides
- Some directories named like bro_logs were renamed to zeek_logs on Hedgehog Linux
- The Community ID field is now being added to Zeek's notice.log
- Attempt to install necessary Python 3 packages at the beginning of install.py instead of just failing
Component version updates
- Zeek to v6.2.0
- opensearch-py to v2.5.0
- Fluent Bit to v3.0.0
- Moved from the no-longer-maintained Salesforce repo for HASSH to Corelight's
Bug fixes
- AF_PACKET was not being utilized for capturing traffic on Malcolm in the zeek-live container. This did not affect Suricata, Arkime capture, or any of the Hedgehog Linux processes. (#437)
- The Packet Capture Statistics dashboard was not correctly computing seen and dropped packets for Suricata. (#442)
- A STDERR warning from the new Docker Compose v2.25 was messing up the creation of the OpenSearch keystore file. (#452)
- Fixed an issue in which the Dashboards for non-network data (e.g., temperatures, resource usage, etc.) would not see the correct data if the MALCOLM_OTHER_INDEX_PATTERN variable had been set to something other than the default.
- Ensure that index names created for use by Logstash sending to OpenSearch/Elasticsearch are lowercase
- Major cleanup and refactoring of the NetBox enrichment code used by Logstash
Configuration changes (in environment variables in ./config/)
- ARKIME_DEBUG_LEVEL=0 has been added to arkime.env to control the debug level for Arkime's config.ini.
- Additions/deletions in netbox-common.env (also, see below for some existing variables that were moved from logstash.env):
  - NETBOX_PRELOAD_PREFIXES has been removed and replaced with NETBOX_AUTO_CREATE_PREFIX for #436
  - NETBOX_ENRICHMENT_LOOKUP_SERVICE=true has been added to indicate whether or not services (i.e., destination IP/port) should be looked up during NetBox enrichment
- Comments were added to opensearch.env to give examples of how to set the time bucketing for OpenSearch/Elasticsearch indexes
- In addition to the new variables mentioned above, some cleanup and organization was done in the environment variable files used for configuring Malcolm:
  - LOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES are now in filebeat.env, moved from upload-common.env
  - Some NetBox related variables have been moved from logstash.env to netbox-common.env and renamed:
    - LOGSTASH_NETBOX_ENRICHMENT is now NETBOX_ENRICHMENT
    - LOGSTASH_NETBOX_AUTO_POPULATE is now NETBOX_AUTO_POPULATE
    - LOGSTASH_NETBOX_CACHE_SIZE is now NETBOX_CACHE_SIZE
    - LOGSTASH_NETBOX_CACHE_TTL is now NETBOX_CACHE_TTL

Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Assets 18

05 Mar 21:42

mmguero

v24.03.0

6c39176

Malcolm v24.03.0

Malcolm v24.03.0 contains new features, improvements, bug fixes and component version updates.

v24.02.0...v24.03.0

Features and enhancements
- support json-delimited import for Zeek logs (#65)
- go through list of Trivy security findings (#236)
- support /attributes and /events enpoints from MISP feed for Zeek intel generation (#336)
- KEV detections for Unitronics VisiLogic CVE-2023-6448 (#394)
- create dashboards for other non-network log data (#414)
- links on landing page should open in a new tab (#427)
- incorporate ICSNPP Profinet IO CM parser (#429)
Component version updates
- Arkime to v5.0.1
- OpenSearch and OpenSearch Dashboards to v2.12.0
Bug fixes
- fix the way we do environment variables in local.zeek (#413)
- a few issues with the install.py script when installing from GitHub releases (#416)
- htadmin creating entries without a newline between them in the htpasswd file (#426)
- hard-coded date value in Kibana pivot links (#428)
- unencrypted, unzipped extracted file download not working (#431)

Configuration changes (in environment variables in ./config/)

these variables in zeek.env

# Set to true to indicate that Zeek should output logs in JSON format
ZEEK_JSON=
# Whether or not to require SSL certificate verification when querying a TAXII or MISP feed
ZEEK_INTEL_FEED_SSL_CERTIFICATE_VERIFICATION=false
# Whether or not to disable the ICSNPP Profinet IO CM parser
ZEEK_DISABLE_ICS_PROFINET_IO_CM=

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Assets 8

20 Feb 03:46

mmguero

v24.02.1

b95dd3f

Malcolm v24.02.1

Malcolm v24.02.1 is identical to v24.02.0 except for a minor fix to the code that builds the Hedgehog Linux Raspberry Pi image.

The usual build artifacts are not included in this release, and new docker images for Malcolm have not been published.

Assets 2

15 Feb 20:23

mmguero

v24.02.0

a885008

Malcolm v24.02.0

Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

Features and enhancements
- Hedgehog Linux SD card image for Raspberry Pi (#250; special thanks to @aut0exec for his work on this)
- allow configuration of Arkime's ILM/ISM settings (#300)
- add option for customizing which log types get NetBox enrichment (#316)
- improve the extracted_files download page (#329)
- include missing aggregations in API bucket queries (#386)
- more intelligent .env file checking on startup (#387)
- Malcolm report to itself on capture statistics (#395)
- link to Dashboards/Arkime from NetBox devices view (#410)
- changed default PCAP storage format to zstd(3) for new installations
- various documentation updates and improvements
- changed back to using official Zeek .deb files rather than building from source to reduce build times
Component version updates
- Arkime to v5.0.0
- Capa to v7.0.1
- YARA to v4.5.0
- Beats to v8.12.1
- Logstash to v8.12.1
- Zeek to v6.1.1
Bug fixes
- pivot links from Arkime to Kibana in external elasticsearch are not working (#335)
- redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403)
- allow netbox-restore and netbox-backup to specify container name (#337)
- fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and updated documentation)
- source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401)
- event.severity_tags is not being assigned correctly based on rule.category (#402)
- basic authentication breaks with special characters (#404)
- changed some Logstash Ruby variables from global ($) to instance (@) (see "avoiding concurrency issues")

Configuration changes (in environment variables in ./config/)

these variables in arkime.env to allow configuration of Arkime's ILM/ISM settings (#300)

# These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
# Whether or not Arkime should perform index management
INDEX_MANAGEMENT_ENABLED=false
# Time in hours/days before moving to warm and force merge (number followed by h or d)
INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
# Time in hours/days before deleting index (number followed by h or d)
INDEX_MANAGEMENT_RETENTION_TIME=90d
# Number of replicas for older sessions indices
INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
# Number of weeks of history to retain
INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
# Number of segments to optimize sessions for
INDEX_MANAGEMENT_SEGMENTS=1
# Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
INDEX_MANAGEMENT_HOT_WARM_ENABLED=false

these variables in dashboards.env to override the values automatically configured for pivot links (#335) and /dashboard/ redirect (#403) for Elasticsearch backend

# These values are used to handle the Arkime value actions to pivot from Arkime
#   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
#   them automatically, but they may be specified explicitly here.
NGINX_DASHBOARDS_PREFIX=
NGINX_DASHBOARDS_PROXY_PASS=

these variables in logstash.env for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash

# Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird

# Zeek log types that will be ignored (dropped) by LogStash
LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout

these variables in netbox-common.env for adjusting matching device manufacturers to OUIs in NetBox autopopulation

# Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95

these variables in suricata-live.env and zeek-live.env that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395)

# Whether or not enable capture statistics and include them in eve.json
SURICATA_STATS_ENABLED=false
SURICATA_STATS_EVE_ENABLED=false
SURICATA_STATS_INTERVAL=30
SURICATA_STATS_DECODER_EVENTS=false

# Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
ZEEK_DISABLE_STATS=true

this variable in zeek.env related to the improvements to the extracted_files download page (#329)

# Whether or not to use libmagic to show MIME types for Zeek-extracted files served
EXTRACTED_FILE_HTTP_SERVER_MAGIC=false

Contributors

aut0exec

Assets 8

17 Jan 17:47

mmguero

v24.01.0

c78a043

Malcolm v24.01.0

Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.01.0

Features and enhancements
- new Malcolm instance landing page (#252)
- file carve download with password-protected .zip file (#288)
- new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (#290)
- allow customizing indexes for logs written to OpenSearch/Elasticsearch (#313)
- more consistently differentiate between uploaded and live-captured traffic (#321)
- make download extracted file context item from Arkime smarter (#330)
- improve netbox device type library import by using "official" import script (#384)
Component version updates
- Alpine Linux to v3.19 as the base for some Docker images
- Fluent Bit to v2.2.2
- Beats to v8.11.4
- LogStash to v8.11.4
Bug fixes
- Suricata Alerts dashboard "Alerts - Tags" visualization is useless (#314)
- third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (#318)
- update document lookup APIs to search either network or host data (#322)
- suricata rule update is broken (#323)
- time sync from hedgehog to Malcolm opensearch instance not working (#324)
- fix issue specifying database mode via command-line
- have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)

Configuration changes (in environment variables in ./config/)

added the following variables with relation to #313

added ARKIME_ROTATE_INDEX to arkime.env with default value of daily (see Arkime docs on rotateIndex)
added the following variables and defaults to opensearch.env:

# OpenSearch index patterns and timestamp fields
# Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field to use for network traffic logs in Logstash and Dashboards
MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
# Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
# Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
# Default time field to use for other logs in Logstash and Dashboards
MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
# Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
# Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field used by for sessions in Arkime viewer
ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket

changed default for EXTRACTED_FILE_HTTP_SERVER_KEY to infected in zeek-secret.env
added EXTRACTED_FILE_HTTP_SERVER_ZIP with default value of false in zeek.env, see (#288)

Assets 8

20 Dec 19:07

mmguero

v23.12.1

cef8f9d

Malcolm v23.12.1

Malcolm v23.12.0 is a minor release with a few updates and bug fixes

v23.12.0...v23.12.1

Features and enhancements
- have install.py offer to pull the docker images (#310)
- only overwrite Arkime's config.ini with config.orig.ini if config.ini doesn't already exist (#311)
- create Suricata rules for Zyxel vulnerabilities from KEV (#312)
- provide alternate configuration for Arkime capture to listen on the interface directly rather than post-processing PCAPs (#281)
- added SURICATA_DISABLE_ICS_ALL environment variable to disable OT/ICS analysis in Suricata
- added ZEEK_INTEL_REFRESH_THREADS to allow setting the number of threads for intel feed pulls
- documented the different run profiles (hedgehog vs. malcolm profiles) and generally improved documentation of live capture options
- route /mapi/opensearch/, /mapi/logstash/ and /mapi/netbox/ from the Malcolm API endpoint to their respective component APIs
- minor improvements to how the user supplies custom rules/config for Suricata, Zeek, and Arkime
Component version updates
- updated GitHub workflow actions for automated builds
- supercronic to v0.2.29
- LogStash to v8.11.3
- Beats to v8.11.3
- NetBox to v3.6.7
- elasticsearch-py to v8.11.1
Bug fixes
- review and fix capabilities granted to containers (#282)
- change URL for downloading manuf list to new wireshark.org URL / wireshark no longer publishes raw manuf (OUI) list (#230 and #306)
- directory hierarchies not being created as Kubernetes configmap correctly (#308)
- rsyslog no longer in Debian bookworm (#309)
- removed unused Arkime log and raw directories

Assets 8

05 Dec 04:29

mmguero

v23.12.0

5848a05

Malcolm v23.12.0

Malcolm v23.12.0 is a feature release with many improvements, updates and fixes

v23.10.0...v23.12.0

Features and enhancements
- replace kbn_sankey_vis with vega or transform (#147)
- address issues with NetBox database and Logstash's NetBox cache (#259)
- integrate nsacyber/ELITEWOLF signatures into default rule set CISA (#275)
- improve error messages for PCAP/artifact processing beyond just icons (#276)
- option to auto-create "catch-all" NetBox IPAM prefixes for private IP space (#279)
- use prefix.description instead of VRF for identifying subnets in NetBox (#280)
- allow customizing Arkime's freeSpaceG setting (for PCAP deletion) in an environment variable (#285)
- replace master/slave with client/server when parsing modbus logs (#291)
- put netbox restore database functionality inside container (#294)
- provide way to customize zeek Site::local_nets (#295)
- allow configuration of docker's logging driver to prevent disk-exhaustion (#301)
- allow user to include other suricata config YML files (#302)
- allow user to be able to provide custom zeek config (#303)
- allow tuning Suricata's max-pending-packets via environment variable (#304)
- enable OpenSearch dashboards condensed header
Component version updates
- Alpine Linux to v3.18.5
- elasticsearch-py to v8.11.0
- Logstash to v8.11.1
- NetBox v3.6.6
- OpenSearch and OpenSearch Dashboards (#298)
  - v2.9.0
  - v2.10.0
  - v2.11.0
  - v2.11.1
- opensearch-py to v2.4.2
- supercronic to v0.2.28
- Zeek to v6.1.0
  - Update logstash parsers for Zeek v6.1.0 (#289)
Bug fixes
- Malcolm Sensor Temperature dashboard issue (#265)
- strip out broken Arkime and NetBox links from dashboards for Kibana import (#286)
- have netbox-restore script restart necessary services or set necessary permissions (#287)
- file type validation not working for upload from (some?) windows browsers (#292)
- go through list of Qualys image scan results (#299)

Assets 8

26 Oct 02:27

mmguero

v23.10.0

6eb2a19

Malcolm v23.10.0

Malcolm v23.10.0 is a feature release.

v23.09.0...v23.10.0

Features and enhancements
- support both OpenSearch and Elasticsearch output (#258)
- "capture-only" Malcolm configuration (AKA "dockerized Hedgehog") (#254)
- don't run kiosk mode on Hedgehog first boot (#263)
- let Arkime check its own database to see if it needs to be upgraded
- allow specifying Arkime password hash secret for Viewer clusters
- documentation improvements
- minor updates to slide decks
- allow specifying ports for EtherNet/IP parser via environment variable
Component version updates
- Arkime to v4.6.0
- supercronic to v0.2.27
- beats to v8.10.4
- Logstash to v8.10.4
- NetBox to v3.6.4
- Fluent Bit to v2.1.10
Bug fixes
- set "autorestart" to true for all started services (#267)
- changed toolchain for building Zeek and Zeek plugins to clang/libc++ to address some build issues with Spicy plugins using GCC
- ensure Arkime is started before creating OpenSearch artifacts
- ensure Arkime and OpenSearch artifacts are populated before starting LogStash
- don't log "0.0" temperatures from Fluent Bit thermal forwarders

Assets 8

15 Sep 21:36

mmguero

v23.09.0

d9204a0

Malcolm v23.09.0

Malcolm v23.09.0 is a release containing enhancements and bug fixes.

v23.08.1...v23.09.0

Features and enhancements
- enable/disable Zeek's ICS parsers via environment variable (#256)
- fully automated configuration and installation (#237) via command-line arguments
- improvements to several dashboards
- improvements to field normalization for BACnet and Modbus
- improvements to the install.py and control.py scripts
Component version updates
- ICSNPP parsers to latest releases
- Arkime to v4.5.0
- Beats to v8.10.0
- capa to v6.1.0
- Fluent Bit to v2.1.9
- Logstash to v8.10.0
- NetBox to v3.6.1 (see also v3.6.0 release notes)
- opensearch-py to v2.3.1
- Zeek to v6.0.1 (see also v6.0.0 release notes)
Bug fixes
- filtering in Arkime sessions view returned zero rows for some reason (#212)
- Hedgehog - logrotate service not starting (#243)
- Documentation issue (#245)
- Error with configure-interfaces.py on both new server images (23.08.1) when setting ntp to 0.pool.ntp.org (#247)
- installer script not loading prepackaged tarball correctly (#257)
- logs inserted before template gets created cause field conflicts (#261)

Assets 8

16 Aug 17:48

mmguero

v23.08.1

e20cc45

Malcolm v23.08.1

Malcolm v23.08.1 is a patch release fixing a regression in Hedgehog Linux which would cause disks to not be detected and used for artifact storage.

v23.08.0...v23.08.1

Bug fixes
- sensor-capture-disk-config.py not detecting disks correctly (#239)

Assets 8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Contributors

Releases: idaholab/Malcolm

Malcolm v24.03.1

Malcolm v24.03.0

Malcolm v24.02.1

Malcolm v24.02.0

Contributors

Malcolm v24.01.0

Malcolm v23.12.1

Malcolm v23.12.0

Malcolm v23.10.0

Malcolm v23.09.0

Malcolm v23.08.1