Changelog

Breaking Changes

• Due to the recent Twitter API changes, the Twitter operator is no longer supported (https://github.com/InQuest/ThreatIngestor/pull/157)

What's Changed

• BugSnag 🐛 by @azazelm3dj3d in https://github.com/InQuest/ThreatIngestor/pull/157
• Regex URL filtering for RSS and sitemap sources by @azazelm3dj3d in https://github.com/InQuest/ThreatIngestor/pull/158

Full Changelog: https://github.com/InQuest/ThreatIngestor/compare/v1.3.3...v1.4.0

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.4.0/

Changelog

Breaking Changes

• Due to the recent API paid transition from Twitter ("X"), we had to rebuild our Twitter source from the ground up to accommodate their new API schema. While almost verbatim to the old structure, we did have to make some modifications to the configuration. (https://github.com/InQuest/ThreatIngestor/pull/155)

Bug Fixes

• Sitemap ingestion was missing certain IOCs due to some HTML content being skipped, this is now fixed (https://github.com/InQuest/ThreatIngestor/commit/b661a082744c571aebe61317a60794f020c8a06f)

Features

• Improved config.yml validation script
  • It now includes a verbosity (-v) flag to debugging and cleaner output (https://github.com/InQuest/ThreatIngestor/commit/a646830fd047fb221774f19694b829289e418404)
  • Better check when validating operators (https://github.com/InQuest/ThreatIngestor/commit/c59af56f86e21808b9af363e760339fc8f4c7257)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.3.3/

Changelog: https://github.com/InQuest/ThreatIngestor/compare/v1.2.0...v1.3.3

Changelog

Bug Fixes

• A small patch was made to update how the sitemap source ingests artifacts. Certain blog URLs should no longer be skipped (https://github.com/InQuest/ThreatIngestor/commit/5dc79f6dc24b8bee67d3710dff8df5bca2e2a4c1)

Features

• New independent config.yml validation script for verifying the configuration is appropriately structured as both a YAML file and the minimum requirements for ThreatIngestor are met (https://github.com/InQuest/ThreatIngestor/issues/149)
  • Script: scripts/validate.py
• New source now allows for VirusTotal user comments ingestion (https://github.com/InQuest/ThreatIngestor/issues/87) (https://github.com/InQuest/ThreatIngestor/commit/f08946d96aed778a59261e61860cb7afe9f7fcaa, https://github.com/InQuest/ThreatIngestor/commit/de66d6e0d2d5d1b76ea679613b5533c458c4a173)
• Web source now runs an extra check against the modified header and saves the status code in the "saved_state" as an additional validation checkpoint before ingesting (https://github.com/InQuest/ThreatIngestor/issues/101) (https://github.com/InQuest/ThreatIngestor/commit/d91e6f1afdc0e1e58f848897aa770f0d58c16e97)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.2.0/

Changelog: https://github.com/InQuest/ThreatIngestor/compare/v1.1.0...v1.2.0

Changelog

Bug Fixes

• Merged the url_controller utility into the twitter source due to a broken import (https://github.com/InQuest/ThreatIngestor/issues/144)
• Restructured imports for the image and twitter sources. This should improve compatibility with Python 3.6 (https://github.com/InQuest/ThreatIngestor/commit/d3ecc5a8de3ed1f9eacae52485db317dbc02103e)
• No longer uses urllib module for the sitemap source. Now uses the requests module (https://github.com/InQuest/ThreatIngestor/commit/d3ecc5a8de3ed1f9eacae52485db317dbc02103e)
• RSS and sitemap sources now have better ingestion thanks to improvements made to the HTML content parsing (https://github.com/InQuest/ThreatIngestor/issues/140)

Features

• Automated image extraction from twitter sources (https://github.com/InQuest/ThreatIngestor/issues/132)
• New indicator of comprise type for ingested sources: email (https://github.com/InQuest/ThreatIngestor/issues/122)
• Updated codebase to match the newest version of iocextract (https://github.com/InQuest/ThreatIngestor/issues/143)
• Regex parsing is now supported for RSS sources (https://github.com/InQuest/ThreatIngestor/issues/142)

Hot Fix

• v1.1.1 - Fixed suffocating ingestion when working with RSS and sitemap feeds (https://github.com/InQuest/ThreatIngestor/commit/2b6446162ee04454c71bec9affc15463b8c90697)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.1.1/

Changelog: https://github.com/InQuest/ThreatIngestor/compare/v1.0.3...v1.1.0

Changelog

Bug Fixes

• Improved URL extraction for Twitter by utilizing the pyshorteners module. Now when the expansion attempt fails the first time, it'll attempt a different method for expanding the URL before returning the artifact (https://github.com/InQuest/ThreatIngestor/issues/128)

Features

• Now offers custom regex filtering for the sitemap ingestion source (https://github.com/InQuest/ThreatIngestor/issues/129)
• Modernized documentation (https://github.com/InQuest/ThreatIngestor/commit/f394da0b5c21bd172529b15effe7810993694cc3, https://github.com/InQuest/ThreatIngestor/commit/d2a8ab317e5483361c508ab49bcf21bffb2f56a1, https://github.com/InQuest/ThreatIngestor/commit/31dd2b3ee851684ecbb044ba6c2fcdeb7ea66271)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.0.3/

Changelog: https://github.com/InQuest/ThreatIngestor/compare/v1.0.2...v1.0.3

Changelog

Bug Fixes

• Updated Dockerfile to now include more pip packages and Google tesseract (https://github.com/InQuest/ThreatIngestor/pull/126/commits/126eb85cd028c7046df6d0610458fcbba5050ee6)
• Converted versioning to remove the 'beta' tag (https://github.com/InQuest/ThreatIngestor/pull/126/commits/126eb85cd028c7046df6d0610458fcbba5050ee6)

Features

• New sources: image, sitemap
  • image: Allows for image string extraction to parse out IOCs (https://github.com/InQuest/ThreatIngestor/pull/123/commits/1b066835415447fa128012f2d17df2665b5b1462)
  • sitemap: Parses sitemap XML data to locate blogs (https://github.com/InQuest/ThreatIngestor/pull/127/commits/079985e8b6814649ab31d9ac134a98d761f5ab84)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.0.2/

Changelog

Bug Fixes

• Bug fix for GitHub configuration. Now allows the user to select a specific number of days since the creation date (num_of_days in config.yml) when searching for a repository. (https://github.com/InQuest/ThreatIngestor/issues/113)
• Bug fix for retweeted bodies not being properly ingested. Now when collecting artifacts, the retweet body should be included. (https://github.com/InQuest/ThreatIngestor/issues/114)

Features

• Added GitHub workflow for running tests when a new commit is pushed.
• Now includes a Docker build for running in a containerized environment.
• New ingestion stream included. Users can now run a search against GitHub gists, searching by username. (https://github.com/InQuest/ThreatIngestor/issues/88)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.0.0b9/

• Added a whitelist feature for skipping common, non-malicious domains and hosts.

• Updated Twitter ingestion to support extended tweets.
• MISP upgrades.

Sixth beta release.

• Updated some ThreatKB plugin internals.