Highlights

• Added debug-iter-topology in makefile to debug running topology
• Added more stable topologies with iter-files-mails, that uses generator to send mails to topology. It's memory safe and stable
• Fixed many bug to avoid topology restarts
• Added three new topologies that use new spout

Highlights

• Resolved many bugs
• Getting only main headers fields from mails. This number can grow very fast.
• Added a field headers with all headers.
• Upgraded mail-parser and streamparse.
• Disabled Zemana integration from code (deprecated).

Highlights

• Added module that analyze SMTP dialects. Added dialect section in main configuration file
• Upgraded mail-parser and Apache Tika modules
• Resolved many bugs

Highlights

• Resolved many bug
• Performance improvement
• Added persistent store for RAM runtime filter. Now SpamScope saves the filter stores, so it can reload lists after a reboot
• Upgraded mail-parser library, now SpamScope stores all mail headers
• Added more logs
• Added filter for size for attachments

Highlights

• New documentation, with more details about configuration and installation.
• Added Ansible playbooks to install Apache Storm and SpamScope.
• Added priority for post processing.
• Added timeout for long time analysis function.

Highlights

• Fixed Travis CI configuration.
• New store_samples attachments post processing, to store samples on filesystem.
• Splitted Elasticsearch template for attachments and mails.
• Many fix to Elasticsearch template to upgrade at 6.x.
• Added whitelist to VirusTotal post processing.
• Storing analyzed mails in daily folders.
• Fixed many bugs.

:warning: Breaking Changes :warning:

• Changed main configuration file. All lists are under list key, so they are automatically loaded:

# Attachments bolt configuration
attachments:
    # The lists of all components must be under lists keyword to load them
    # automatically
    commons:
        lists:
            blacklist_content_types:
                # All content types to remove from results
                # Example in content_types/blacklist/generic.example.yml
                generic: /path/to/generic_content_types
                custom: /path/to/custom_content_types

            not_extract_content_types:
                # All content types that you don't want extract from archive
                # Example: application/java-archive (jar), you can save the jar
                # but do not extract the class inside.
                generic: /path/to/generic_content_types
                custom: /path/to/custom_content_types

Highlights

• New Travis CI configuration. Added automatic deploy.
• Fixed Elasticsearch template (index.mapping.ignore_malformed).
• Added Dockerfile example: debug images.
• Upgraded mail-parser and streamparse.
• New field to_domains in main report: domains of to header.
• Bugfix

Highlights

• More stable version. Solved three bug: heartbeat timeout when using Thug, getting custom header now not raise pystorm/serializers/json_serializer.py and you can avoid to set blacklist or whitelist in configuration.
• Removed waiting.sleep in configuration spout file to avoid Apache Storm timeout.
• Thug section has more important options.
• Added debug environment in config.json.
• Added docker-compose example for debug use.
• Splitted requirements in optional and not.
• Fixed Thug analysis integration: replaced os.kill with raise. Added more options.

Highlights

• Upgraded requirements
• Upgraded mail-parser
• Upgraded Apache Tika support to 1.16 version
• New installer that manages all Python dependencies. SpamScope is on PyPI
• Command line spamscope-elasticsearch can be used in the cases where Elasticsearch is behind a reverse proxy. You can use RFC-1738 formatted URLs.

:warning: Breaking Changes :warning:

• The mail format is different. See mail-parser for more details.

Highlights

• Support Outlook mail (msg format). Enable flag outlook: True in mailboxes main configuration for folders that have this mail format.
• Added SpamAssassin post processing. Every mails can be analyzed from SpamAssassin. Be careful the parsing can be very slow.
• Now you can extract custom headers from mail with headers: list in mailboxes main configuration (see configuration example in this repository).
• Merged urls bolts of body and attachments in one bolt.
• Fixed Elasticsearch template.
• Upgraded mail-parser and streamparse.
• Added headers receiveds in main output.
• The form in mails now are analyzed from phishing bolt.
• New code to analyze raw mail with third part tools.
• The binary attachments have a flag binary.
• SpamScope can extract samples from mailformed zip attachments.
• SpamScope can extract samples from quoted-printable headers mail.