Credential Digger Versions Save

Restructure the project and repackage the tool in a more modern way. Introduce src layout and a pyproject.toml while deprecating setup.py. Version tracking is now delegated to a dynamic file and old package data (old folders for BoW models deprecated in v4.4) have been deleted. A new badge has been introduced to show what python versions are currently supported by Credential Digger (this point is often source of confusion as we have to rely on external libraries not necessarily supporting the latest available python versions). This badge will be updated only after the release of pypi package (it relies on metadata published there).

Minor version has been increased to better distinguish from the old package structure, even if it could also not be strongly needed in such a case.

• Update hyperscan dependency. New versions of python-hyperscan don't require installing the OS package (i.e., libhyperscan for linux or hyperscan for macOS) because it's statically linked inside the python package
• Support python versions 3.8 to 3.11
• update Dockerfile to use python3.10 instead of 3.9

Minor:

• update dependencies
• update README

New features

• Add get_discoveries_with_rules method in both client and CLI
• Add endpoint for scan_file method in server/UI

Minor

• update dependencies

From v4.10 (not released before)

• enhance get_discoveries to also return the matching rule of a discovery. A with_rule optional parameter has been added (default to False)

New features:

• Add the possibility of scanning pull requests with scan_pull_request method. All the new lines introduced (i.e., either added or modified) will be scanned
• Integrate the new features in CLI with scan_pr command
• Write tests and add documentation for the new features

New features

• A first version of pre-commit hook is released. The hook can be installed (also via pre-commit framework) or run as CLI.

Fixes

• SqliteClient supports paths containing ~
• credentialdigger can now be run as a command, without the python -m preamble

Main features:

• Use hyperscan version 0.2.0 (i.e., based on libhyperscan5) for python3.8 (was hyperscan 0.1.5 before, based on libhyperscan4).
• Add support for python 3.10
• Abandon python 3.6, whose eol was in 2021
• Bump requirements and make Credential Digger ready for ubuntu 22.04
• Update tests
• Fix bugs

Bugfixes details:

• #183
• #200
• #214
• #224 (bug in update discoveries functions)

git_username authentication

We add an optional parameter git_username that can be set to authenticate in order to perform a scan. While this parameter is not mandatory for GitHub (neither .com nor enterprise), it is needed for some private git servers and for private bitbucket repos.

If the git_token is not set, this parameter is ignored (since the tool cannot authenticate with a username without token). On the contrary, if the git_token is set, the username used for authenticating the tool is either git_username (if set) or oauth2 (the default value, which is the one adopted by github)

The git_username parameter is supported not only in the python library but also in the CLI and in the UI (with a new optional input field).

Minor

• Fix dependencies for tensorflow
• Fix the double "category" input field in the UI (add_rule)
• Make the export_discoveries more efficient by not loading all the discoveries of a repo (unless needed)

With this release we restructure the ML models in order to improve their precision. Moreover, the new models will be directly integrated in the project, overcoming the painful download&linking needed for the former ones.

All the changes are transparent to the final user (i.e., no API or function definition changed), thus there was no need for a major upgrade to v5.

Path Model

We decided to deprecate the fasttext approach and shifted to the usage of a regex to filter out false positive file paths. Indeed, according to our tests, we noticed that we can keep a good precision while decreasing the overhead

~SnippetModel~ PasswordModel

We decided to deprecate the old fasttext double-model (extractor+classifier) approach in order to shift to a NLP approach based on CodeBERT. Overall, it's slower but way more precise, even if it only works for password. Hence, the change of name from SnippetModel to PasswordModel. Moreover, since the PasswordModel only works for passwords, we added a check in the Client to only run this model over password discoveries.

AoB

• The download function has been deprecated and models are managed automatically by Credential Digger
• The generator was strongly linked to the SnippetModel, so it has been deprecated
• The documentation has been updated, both in the README and in the wiki
• We added a categories enum in the postgres db in order to drive the users to 4 main rule categories. Nevertheless, this enum is only enforced in new postgres installations to make the transition smoother
• The UI has been updated to use the new models
• We ported the incremental scan_snapshot from v4.3.1
• Minor bug fixes
• Refresh the UI every 8s (was 5s)

Credits also go to the wonderful work from @melisande1

Docker

• Upgrade the base image of the docker containers to adopt python3.9 (was 3.7) (Docker upgrade #141)
• Name the images for the backend with sqlite and postgres accordingly
• Update instructions on the wiki

AoB

• A bug preventing the use of the similarity model with postgres has been fixed
• Some Dependencies versions have been updated

New features:

• export discoveries in a csv file
• Add tests for get_discoveries

Minor improvements and fixes:

• reduce tensorflow warnings
• improve dependency management
• Fix docker container for Windows
• Minor UI fixes