Monitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.
Monitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.
CertStreamMonitor architecture relies on 3 scripts :
certstreammonitor.py
scanhost.py
gethost.py
gethost.py
script that provides a way for security operators to request the last hostnames which have been detected by certstreammointor.py
since X seconds.check_rules.py
check_rules.py
script that provides a way for security operators to check their own rules, especially SearchKeywords, DetectionThreshold and BlacklistKeywords.wss://certstream.calidog.io
CT logs aggregator server with certstream-python (see certstream-python), but you can choose, and operate, your own server (see certstream-server).Subject Alt Names
field of certificates/alerts
directory (default value) to push forward investigation.Stop checking on 2019-09-27
is included into the StillInvestig columnInstall the requirements
$ pip3 install -r requirements.txt
You can find a configuration file example in 'conf' directory. Configurable parameters are:
SearchKeywords
: Keywords to look for (with '|' (or) as separator)DetectionThreshold
: set the minimum number of detected SearchKeywords in a hostname before writing it to DB. Under this value but above zero, detected hostnames are only written to logfile. Default value: 2.DBFile
: SQLite3 database file (the path and file will be created if don't exist)TABLEname
: The name of the database tableLogFile
: The logging file (the path and file will be created if don't exist)UAfile
: you can provide a User-Agent file to masquerade this value of requests (random change for each request)Alerts_dir
: you can specify where JSON alert files are written
You can use the following strings to add time/date hashed based subdirectories:
%%m -> month, %%d -> day, %%Y -> year, %%H -> hour, %%M -> minute.
Example: Alerts_dir = ./alerts/%%Y/%%m/%%dOptional:
Proxy
: allows to give a SOCKS or HTTP proxy to process your scanhost.py's requests (as Tor)Proxy_*
parameters : allow you to specify HTTP proxy informations (server, port[, user, password]) for CertStreamMonitor.py script to connect to the CT logs aggregator server.ACTServer
: you can specify the CT logs aggregator server of your choice. By default, it is the server run by Calidog Security.Safe_Browsing_API_Key
: indicate (if you want) your Google Safe Browsing API key in order to check hostnames that are UP against Google Safe Browsing Lookup API (How-To get an API key for the Safe Browsing Lookup API).Notification_Destination
: specify a notification destination as attended by apprise package. Documentation about the format of this parameter is available on the apprise Github page.BlacklistKeywords
: Keywords to ignore matched hosts (with '|' (or) as separator)$ python3 ./CertStreamMonitor.py -c conf/example.conf
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
Connection established to CertStream! Listening for events...
[2018-03-12T11:40:15] cpanel.my-appleid-apple.net (SAN: mail.my-appleid-apple.net,my-appleid-apple.net,webdisk.my-appleid-apple.net,webmail.my-appleid-apple.net,www.my-appleid-apple.net) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 45:11:51:2D:24:D3:04:6E:DF:49:46:6D:64:56:67:4B:0A:48:8D:93) (StartTime: 2018-03-12T10:39:40)
[2018-03-12T11:41:19] cpanel.verification-account-apple-now.com (SAN: mail.verification-account-apple-now.com,verification-account-apple-now.com,webdisk.verification-account-apple-now.com,webmail.verification-account-apple-now.com,www.verification-account-apple-now.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 2D:90:F9:F7:83:F6:48:26:EF:C9:72:50:4B:06:FA:36:53:94:3C:8B) (StartTime: 2018-03-12T10:40:49)
[2018-03-12T11:41:36] login-apple.sytes.net (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: C7:78:2F:08:1E:CC:83:6C:06:EF:77:14:D2:1A:4E:06:A8:B3:F9:77) (StartTime: 2018-03-12T10:41:08)
[2018-03-12T11:42:26] cpanel.restore-account-apple.com (SAN: mail.restore-account-apple.com,restore-account-apple.com,webdisk.restore-account-apple.com,webmail.restore-account-apple.com,www.restore-account-apple.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: F3:CA:B1:C6:DE:4F:05:16:FD:06:F3:FF:29:8A:D3:1F:10:9D:50:1A) (StartTime: 2018-03-12T10:41:59)
[2018-03-12T11:49:37] securelogin.here.att.thysseankrupp.com (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 8F:9B:98:8D:5D:9B:03:0B:4F:62:56:40:C1:DE:9A:A4:FB:2D:A3:3E) (StartTime: 2018-03-12T09:22:41)
...
$ python3 scanhost.py --help
-h --help Print this help
-c --config Configuration file to use
-f --fqdn-dirs Store JSON files in sub-directories based on the hostname
$ python3 ./scanhost.py -c conf/example.conf
Test all domains in DB for Internet Presence:
*********************************************
14:30:12 - ERROR - https://socialparadiseweb.cf.socialparadise.cf - Connection error
14:32:18 - ERROR - https://rapportannuel-assurancemaladie.paris - Connection error
14:32:23 - SUCCESS - HTTP 200 - socialmediaforsocialaction.com
Creating ./alerts/socialmediaforsocialaction.com.json : {'hostname': 'socialmediaforsocialaction.com', 'http_code': 200, 'cert_serial_number': '89:6C:03:F6:82:57:03:2A:A8:D0:E1:2F:E8:56:0E:32:83:E5:EC:29', 'webpage_title': 'Social Media for Social Action', 'ip_addr': '198.49.23.145', 'asn': '53831', 'asn_cidr': '198.49.23.0/24', 'asn_country_code': 'US', 'asn_description': 'SQUARESPACE - Squarespace, Inc., US', 'asn_abuse_email': '[email protected]'}
14:32:25 - ERROR - https://social.socialbride.co.za - Connection error
14:32:34 - SUCCESS - HTTP 503 - assurances-sociales.com
Creating ./alerts/assurances-sociales.com.json : {'hostname': 'assurances-sociales.com', 'http_code': 503, 'cert_serial_number': '1A:0D:45:D9:05:15:DC:17:6C:9F:9E:47:A5:62:03:D9:25:02:F9:3C', 'webpage_title': 'Accueil', 'ip_addr': '164.132.235.17', 'asn': '16276', 'asn_cidr': '164.132.0.0/16', 'asn_country_code': 'FR', 'asn_description': 'OVH, FR', 'asn_abuse_email': '[email protected]'}
$ python3 gethost.py --help
-h --help Print this help
-c --config Configuration file to use
--since Since when it displays findings (seconds)
$ python3 ./gethost.py -c conf/example.conf --since 36000 # 10 hours
Display all domains in DB for Internet Presence:
************************************************
socialparadiseweb.cf.socialparadise.cf None
rapportannuel-assurancemaladie.paris None
socialmediaforsocialaction.com 2019-06-12T15:54:31
social.socialbride.co.za None
$ python3 check_rules.py --help
-h --help Print this help
-c --config Configuration file to use
-d --domain Domain name to check
# No match - Keywords not found.
$ ./check_rules.py -c conf/example.conf -d www.google.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Keywords not found.
# No match - Detection threashold not reached.
$ ./check_rules.py -c conf/example.conf -d paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Detection threashold not reached.
# No match - Blacklisted keywords.
$ ./check_rules.py -c conf/example.conf -d login-paypal.gouv
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Blacklisted keywords.
# This is a match, detection threashold reached.
$ ./check_rules.py -c conf/example.conf -d login-paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
This is a match, detection threashold reached.
GNU GENERAL PUBLIC LICENSE (GPL) Version 3