Anti-Abuse for servers at authentication time
New tool to dump the contents of Stats DBs for debugging purposes.
man wforce_dump_entries for more information.
Replication messages now have a 'forwarding' flag, which is used to indicate when a message has been forwarded. This can be used to prevent forwarding loops.
Both the wforce and trackalert daemons support native Prometheus metrics via the new /metrics REST API endpoint. This endpoint follows the format described here.
The prometheus metrics deprecate the existing metrics functionality including the following console commands:
as well as the following REST API endpoints:
The prometheus metrics include metrics for many components that were not previously instrumented, including:
All HTTP responses now include the following headers:
Last-Modified and Date headers will always reflect the current date/time as seen by the wforce server.
The allow and report logs will now contain session_id information.
The Lua wrapper code has been updated to provide better traceback information, including line numbers, for Lua errors. This helps when writing Lua policy that triggers a Lua exception.
Redis connections could be timed-out by middleboxes, which would not be detected because keepalive was not enabled for Redis connections. Now it is enabled (always).
Previously reads from and writes to Redis were subject to the underlying socket timeout defaults. Now the timeout defaults to 100000 microseconds, and is configurable with new Lua functions: blacklistPersistRWTimeout() and whitelistPersistRWTimeout(). See wforce.conf for more details.
Under certain circumstances, relating to EOF handling when sockets are closed, REST API command statistics would be double counted. This has been fixed by refactoring the EOF handling code.
New tool to dump the contents of Stats DBs for debugging purposes.
man wforce_dump_entries for more information.
Replication messages now have a 'forwarding' flag, which is used to indicate when a message has been forwarded. This can be used to prevent forwarding loops.
Both the wforce and trackalert daemons support native Prometheus metrics via the new /metrics REST API endpoint. This endpoint follows the format described here.
The prometheus metrics deprecate the existing metrics functionality including the following console commands:
as well as the following REST API endpoints:
The prometheus metrics include metrics for many components that were not previously instrumented, including:
All HTTP responses now include the following headers:
Last-Modified and Date headers will always reflect the current date/time as seen by the wforce server.
The allow and report logs will now contain session_id information.
The Lua wrapper code has been updated to provide better traceback information, including line numbers, for Lua errors. This helps when writing Lua policy that triggers a Lua exception.
Redis connections could be timed-out by middleboxes, which would not be detected because keepalive was not enabled for Redis connections. Now it is enabled (always).
Previously reads from and writes to Redis were subject to the underlying socket timeout defaults. Now the timeout defaults to 100000 microseconds, and is configurable with new Lua functions: blacklistPersistRWTimeout() and whitelistPersistRWTimeout(). See wforce.conf for more details.
Under certain circumstances, relating to EOF handling when sockets are closed, REST API command statistics would be double counted. This has been fixed by refactoring the EOF handling code.
Previously the GeoIP2 support was limited to retrieving City and County DB information, with a fixed set of fields returned. However this meant that other DBs (e.g. ISPs, Anonymizers etc.) could not be queried, as well as additional fields in City or Country DBs.
Now there are four new Lua functions to query arbitrary fields in the Maxmind DBs:
For example:
local citydb = getGeoIP2DB("City")
local city_name = citydb:lookupStringValue(newCA(ip_address), {"city", "names", "en"})
local city_lat = citydb:lookupDoubleValue(newCA(ip_address), {"location", "latitude"})
local city_long = citydb:lookupDoubleValue(newCA(ip_address), {"location", "longitude"})
local accuracy = citydb:lookupUIntValue(newCA(ip_address), {"location", "accuracy_radius"})
local eu = citydb:lookupBoolValue(newCA(ip_address), {"country", "is_in_european_union"})
It can be useful for wforce to recognise when multiple logins belong to the same session. Therefore a new field "session_id" is added to the LoginTuple table, which allows a session id to be passed to wforce. This field is optional, so if empty it should be ignored. For example:
local session_id = lt.session_id
Previously the Report API was supplied as an experimental feature, which was supplied as a Flask application, but without support for running in production, packaging etc.
Now the Report API is fully deployable; it is packaged in a new package "wforce-report-api", and can be started/stopped via systemd. For example:
systemctl enable wforce-report-api
systemctl start wforce-report-api
The Report API is still a Flash application, which runs under gunicorn, and the configuration and deployment settings are configurable.
Report API configuration is via:
/etc/wforce-report-api/wforce-report-api-instance.conf
Gunicorn configuration is via:
/etc/wforce-report-api/wforce-report-api-web.conf
It is recommended to run Gunicorn behind an nginx proxy in a production deployment.
The existing Custom Endpoint functionality is based only on POST commands, and requires return values to be json-encoded. This presents problems with more limited clients such as firewalls or other network equipment, which only support HTTP GET, and cannot parse json-encoded return values.
The Custom GET endpoints solve this issue by allowing endpoints to be created which are based on HTTP GET, and which return data using the text/plain content type.
Custom GET endpoints do not take any parameters, so there is no way to pass data to the endpoints.
For example the following function uses the new getIPBlacklist() functionality to return a text version of the IP blacklist:
function returnTextBlacklist()
local ipbl = getIPBlacklist()
local ret_table = {}
for i,j in pairs(ipbl)
do
for k,v in pairs(j)
do
if k == "ip"
then
table.insert(ret_table, v)
end
end
end
local s = table.concat(ret_table, "\n") .. "\n"
unblacklistIP(newCA("1.2.3.4"))
return s
end
setCustomGetEndpoint("textBlacklist", returnTextBlacklist)
New Kibana Reports and Dashboard have been added to the kibana_saved_objects.json file. These reports are based on "allow" webhooks sent to elasticsearch.
The built-in webhooks add a "type" field to the webhook json to make it easier to search for specific webhook types in elasticsearch. The type field can have the following values:
In addition to the built-in blacklists there are now built-in whitelists. Entries can be added and deleted from the built-in whitelists using either the REST API or using Lua commands.
For example using the REST API:
curl -XPOST --data '{ "ip":"1.2.3.4" }' -u user:pass http://localhost:8084/?command=addWLEntry
For example using Lua:
whitelistIP(lt.remote, 3600, "This is the reason why it is blacklisted")
New functions to check the built-in black and whitelists are available from Lua:
See the wforce.conf man page for more details.
The built-in black/whitelisting functionality can be disabled so that the checks can instead be performed from Lua. This is achieved with the following Lua commands:
disableBuiltinBlacklists()
disableBuiltinWhitelists()
Every thread type in wforce is now named, so that top -H for example will show individual thread names. This can be very useful for diagnosing when particular threads are CPU-bound and could benefit from increasing the size of the thread pool for example.
The following new functions enable the return messages for built-in blacklists to be configured:
See the wforce.conf man page for more details.
The wforce daemon previously did not enable TCP keepalive on accepted sockets. The TCP keepalive socketoption is now enabled for all sockets.
Previously the GeoIP2 support was limited to retrieving City and County DB information, with a fixed set of fields returned. However this meant that other DBs (e.g. ISPs, Anonymizers etc.) could not be queried, as well as additional fields in City or Country DBs.
Now there are four new Lua functions to query arbitrary fields in the Maxmind DBs:
For example:
local citydb = getGeoIP2DB("City")
local city_name = citydb:lookupStringValue(newCA(ip_address), {"city", "names", "en"})
local city_lat = citydb:lookupDoubleValue(newCA(ip_address), {"location", "latitude"})
local city_long = citydb:lookupDoubleValue(newCA(ip_address), {"location", "longitude"})
local accuracy = citydb:lookupUIntValue(newCA(ip_address), {"location", "accuracy_radius"})
local eu = citydb:lookupBoolValue(newCA(ip_address), {"country", "is_in_european_union"})
It can be useful for wforce to recognise when multiple logins belong to the same session. Therefore a new field "session_id" is added to the LoginTuple table, which allows a session id to be passed to wforce. This field is optional, so if empty it should be ignored. For example:
local session_id = lt.session_id
Previously the Report API was supplied as an experimental feature, which was supplied as a Flask application, but without support for running in production, packaging etc.
Now the Report API is fully deployable; it is packaged in a new package "wforce-report-api", and can be started/stopped via systemd. For example:
systemctl enable wforce-report-api
systemctl start wforce-report-api
The Report API is still a Flash application, which runs under gunicorn, and the configuration and deployment settings are configurable.
Report API configuration is via:
/etc/wforce-report-api/wforce-report-api-instance.conf
Gunicorn configuration is via:
/etc/wforce-report-api/wforce-report-api-web.conf
It is recommended to run Gunicorn behind an nginx proxy in a production deployment.
The existing Custom Endpoint functionality is based only on POST commands, and requires return values to be json-encoded. This presents problems with more limited clients such as firewalls or other network equipment, which only support HTTP GET, and cannot parse json-encoded return values.
The Custom GET endpoints solve this issue by allowing endpoints to be created which are based on HTTP GET, and which return data using the text/plain content type.
Custom GET endpoints do not take any parameters, so there is no way to pass data to the endpoints.
For example the following function uses the new getIPBlacklist() functionality to return a text version of the IP blacklist:
function returnTextBlacklist()
local ipbl = getIPBlacklist()
local ret_table = {}
for i,j in pairs(ipbl)
do
for k,v in pairs(j)
do
if k == "ip"
then
table.insert(ret_table, v)
end
end
end
local s = table.concat(ret_table, "\n") .. "\n"
unblacklistIP(newCA("1.2.3.4"))
return s
end
setCustomGetEndpoint("textBlacklist", returnTextBlacklist)
New Kibana Reports and Dashboard have been added to the kibana_saved_objects.json file. These reports are based on "allow" webhooks sent to elasticsearch.
The built-in webhooks add a "type" field to the webhook json to make it easier to search for specific webhook types in elasticsearch. The type field can have the following values:
In addition to the built-in blacklists there are now built-in whitelists. Entries can be added and deleted from the built-in whitelists using either the REST API or using Lua commands.
For example using the REST API:
curl -XPOST --data '{ "ip":"1.2.3.4" }' -u user:pass http://localhost:8084/?command=addWLEntry
For example using Lua:
whitelistIP(lt.remote, 3600, "This is the reason why it is blacklisted")
New functions to check the built-in black and whitelists are available from Lua:
See the wforce.conf man page for more details.
The built-in black/whitelisting functionality can be disabled so that the checks can instead be performed from Lua. This is achieved with the following Lua commands:
disableBuiltinBlacklists()
disableBuiltinWhitelists()
Every thread type in wforce is now named, so that top -H for example will show individual thread names. This can be very useful for diagnosing when particular threads are CPU-bound and could benefit from increasing the size of the thread pool for example.
The following new functions enable the return messages for built-in blacklists to be configured:
See the wforce.conf man page for more details.
The wforce daemon previously did not enable TCP keepalive on accepted sockets. The TCP keepalive socketoption is now enabled for all sockets.
Fix unitialized variable causing getDBStats command to always return blacklisted: true.
Threadn support was implemented, but wasn't working due to compile dependency issues. This is now fixed meaning that top will show thread names when called with the -H option for example.
Support for sibling using tcp was added in 2.0.0, however when siblings are defined as TCP, wforce was attempting to connect to them on startup. If several wforce servers were started at the same time, this would cause a delay while each tries to connect to the other over TCP but fails. This fix delays TCP connection until the first replication attempt. Note that any static blacklist entries in the config will cause a replication attempt on startup and this will trigger the same startup delay behaviour. Thus it is not recommended to create static blacklist entries in the config.
Replication of data used the protobuf "string" type, which gets validated on parsing to ensure it is a UTF-8 string. However, certain fields such as login can contain non-UTF-8 characters, so this fix changes to use the "bytes" type instead. This fix is backwards compatible because string and bytes types are identical on the wire.
The Lua infoLog, errorLog etc. functions would previously, when called as 'errorLog("foo", {})' log:
foo :
Now the same call will log only:
foo
The allow command will now return additional information when an IP/Login is blacklisted. The 'r_attrs' object will contain four new fields:
The getDBStats command will return additional information about blacklisted objects:
The following new Lua functions are available:
See the wforce.conf manpage for more details.
The webhook support has been completely refactored in order to achieve much higher performance with fewer resources. Previously a very high number of webhook threads was required to achieve good performance, whereas now a much smaller number of threads can achieve the same performance.
The previous per-webhook configuration key "num_conns" is no longer supported. Instead the global configuration setting "setNumWebHookConnsPerThread" is used.
For example:
setNumWebHookConnsPerThread(10)
The default is 10 connections per webhook thread.
The function setWebHookTimeoutSecs() is used to control the time for webhook requests, e.g.:
setWebHookTimeoutSecs(2)
The showStringStatsDB() command now shows whether each StatsDB is configured for replication or not.
For example:
> showStringStatsDB()
DB Name Repl? Win Size/No Max Size Cur Size Field Name Type
MyDB1 yes 1/15 524288 0 countLogins int
diffPasswords hll
MyDB2 no 600/6 5000 2093 diffIPs hll
The siblings() command now shows success and failure stats about received messages as well as sent messages.
For example:
> siblings()
Address Send Successes Send Failures Rcv Successes Rcv Failures Note
127.0.0.1:4001 0 0 17 0
127.0.0.1:4002 0 0 0 0 Self
Two new functions, "addCustomStat" and "incCustomStat" can be used to keep track of custom statistics. A new custom counter is created with "addCustomStat", e.g.
addCustomStat("custom_stat1")
Custom statistics are counters which track statistics over a 5 minute period. Every 5 minutes, the current values of all the custom stats counters are logged to the wforce log file, before the counters are reset.
Stats can be incremented with the "incCustomStat" command:
incCustomStat("custom_stat1")
Previously there were no statistics logged for all the REST API commands; only allow and report commands. Now all REST API commands are tracked and statistics are reported, including for custom endpoints created from Lua.
For example:
command stats last 300 secs: addBLEntry=42 allow=393827 allow_allowed=299221 allow_blacklisted=3224 allow_denied=9884 allow_tarpitted=8373 delBLEntry=3 getBL=3949 getDBStats=3229 ping=83764 report=38473 reset=0 stats=0 syncDBs=0 syncDone=0
custom stats last 300 secs: customFunc1=3401
Maxmind are in the process of deprecating the GeoIP Legacy DB support, therefore this release supports GeoIP2 format databases, i.e. the MMDB format.
This release therefore deprecates the GeoIP legacy functions, which will be removed in a later release. The following functions are deprecated:
Due to differences in the way that the GeoIP2 API works, GeoIP Databases must be opened by specifying the filename of the database to be used. For example:
newGeoIP2DB("CityDB", "/usr/share/GeoIP/GeoLite2-City.mmdb")
To retrieve a GeoIP DB to conduct queries against, use the following command:
local citydb = getGeoIP2DB("CityDB")
Once a database has been assigned to a local variable, it can be queried, for example:
my_country = countrydb:lookupCountry(newCA("8.8.8.8"))
my_country = countrydb:lookupCountry(lt.remote)
local my_isp = ispdb:lookupISP(newCA("128.243.16.21"))
local gip_record = citydb:lookupCity(lt.remote)
local my_city = gip_record.city
local my_latitude = gip_record.latitude
For full details see "man wforce.conf".
The twReset() function can be used to reset all the fields for a given key, but previously there was no way to reset an individual field. Now the function "twResetField()" can be used to achieve this, e.g.:
statsdb:twResetField(lt.login, "countLogins")
The HLL and CountMin types of StatsDB entries are probablistic data structures, which trade accuracy for memory usage. Previously the accuracy (and thus memory usage) was hardcoded, however now their accuracy can be tuned. Increasing accuracy however means a (potentially very large) increase in memory usage, so extreme care must be taken before modifying these parameters.
The function setHLLBits() can be used to change the accuracy of the HLL type. The value supplied can be between 4 and 30, with the default value being 6.
The function setCountMinBits() can be used to set the accuracy of the CountMin type.
See the wforce.conf manpage for full details.
Normally when a wforce instance starts, it has a "fresh" set of Stats DBs, and therefore can take a reasonable period of time (an hour or more depending on the policy) before it starts giving the same answers as other wforce servers in a cluster which have been running for some time. This issue is now addressed with the ability to tell a wforce server to find another server which has been running for longer than a configurable period of time, from which it can download the entire set of Stats DBs. While a server is in the process of downloading the Stats DBs from another server, it is in a "warmup" state; this fact is reflected in a new return value from the "ping" REST API endpoint.
In order to enable this feature, the "addSyncHosts()" function must be used, once for each host that will be contacted on startup, for example:
-- Add 10.2.3.1:8084 as a sync host,
-- and use the password "super"
-- Send the DB dump to 10.2.1.1:4001
-- and let me know on 10.2.1.1:8084 when the dump is finished
addSyncHost("10.2.3.1:8084", "super", "10.2.1.1:4001", "10.2.1.1:8084")
The default time that the sync hosts must have been "up" for is 3600 seconds, however that can be configured using "setMinSyncHostUptime()", e.g.:
setMinSyncHostUptime(1800)
The replication of data between the sync host and the wforce instance that is starting up is always performed over TCP.
See wforce.conf for full details.
The addSiblings() and setSiblings() functions now take an extra (optional) parameter that specifies whether the replication should use UDP or TCP. The default is UDP. If the protocol is specified, the port must also be specified.
For example:
setSiblings({"127.0.1.2", "127.0.1.3:4004", "127.0.2.23:4004:tcp"})
addSibling("192.168.1.23")
addSibling("192.168.1.23:4001:udp")
addSibling("192.168.1.23:4003:tcp")
The new "-f" or "--facility" command line option can be used to set the syslog facility used for wforce logging.
For example:
wforce -f "local0"
A new daemon "trackalert" is part of the product. This daemon shares a lot of functionality with wforce, particularly in terms of Lua support. However the REST API for trackalert is much simpler, consisting only of "report" and "stats" endpoints.
The trackalert daemon is designed to process login reports sent to it by wforce, use those reports to determine whether the login is suspicious. It is also designed to run Lua functions on a periodic basis using a configurable scheduler, in order to run tasks such as finding suspicious IPs or compromised accounts.
The trackalert daemon works best with the Lua policy delivered in the separate wforce-policy package. That policy implements suspicious login alerts using historical report data stored in Elasticsearch, as well as periodic searches of Elasticsearch to find suspicious IPs and compromised accounts.
For the trackalert daemon to be effective, wforce must be configured to send reports to both trackalert and Elasticsearch using webhooks.
This release ships with sample logstash configuration and Elasticsearch mapping template to ensure that the report data is stored in a consistent form by Elasticsearch.
The minimum version of ELK (Elasticsearch, Logstash, Kibana) that is required is version 6.
This release ships with a sample set of reports and dashboards for Kibana (version 6+).
A REST API to handle querying and modification of the data stored in Elasticsearch.
Currently this ships as an informational feature for experimentation; a future release will ship this API as a package shipping a deployable and supported webapp.
The API is documented using OpenAPI (Swagger); consult the documentation on documentation.open-xchange.com.
The Lua infoLog, errorLog etc. functions would previously, when called as 'errorLog("foo", {})' log:
foo :
Now the same call will log only:
foo
The allow command will now return additional information when an IP/Login is blacklisted. The 'r_attrs' object will contain four new fields:
The getDBStats command will return additional information about blacklisted objects:
The following new Lua functions are available:
See the wforce.conf manpage for more details.
The webhook support has been completely refactored in order to achieve much higher performance with fewer resources. Previously a very high number of webhook threads was required to achieve good performance, whereas now a much smaller number of threads can achieve the same performance.
The previous per-webhook configuration key "num_conns" is no longer supported. Instead the global configuration setting "setNumWebHookConnsPerThread" is used.
For example:
setNumWebHookConnsPerThread(10)
The default is 10 connections per webhook thread.
The function setWebHookTimeoutSecs() is used to control the time for webhook requests, e.g.:
setWebHookTimeoutSecs(2)
The showStringStatsDB() command now shows whether each StatsDB is configured for replication or not.
For example:
> showStringStatsDB()
DB Name Repl? Win Size/No Max Size Cur Size Field Name Type
MyDB1 yes 1/15 524288 0 countLogins int
diffPasswords hll
MyDB2 no 600/6 5000 2093 diffIPs hll
The siblings() command now shows success and failure stats about received messages as well as sent messages.
For example:
> siblings()
Address Send Successes Send Failures Rcv Successes Rcv Failures Note
127.0.0.1:4001 0 0 17 0
127.0.0.1:4002 0 0 0 0 Self
Two new functions, "addCustomStat" and "incCustomStat" can be used to keep track of custom statistics. A new custom counter is created with "addCustomStat", e.g.
addCustomStat("custom_stat1")
Custom statistics are counters which track statistics over a 5 minute period. Every 5 minutes, the current values of all the custom stats counters are logged to the wforce log file, before the counters are reset.
Stats can be incremented with the "incCustomStat" command:
incCustomStat("custom_stat1")
Previously there were no statistics logged for all the REST API commands; only allow and report commands. Now all REST API commands are tracked and statistics are reported, including for custom endpoints created from Lua.
For example:
command stats last 300 secs: addBLEntry=42 allow=393827 allow_allowed=299221 allow_blacklisted=3224 allow_denied=9884 allow_tarpitted=8373 delBLEntry=3 getBL=3949 getDBStats=3229 ping=83764 report=38473 reset=0 stats=0 syncDBs=0 syncDone=0
custom stats last 300 secs: customFunc1=3401
Maxmind are in the process of deprecating the GeoIP Legacy DB support, therefore this release supports GeoIP2 format databases, i.e. the MMDB format.
This release therefore deprecates the GeoIP legacy functions, which will be removed in a later release. The following functions are deprecated:
Due to differences in the way that the GeoIP2 API works, GeoIP Databases must be opened by specifying the filename of the database to be used. For example:
newGeoIP2DB("CityDB", "/usr/share/GeoIP/GeoLite2-City.mmdb")
To retrieve a GeoIP DB to conduct queries against, use the following command:
local citydb = getGeoIP2DB("CityDB")
Once a database has been assigned to a local variable, it can be queried, for example:
my_country = countrydb:lookupCountry(newCA("8.8.8.8"))
my_country = countrydb:lookupCountry(lt.remote)
local my_isp = ispdb:lookupISP(newCA("128.243.16.21"))
local gip_record = citydb:lookupCity(lt.remote)
local my_city = gip_record.city
local my_latitude = gip_record.latitude
For full details see "man wforce.conf".
The twReset() function can be used to reset all the fields for a given key, but previously there was no way to reset an individual field. Now the function "twResetField()" can be used to achieve this, e.g.:
statsdb:twResetField(lt.login, "countLogins")
The HLL and CountMin types of StatsDB entries are probablistic data structures, which trade accuracy for memory usage. Previously the accuracy (and thus memory usage) was hardcoded, however now their accuracy can be tuned. Increasing accuracy however means a (potentially very large) increase in memory usage, so extreme care must be taken before modifying these parameters.
The function setHLLBits() can be used to change the accuracy of the HLL type. The value supplied can be between 4 and 30, with the default value being 6.
The function setCountMinBits() can be used to set the accuracy of the CountMin type.
See the wforce.conf manpage for full details.
Normally when a wforce instance starts, it has a "fresh" set of Stats DBs, and therefore can take a reasonable period of time (an hour or more depending on the policy) before it starts giving the same answers as other wforce servers in a cluster which have been running for some time. This issue is now addressed with the ability to tell a wforce server to find another server which has been running for longer than a configurable period of time, from which it can download the entire set of Stats DBs. While a server is in the process of downloading the Stats DBs from another server, it is in a "warmup" state; this fact is reflected in a new return value from the "ping" REST API endpoint.
In order to enable this feature, the "addSyncHosts()" function must be used, once for each host that will be contacted on startup, for example:
-- Add 10.2.3.1:8084 as a sync host,
-- and use the password "super"
-- Send the DB dump to 10.2.1.1:4001
-- and let me know on 10.2.1.1:8084 when the dump is finished
addSyncHost("10.2.3.1:8084", "super", "10.2.1.1:4001", "10.2.1.1:8084")
The default time that the sync hosts must have been "up" for is 3600 seconds, however that can be configured using "setMinSyncHostUptime()", e.g.:
setMinSyncHostUptime(1800)
The replication of data between the sync host and the wforce instance that is starting up is always performed over TCP.
See wforce.conf for full details.
The addSiblings() and setSiblings() functions now take an extra (optional) parameter that specifies whether the replication should use UDP or TCP. The default is UDP. If the protocol is specified, the port must also be specified.
For example:
setSiblings({"127.0.1.2", "127.0.1.3:4004", "127.0.2.23:4004:tcp"})
addSibling("192.168.1.23")
addSibling("192.168.1.23:4001:udp")
addSibling("192.168.1.23:4003:tcp")
The new "-f" or "--facility" command line option can be used to set the syslog facility used for wforce logging.
For example:
wforce -f "local0"
A new daemon "trackalert" is part of the product. This daemon shares a lot of functionality with wforce, particularly in terms of Lua support. However the REST API for trackalert is much simpler, consisting only of "report" and "stats" endpoints.
The trackalert daemon is designed to process login reports sent to it by wforce, use those reports to determine whether the login is suspicious. It is also designed to run Lua functions on a periodic basis using a configurable scheduler, in order to run tasks such as finding suspicious IPs or compromised accounts.
The trackalert daemon works best with the Lua policy delivered in the separate wforce-policy package. That policy implements suspicious login alerts using historical report data stored in Elasticsearch, as well as periodic searches of Elasticsearch to find suspicious IPs and compromised accounts.
For the trackalert daemon to be effective, wforce must be configured to send reports to both trackalert and Elasticsearch using webhooks.
This release ships with sample logstash configuration and Elasticsearch mapping template to ensure that the report data is stored in a consistent form by Elasticsearch.
The minimum version of ELK (Elasticsearch, Logstash, Kibana) that is required is version 6.
This release ships with a sample set of reports and dashboards for Kibana (version 6+).
A REST API to handle querying and modification of the data stored in Elasticsearch.
Currently this ships as an informational feature for experimentation; a future release will ship this API as a package shipping a deployable and supported webapp.
The API is documented using OpenAPI (Swagger); consult the documentation on documentation.open-xchange.com.
The Lua infoLog, errorLog etc. functions would previously, when called as 'errorLog("foo", {})' log:
foo :
Now the same call will log only:
foo
The allow command will now return additional information when an IP/Login is blacklisted. The 'r_attrs' object will contain four new fields:
The getDBStats command will return additional information about blacklisted objects:
The following new Lua functions are available:
See the wforce.conf manpage for more details.
The webhook support has been completely refactored in order to achieve much higher performance with fewer resources. Previously a very high number of webhook threads was required to achieve good performance, whereas now a much smaller number of threads can achieve the same performance.
The previous per-webhook configuration key "num_conns" is no longer supported. Instead the global configuration setting "setNumWebHookConnsPerThread" is used.
For example:
setNumWebHookConnsPerThread(10)
The default is 10 connections per webhook thread.
The function setWebHookTimeoutSecs() is used to control the time for webhook requests, e.g.:
setWebHookTimeoutSecs(2)
The showStringStatsDB() command now shows whether each StatsDB is configured for replication or not.
For example:
> showStringStatsDB()
DB Name Repl? Win Size/No Max Size Cur Size Field Name Type
MyDB1 yes 1/15 524288 0 countLogins int
diffPasswords hll
MyDB2 no 600/6 5000 2093 diffIPs hll
The siblings() command now shows success and failure stats about received messages as well as sent messages.
For example:
> siblings()
Address Send Successes Send Failures Rcv Successes Rcv Failures Note
127.0.0.1:4001 0 0 17 0
127.0.0.1:4002 0 0 0 0 Self
Two new functions, "addCustomStat" and "incCustomStat" can be used to keep track of custom statistics. A new custom counter is created with "addCustomStat", e.g.
addCustomStat("custom_stat1")
Custom statistics are counters which track statistics over a 5 minute period. Every 5 minutes, the current values of all the custom stats counters are logged to the wforce log file, before the counters are reset.
Stats can be incremented with the "incCustomStat" command:
incCustomStat("custom_stat1")
Previously there were no statistics logged for all the REST API commands; only allow and report commands. Now all REST API commands are tracked and statistics are reported, including for custom endpoints created from Lua.
For example:
command stats last 300 secs: addBLEntry=42 allow=393827 allow_allowed=299221 allow_blacklisted=3224 allow_denied=9884 allow_tarpitted=8373 delBLEntry=3 getBL=3949 getDBStats=3229 ping=83764 report=38473 reset=0 stats=0 syncDBs=0 syncDone=0
custom stats last 300 secs: customFunc1=3401
Maxmind are in the process of deprecating the GeoIP Legacy DB support, therefore this release supports GeoIP2 format databases, i.e. the MMDB format.
This release therefore deprecates the GeoIP legacy functions, which will be removed in a later release. The following functions are deprecated:
Due to differences in the way that the GeoIP2 API works, GeoIP Databases must be opened by specifying the filename of the database to be used. For example:
newGeoIP2DB("CityDB", "/usr/share/GeoIP/GeoLite2-City.mmdb")
To retrieve a GeoIP DB to conduct queries against, use the following command:
local citydb = getGeoIP2DB("CityDB")
Once a database has been assigned to a local variable, it can be queried, for example:
my_country = countrydb:lookupCountry(newCA("8.8.8.8"))
my_country = countrydb:lookupCountry(lt.remote)
local my_isp = ispdb:lookupISP(newCA("128.243.16.21"))
local gip_record = citydb:lookupCity(lt.remote)
local my_city = gip_record.city
local my_latitude = gip_record.latitude
For full details see "man wforce.conf".
The twReset() function can be used to reset all the fields for a given key, but previously there was no way to reset an individual field. Now the function "twResetField()" can be used to achieve this, e.g.:
statsdb:twResetField(lt.login, "countLogins")
The HLL and CountMin types of StatsDB entries are probablistic data structures, which trade accuracy for memory usage. Previously the accuracy (and thus memory usage) was hardcoded, however now their accuracy can be tuned. Increasing accuracy however means a (potentially very large) increase in memory usage, so extreme care must be taken before modifying these parameters.
The function setHLLBits() can be used to change the accuracy of the HLL type. The value supplied can be between 4 and 30, with the default value being 6.
The function setCountMinBits() can be used to set the accuracy of the CountMin type.
See the wforce.conf manpage for full details.
Normally when a wforce instance starts, it has a "fresh" set of Stats DBs, and therefore can take a reasonable period of time (an hour or more depending on the policy) before it starts giving the same answers as other wforce servers in a cluster which have been running for some time. This issue is now addressed with the ability to tell a wforce server to find another server which has been running for longer than a configurable period of time, from which it can download the entire set of Stats DBs. While a server is in the process of downloading the Stats DBs from another server, it is in a "warmup" state; this fact is reflected in a new return value from the "ping" REST API endpoint.
In order to enable this feature, the "addSyncHosts()" function must be used, once for each host that will be contacted on startup, for example:
-- Add 10.2.3.1:8084 as a sync host,
-- and use the password "super"
-- Send the DB dump to 10.2.1.1:4001
-- and let me know on 10.2.1.1:8084 when the dump is finished
addSyncHost("10.2.3.1:8084", "super", "10.2.1.1:4001", "10.2.1.1:8084")
The default time that the sync hosts must have been "up" for is 3600 seconds, however that can be configured using "setMinSyncHostUptime()", e.g.:
setMinSyncHostUptime(1800)
The replication of data between the sync host and the wforce instance that is starting up is always performed over TCP.
See wforce.conf for full details.
The addSiblings() and setSiblings() functions now take an extra (optional) parameter that specifies whether the replication should use UDP or TCP. The default is UDP. If the protocol is specified, the port must also be specified.
For example:
setSiblings({"127.0.1.2", "127.0.1.3:4004", "127.0.2.23:4004:tcp"})
addSibling("192.168.1.23")
addSibling("192.168.1.23:4001:udp")
addSibling("192.168.1.23:4003:tcp")
The new "-f" or "--facility" command line option can be used to set the syslog facility used for wforce logging.
For example:
wforce -f "local0"
A new daemon "trackalert" is part of the product. This daemon shares a lot of functionality with wforce, particularly in terms of Lua support. However the REST API for trackalert is much simpler, consisting only of "report" and "stats" endpoints.
The trackalert daemon is designed to process login reports sent to it by wforce, use those reports to determine whether the login is suspicious. It is also designed to run Lua functions on a periodic basis using a configurable scheduler, in order to run tasks such as finding suspicious IPs or compromised accounts.
The trackalert daemon works best with the Lua policy delivered in the separate wforce-policy package. That policy implements suspicious login alerts using historical report data stored in Elasticsearch, as well as periodic searches of Elasticsearch to find suspicious IPs and compromised accounts.
For the trackalert daemon to be effective, wforce must be configured to send reports to both trackalert and Elasticsearch using webhooks.
This release ships with sample logstash configuration and Elasticsearch mapping template to ensure that the report data is stored in a consistent form by Elasticsearch.
The minimum version of ELK (Elasticsearch, Logstash, Kibana) that is required is version 6.
This release ships with a sample set of reports and dashboards for Kibana (version 6+).
A REST API to handle querying and modification of the data stored in Elasticsearch.
Currently this ships as an informational feature for experimentation; a future release will ship this API as a package shipping a deployable and supported webapp.
The API is documented using OpenAPI (Swagger); consult the documentation on documentation.open-xchange.com.