BenchExec: A Framework for Reliable Benchmarking and Resource Measurement
table-generator
computes scores according to SV-COMP'24 scoring scheme.table-generator
.witnesslint
.project_url()
and url_for_version()
that can be implemented, and table-generator
will put links to these URLs in the summary table of the HTML tables (for the tool name and the version). Most existing tool-info modules were extended with project_url()
.The big change in this release is the long-awaited support for cgroups v2! Please refer to the installation instructions for how to use it (on Ubuntu/Debian, installing our package is enough). Note that this of course has not been tested yet on as many different systems as our support for cgroups v1, so there might still be some rough edges. Please provide feedback if you encounter any problems or have questions. On systems with cgroups v1, everything should work the same way as before. Thanks a lot to @globin for providing the implementation of this!
There are also some other minor improvements:
table-generator
for result files that conform to the specification but were not produced by benchexec
.This release works only on Python 3.7 and newer!
table-generator
.table-generator
for cases with many child processes and heavy system load.table-generator
will now use more than 32 child processes.correct-unconfirmed
external postprocessing of the results is necessary for this.<propertyfile>
tag in the benchmark definition does not match a file, BenchExec now terminates with an error instead of just logging a warning.table-generator
.dpkg
on Debian is incompatible with our PPA on Launchpad.This release does not change the minimum supported Python version, but we would like to remind you that BenchExec will soon stop supporting Python 3.6.
benchexec
in container mode on machines with many cores and added an optimization that reduces its impact.table-generator
no longer attempts to spawn a large number of threads (failing due to the limit on open files) on machines with many cores.This release does not change the minimum supported Python version, but we would like to remind you that BenchExec will soon stop supporting Python 3.6.
--no-container
or --full-access /sys/fs/cgroups
).
However, some users want to execute trusted tools with cgroup access (e.g., nesting runexec
) and we aim to support this.
It was found that BenchExec could hang or crash in certain cases where child cgroups were created by the process within a run and used to freeze processes (done by nested runexec
, for example), or were made inaccessible using file-system permissions.
This lead to incomplete run cleanup and processes left running or hanging in an unkillable state.
This release fixes such situations and users who allow cgroup access within runs are strongly recommended to upgrade.This release does not change the minimum supported Python version, but we would like to remind you that BenchExec will soon stop supporting Python 3.6.
--format=statistics-tex
.
These can then be included in a paper and the commands can be used to retrieve the statistics values (documentation).
The script contrib/statistics-tex.py
that provided a subset of this functionality so far is removed.
Thanks to @Sowasvonbot for implementing this!