Pgbadger Versions Save

2023-11-27 - v12.3

This is a maintenance release of pgBadger that fixes issues reported by users since last release. It also adds some new features:

• Add option --include-pid to only report events related to a session pid (%p). Can be used multiple time. Thanks to Henrietta Dombrovskaya for the feature request.
• Add option --include-session to only report events related to the session id (%c). Can be used multiple time. Thanks to Henrietta Dombrovskaya for the feature request.
• Add option --dump-raw-csv to only parse the log and dump the information into CSV format. No further processing is done, no report is generated. Thanks to Henrietta Dombrovskaya for the feature request.

Here is the complete list of changes and acknowledgments:

• Update pgFormatter to version 5.5
• Fix end date of parsing with jsonlog format. Thanks to jw1u1 for the report.
• Fix typo in "Sessions per application". Thanks to fairyfar for the patch.
• Fix "INSERT/UPDATE/DELETE Traffic" chart bug. Thanks to fairyfar for the patch.
• Fix parsing of orphan lines with bind queries. Thanks to youxq for the report.
• Fix Analyze per table report with new PG versions. Thanks to Jean-Christophe Arnu for the patch.
• Fix syslog entry parser when the syslog timestamp contains milliseconds. Thanks to Pavel Rabel for the report.

2023-12-25 - v12.4

This is a maintenance release of pgBadger that fixes issues reported by users since last release.

• Fix pgbouncer report with version 1.21. Thanks to Ales Zeleny for the patch.
• Prevent parallelism perl file to be higher than the number of files. Thanks to maliangzhu for the report.
• Fix regression test broken since v12.3. Thanks to ieshin for the report.
• Fix cases where LOG entries where counted as ERROR log level entries. Thanks to Matti Linnanvuori for the report.

2023-08-20 - v12.2

This is a maintenance release of pgBadger that fixes issues reported by users since last release. It also adds two new features:

• Add support for max, avg, min autovacuum duration. Thanks to Francisco Reinolds for the patch.
• Add support for pgbouncer's average waiting time. Thanks to Francisco Reinolds for the patch.

Here is the complete list of changes and acknowledgments:

• Fix broken HTML output when application name contains <...>. Thanks to Fabio Geiss for the report.
• Fix incorrect association of orphan lines when a filter on database was applied. Thanks to jcasanov for the report.
• Fix logplex prefix parsing.
• Fix logplex orphan lines detection.
• Fix autovacuum's system usage: CPU: ... line parsing. Thanks to Francisco Reinolds for the patch.
• Avoid prepending output directory if output is stdout.
• Standardize Average Query Duration label. Thanks to Francisco Reinolds for the patch
• Update documentation for new pgbadger options. Thanks to Francisco Reinolds for the patch.
• Fix case where parsing was not aborted when no file handle can be opened. Thanks to vp for the report.
• Fix help by adding %p and %t as mandatory placeholders log information. Thanks to Christophe Courtois for the patch.
• Fix --retention parameter. Thanks to Bertrand Bourgier for the patch.
• Fix cleanup output directory removed by commit 0e5c7d5 when HTML output dir is set. Thanks to Bertrand Bourgier for the report.
• Fix output extension when destination directory contain a character that need to be escaped in regexp. Thanks to Bertrand Bourgier for the patch.
• Replace calls to POSIX::strftime("%s", ....) by a call to localtime for Windows port. Thanks to Bertrand Bourgier for the patch.
• Fix html output dir cleanup. Thanks to Bertrand Bourgier for the patch.
• Use https for explain URL by default. Thanks to Philipp Trulson for the patch.

2023-03-20 - v12.1

This is a maintenance release of pgBadger that fixes issues reported by users since past six months.

Here is the complete list of changes and acknowledgments:

• Fix parsing of multiline parameters. Thanks to Bekir Niyaz for the report.
• Fix failure to normalize query with ::tsrange. Thanks to Philippe Griboval for the report.
• Add logical decoding consistent point and start for slot log entries to the events report.
• Handle other ns + timezone format in timestamp. Thanks to Ronan Dunklau for the report.
• Fix detection of %m when notation with T is used. Thanks to Ronan Dunklau for the report.
• Add parsing of CloudNativePG generated logs. Thanks to codrut panea for the patch.
• Fix unused option --outdir in report generation. Thanks to Frederic Guiet for the report.
• Update README with last documentation changes. Thanks to Manisankar for the report.
• Fix a typo in pgbadger examples. Thanks to Shinichi Hashiba for the patch.

2022-09-13 - v12.0

This major release of pgBadger fixes some issues reported by users since past five months. As usual there is also new features and improvements:

• Remove support to Tsung output.
• Improve pgbadger performances when there are hundred of bind parameters to replace.
• Remove option -n | --nohighlight which is no more used since upgrade to pgFormatter 4.
• Use POST method to send auto_explain plan to explain.depesz.com to avoid GET length parameter limit.
• Apply --exclude-query and --include-query to bind/parse traces.
• Add link to pgBadger report examples to documentation.

Here is the complete list of changes and acknowledgments:

• Fix monthly reports that was failing on "log file ... must exists". Thanks to Jaume Sabater for the report.
• Fix pgbouncer start parsing debug message when input is stdin. Thanks to Ales Zeleny for the report.
• Remove support to Tsung output.
• Drastically improve pgbadger performances for bind parameters replacement that could make pgbadger run infinitely when there was hundred of parameters. Thanks to Monty Mobile for the report.
• Fix documentation about pgBadger return codes and also some wrong return code at some places. Thanks to Jaume Sabater for the report.
• Fix several typo. Thanks to David Gilman for the patch.
• Remove option -n | --nohighlight which is no more used since upgrade to pgFormatter 4. Thanks to Elena Indrupskaya for the report.
• Lot of pgbadger documentation fixes. Thanks to Elena Indrupskay from Postgres Pro for the patch.
• Allow half hour in --log-timezone and --timezone, value can be an integer, ex: 2 or a float, ex: 2.5. Thanks to Mujjamil-K for the feature request.
• Allow use of regexp for --exclude-app and --exclude-client. Thanks to rdnkrkmz for the feature request.
• Allow use of --explain-url with previous commit and restore the limitation to explain text format.
• Use POST method to send auto_explain plan to explain.depesz.com to avoid GET length parameter limit. Thanks to hvisage for the report.
• Apply --exclude-query and --include-query to bind/parse traces. Thanks to Alec Lazarescu for the report.
• Fix parsing of autovacuum stats from RDS logs. Thanks to David Gilman for the report.
• Fix passing of log format when parsing remote log. Thanks to spookypeanut the report.
• Add link to pgBadger report examples to documentation.
• Fix Session per user reports. Thanks to vitalca for the report.
• Fix jsonlog parsing from PG15 ouput
• Fix text-based error/events reporting. Thanks to Michael Banck for the patch
• Fix regexp typo in normalize_error(). Thanks to Michael Banck for the patch.

2022-04-08 - v11.8

This release of pgBadger fix some issues reported by users since past three months and especially two fixes on new log entries detection in incremental mode.

• Fix detection of new log entries with timestamp when millisecond (%m) or epoch (%n) was used in log_line_prefix.
• Fix detection of new log entries in local file when multi-process was not used.

Here is the complete list of changes and acknowledgments:

  - Full review and simplification of the log file change detection.
  - Reports messages "could not (receive|send) data (from|to) client" in the
    Events reports. Thanks to Adrien Nayrat for the report.
  - Fix parsing issue when the name of a prepared query contain the ':'
    character. Thanks to aleszeleny for the report.
  - Fix detection of new log entries with timestamp when millisecond (%m) or
    epoch (%n). Thanks to aleszeleny for the report.
  - Fix detection of new log entries in local file when multiprocess was not
    used. Thanks to aleszeleny for the report.
  - Fix detection of new log entries in remote files through ssh. Thanks to
    Luca Ferrari for the report
  - Fix garbage in username of "Connections per user" report. Thanks to
    caseyandgina for the report.
  - Fix ssh command when using URI, the ssh options was missing. Thanks to Luca
    Ferrari for the report.
  - Handle queryid %Q placeholder. Thanks to Adrien Nayrat for the patch.
  - Fix typo in error sentence. Thanks to Luca Ferrari for the patch
  - Report message: "server process was terminated by signal" in the Events
    report. Thanks to Avi Vallarapu for the report.
  - doc: fix filename for incremental every week command. Thanks to Theophile
    Helleboid for the patch.
  - t/04_advanced.t: Fix syslog test. Thanks to Christoph Berg for the patch.

2022-01-23 - v11.7

This release of pgBadger fix some issues reported by users since past five months as well as some improvements:

• Add new option --no-progressbar option to not display it but keep the other outputs.
• Add new option --day-report that can be used to rebuild an HTML report over the specified day. Like option --month-report but only for a day. It requires the incremental output directories and the presence of all necessary binary data files. The value is date in format: YYYY-MM-DD
• Improve parsing of Heroku logplex and cloudsql json logs.

Here is the complete list of changes and acknowledgments:

• Update contribution guidelines and Makefile.PL to improve consistency, clarity, and dependencies. Thanks to diffuse for the patch.
• Fix use of last parse file (--last-parsed) with binary mode. Thanks to wibrt for the report.
• Add regression test for --last-parsed use and fix regression test on report for temporary files only.
• Fix title for session per host graph. Thanks to Norbert Bede for the report.
• Fix week number when computing weeks reports when --iso-week-number and --incremental options was enabled. Thanks to hansgv for the report.
• Add --no-progressbar option to not display it and keep the other outputs. Thanks to seidlmic for the feature request.
• Prevent too much unknown format line prints in debug mode for multi-line jsonlog.
• Fix parsing of single line cloudsql json log. Thanks to Thomas Leclaire for the report.
• Fix temporary files summary with log_temp_files only.
• Print debug message with -v even if -q or --quiet is used.
• Fix autodetection of jsonlog file.
• Fix parsing of cloudsql log file. Thanks to Luc Lamarle for the report.
• Fixes pid extraction in parse_json_input. Thanks to Francois Scala for the patch.
• Add new option --day-report with value as date in format: YYYY-MM-DD that can be used to rebuild an HTML report over the specified day. Thanks to Thomas Leclaire for the feature request.
• Fix query counter in progress bar. Thanks to Guillaume Lelarge for the report.
• Fix incomplete queries stored for top bind and prepare reports.
• Fix normalization of object identifier, in some case the numbers was replaced by a ?.
• Fix unformatted normalized queries when there is a comment at beginning.
• Fix multi-line in stderr format when --dbname is used. Thanks to Guillaume Lelarge for the report.
• Fix not generated reports in incremental mode when --dbname is used. Thanks to Dudley Perkins for the report.
• Do not die anymore if a binary file is not compatible, switch to next file. Thanks to Thomas Leclaire for the suggestion.
• Fix Heroku logplex format change in pgbadger parser. Thanks to François Pietka for the report.

2021-09-04 - v11.6

This release of pgBadger fix some issues reported by users since past seven months as well as some improvements:

• Add detection of Query Id in log_line_prefix new in PG14. Thanks to Florent Jardin for the report.
• Add advanced regression tests with db exclusion and the explode feature. Thanks to MigOps Inc for the patch.
• Apply multiprocess to report generation when --explode is used. Thanks to MigOps Inc for the patch and Thomas Leclaire for the feature request.
• Add --iso-week-number in incremental mode, calendar's weeks start on a Monday and respect the ISO 8601 week number, range 01 to 53, where week 1 is the first week that has at least 4 days in the new year. Thanks to Alex Muntada for the feature request.
• Add command line option --keep-comments to not remove comments from normalized queries. It can be useful if you want to distinguish between same normalized queries. Thanks to Stefan Corneliu Petrea for the feature request.
• Skip INFO lines introduced in PostgreSQL log file by third parties software. Thanks to David Piscitelli for the report.
• Add compatibility with PostgresPro log file including rows number and size in bytes following the statement duration. Thanks to panatamann for the report.
• Parse times with T's to allow using the timestamps from journalctl. Thanks to Graham Christensen for the patch.
• Improve Windows port. Thanks to Bertrand Bourgier for the patches.

Important note:

• Expect that --iso-week-number will be the default in next major release and that --start-monday option will be removed as the week will always start a Monday. The possibility to have week reports start a Sunday will be removed to simplify the code.

Here is the complete list of changes and acknowledgments:

  - Fix duplicate of warning message: 
      "database ... must be vacuumed within ... transactions".
    Thank to Christophe Courtois for the report.
  - Fix use of uninitialized variable. Thanks to phiresky for the report.
  - Improve query id detection, it can be negative, as well as read it
    from csvlog.
  - Fix case where last file in incremental mode is always parsed even if
    it was already done. Thanks to  Thomas Leclaire for the report.
  - Update syslog format regex to handle where session line indicator
    only contains one int vs two ints separated by dash. Thanks to
    Timothy Alexander for the patch.
  - Fix --exclude-db option to create anyway the related report with json
    log. Thanks to MigOps Inc for the patch and Thomas Leclaire for the
    report.
  - Add regression test about Storable buggy version.
  - Fix use of uninitialized value in substitution iterator in incremental
    mode during the week report generation. Thanks to Thomas Leclaire,
    Michael Vitale, Sumeet Shukla and Stefan Corneliu Petrea for the report.
  - Add 'g' option to replace all bind parameters. Thanks to Nicolas Lutic
    and Sebastien Lardiere for the patch.
  - Documentation improvements. Thanks to Stefan Petrea for the patch.
  - Fixes change log time zone calculation. Thanks to Stefan Petrea for
    the patch.
  - Fix log filter by begin/end time.
  - Fix wrong association of orphan lines for multi-line queries with a
    filter on database. Thanks to Abhishek Mehta for the report.
  - Fix reports in incremental mode when --dbname parameter is partially
    ignored with "explode" option (-E). Thanks to lrevest for the report.
  - Update javascript resources.
  - Fix display of menu before switching to hamburger mode when screen is
    reduced. Thanks to Guillaume Lelarge for the report.
  - Fix bind parameters values over multiple lines in the log that were
    not well supported.
  - Apply same fix for previous patch than in pgFormatter.
  - Fix an other use of uninitialized value in substitution iterator from
    pgFormatter code. Thanks to Christophe Courtois for the report.
  - Fix query normalization. Thanks to Jeffrey Beale for the patch.
  - Be sure that all statements end with a semicolon when --dump-all-queries
    is used. Thanks to Christian for the report.
  - Fix typo and init of EOL type with multiple log files.
  - Add auto detection of EOL type to fix LAST_PARSED offset when OEL is on
    2 bytes (Windows case). Thanks to Bertrand Bourgier for the patch.
  - Fix get_day_of_week() port on Windows where strftime %u is not supported.
    Thanks to Bertrand Bourgier for the patch.
  - Fix Windows port that call pl2bat.bat perl utility to create a corrupted
    pgbadger.bat du to the way __DATA__ was read in pgbadger. Thanks to
    Bertrand Bourgier for the patch.
  - Fix begin/end time filter and add regression test for timestamp filters.
    Thanks to Alexis Lahouze and plmayekar for the report.
  - Fix use of uninitialized value in pattern match introduced by pgFormatter
    update. Thanks to arlt for the report.

2021-02-18 - v11.5

This release of pgBadger fix some issues reported by users since past three months as well as some improvements:

• Add report about sessions idle time, computed using: "total sessions time - total queries time / number of sessions This require that log_connection and log disconnection have been enabled and that log_min_duration_statement = 0 (all queries logged) to have a reliable value. This can help to know how much idle time is lost, and if a pooler transaction mode would be useful. This report is available in the "Sessions" tab of "Global Stats" and in the "Sessions" tab of "General Activity" reports (per hour).
• Add anonymization of numeric values, replaced by 4 random digits.
• Update SQL beautifier based on pgFormatter 5.0.

Here is the complete list of changes and acknowledgments:

• Fix parsing of cloudsql multi-line statement. Thanks to Jon Young for the report.
• Add regression test for anonymization.
• Fix anonymization broken by maxlength truncate. Thanks to artl for the report.
• Add anonymization of parameter in time consuming prepare and bind reports. Thanks to arlt for the report.
• Add support to microseconds in logplex log line prefix. Thanks to Ross Gardiner for the report.
• Add report about sessions idle time. Thanks to Guillaume Lelarge for the feature request.
• Complete patch to support multi-line in jsonlog format.

2020-11-24 - v11.4

This release of pgBadger fix some issues reported by users since past four months. Improve support for PostgreSQL 13 log information and adds some new features:

• Add full autovacuum information in "Vacuums per table" report for buffer usage (hits, missed, dirtied), skipped due to pins, skipped frozen and WAL usage (records, full page images, bytes). In report "Tuples removed per table" additional autovacuum information are tuples remaining, tuples not yet removable and pages remaining. These information are only available on the "Table" tab.
• Add new repartition report about checkpoint starting causes.
• Add detection of application name from connection authorized traces.

Here is the complete list of changes and acknowledgments:

  - Fix typo in an error message. Thanks to Vidar Tyldum for the patch.
  - Fix Windows port with error: "can not load incompatible binary data". Thanks to Eric Brawner for the report.
  - Fix typo on option --html-outdir in pgbadger usage and documentation. Thanks to Vidar Tyldum for the patch.
  - Fix autodetection of jsonlog/cloudsql format. Thanks to Jon Young for the report.
  - Fix CSV log parsing with PG v13. Thanks to Kanwei Li for the report and Kaarel Moppel for the patch. 
  - Fix sort of queries generating the most temporary files report. Thanks to Sebastien Lardiere for the report.
  - Add pgbadger version trace in debug mode.