Fast and robust date extraction from web pages, with Python or on the command-line
Find original and updated publication dates of any web page. On the command-line or with Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included. The package is used in production on millions of documents and integrated by multiple libraries.
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
Htmldate operates by sifting through HTML markup and if necessary text elements. It features the following heuristics:
link
and meta
elements) including Open Graph
protocol attributes.abbr
or time
elements and a series of attributes (e.g.
postmetadata
).fast
mode the HTML page is cleaned and precise patterns are
targeted.extensive
mode all potential dates are collected and a
disambiguation algorithm determines the best one.Finally, the output is validated and converted to the chosen format.
1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10)
Python Package | Precision | Recall | Accuracy | F-Score | Time |
---|---|---|---|---|---|
articleDateExtractor 0.20 | 0.803 | 0.734 | 0.622 | 0.767 | 5x |
date_guesser 2.1.4 | 0.781 | 0.600 | 0.514 | 0.679 | 18x |
goose3 3.1.17 | 0.869 | 0.532 | 0.493 | 0.660 | 15x |
htmldate[all] 1.6.0 (fast) | 0.883 | 0.924 | 0.823 | 0.903 | 1x |
htmldate[all] 1.6.0 (extensive) | 0.870 | 0.993 | 0.865 | 0.928 | 1.7x |
newspaper3k 0.2.8 | 0.769 | 0.667 | 0.556 | 0.715 | 15x |
news-please 1.5.35 | 0.801 | 0.768 | 0.645 | 0.784 | 34x |
For the complete results and explanations see evaluation page.
Htmldate is tested on Linux, macOS and Windows systems, it is compatible
with Python 3.6 upwards. It can notably be installed with pip
(pip3
where applicable) from the PyPI package repository:
pip install htmldate
pip install htmldate[speed]
For more details on installation, Python & CLI usage, please refer to the documentation: htmldate.readthedocs.io
This package is distributed under the Apache 2.0 license.
Versions prior to v1.8.0 are under GPLv3+ license.
This project is part of methods to derive information from web documents in order to build text databases for research (chiefly linguistic analysis and natural language processing).
Extracting and pre-processing web texts to meet the exacting standards is a significant challenge. It is often not possible to reliably determine the date of publication or modification using either the URL or the server response. For more information:
@article{barbaresi-2020-htmldate,
title = {{htmldate: A Python package to extract publication dates from web pages}},
author = "Barbaresi, Adrien",
journal = "Journal of Open Source Software",
volume = 5,
number = 51,
pages = 2439,
url = {https://doi.org/10.21105/joss.02439},
publisher = {The Open Journal},
year = 2020,
}
You can contact me via my contact page or GitHub.
Contributions are welcome as well as issues filed on the dedicated page.
Special thanks to the contributors who have submitted features and bugfixes!
Kudos to the following software libraries: