🕸 Generates and delivers RSS feeds via HTTP. Docker image available! Create your own feeds or get started quickly with the included configs.
This web application scrapes websites to build and deliver RSS 2.0 feeds.
Features:
The functionality of scraping websites and building the RSS is provided by the Ruby gem html2rss
.
This application should be used with Docker. It is designed to require as little maintenance as possible. See Versioning and Releases and consider automatic updates.
docker run -p 3000:3000 gilcreator/html2rss-web
and open http://127.0.0.1:3000/ in your browser and click onto the example feed link.
This is the quickest way to get started. However, it's also the one with least flexibility: it doesn't allow to use custom feed configs and doesn't update automatically.
If you wish more flexibility and automatic updates sounds good to you, read on how to get started with docker-compose...
Create a docker-compose.yml
and paste the following into it:
version: "3"
services:
html2rss-web:
image: gilcreator/html2rss-web
ports:
- "3000:3000"
volumes:
- type: bind
source: ./feeds.yml
target: /app/config/feeds.yml
read_only: true
environment:
- RACK_ENV=production
- HEALTH_CHECK_USERNAME=health
- HEALTH_CHECK_PASSWORD=please-set-YOUR-OWN-veeeeeery-l0ng-aNd-h4rd-to-gue55-Passw0rd!
watchtower:
image: containrrr/watchtower
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- "~/.docker/config.json:/config.json"
command: --cleanup --interval 7200
Start it up per: docker-compose up
.
If you did not create your feeds.yml
yet, download this feeds.yml
as blueprint into the directory containing the docker-compose.yml
.
The watchtower service automatically pulls running docker images and checks for updates. If an update is available, it will automatically start the updated image with the same configuration as the running one. Please read its manual.
The docker-compose.yml
above contains a service description for watchtower.
html2rss-web comes with many feed configs out of the box. See file list of all configs.
To use a config from there, build the URL like this:
Build the URL of the feed config you'd like to use like this:
lib/html2rss/configs/ |
domainname.tld/whatever.yml |
Would becomes this URL: | |
http://localhost:3000/ |
domainname.tld/whatever.rss |
^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
To build your own RSS feed, you need to create a feed config.
That feed config goes into the file feeds.yml
.
Check out the example
feed config.
Please refer to html2rss' README for a description of the feed config and its options. html2rss-web is just a small web application which depends on html2rss.
This web application is distributed in a rolling release fashion from the master
branch.
For the latest commit passing the Github CI/CD on the master branch, an updated Docker image will be pushed to Docker Hub: gilcreator/html2rss-web
.
Github's @dependabot is enabled for dependency updates and are automatically merged to the master
branch when the CI gives the green light.
If you use Docker, you should update to the latest image automatically, by setting up watchtower as described.
This app is published on Docker Hub and therefore easy to use with Docker.
The above docker-compose.yml
is a good starting point.
If you're going to host a public instance, please please please:
/health_check.txt
endpoint.Name | Description |
---|---|
PORT |
default: 3000 |
RACK_ENV |
default: 'development' |
RACK_TIMEOUT_SERVICE_TIMEOUT |
default: 15 |
WEB_CONCURRENCY |
default: 2 |
WEB_MAX_THREADS |
default: 5 |
HEALTH_CHECK_USERNAME |
default: auto generated on start |
HEALTH_CHECK_PASSWORD |
default: auto generated on start |
GET /health_check.txt
It is recommended to setup a monitoring of the /health_check.txt
endpoint. With that, you can find out when one of your own configs break. The endpoint uses HTTP Basic authentication.
First, set username and password via these environment variables: HEALTH_CHECK_USERNAME
and HEALTH_CHECK_PASSWORD
. If these are not set, html2rss-web will generate a new random username and password on each start.
An authenticated GET /health_check.txt
request will be responded with:
success
.To get notified when one of your configs breaks, setup a monitoring of this endpoint.
UptimeRobot's free plan is sufficent for basic monitoring (every 5 minutes). Create a monitor of type Keyword with this information and make it aware of your username and password:
Checkout the git repository and ...
This approach allows you to play around without installing Ruby on your machine. All you need to do is install and run Docker.
# Build image from Dockerfile and name/tag it as html2rss-web:
docker build -t html2rss-web -f Dockerfile .
# Run the image and name it html2rss-web-dev:
docker run \
--detach \
--mount type=bind,source=$(pwd)/config,target=/app/config \
--name html2rss-web-dev \
html2rss-web
# Open a interactive TTY with the shell `sh`:
docker exec -ti html2rss-web-dev sh
# Stop and cleanup container
docker stop html2rss-web-dev
docker rm html2rss-web-dev
# Remove the image
docker rmi html2rss-web
If you're comfortable with installing Ruby directly on your machine, follow these instructions:
>= 3.2
gem install bundler foreman
bundle
foreman start
html2rss-web now listens on port 5000 for requests.
Contributions are welcome!
Open a pull request with your changes, open an issue, or join discussions on html2rss.