Simple robots.txt template. Keep unwanted robots out (disallow). White l...
🤖 A curated list of websites that restrict access to AI Agents, AI craw...
Parser for robots.txt for node.js
Generator robots.txt for node js
Dark Web Informationgathering Footprinting Scanner and Recon Tool Relea...
ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze web...
A pure-Python robots.txt parser with support for modern conventions.
List of useful links, tools and resources
Java sitemap generator. This library generates a web sitemap, can ping G...
A webpack plugin to generate a robots.txt file
🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creat...