基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器...
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
An R web crawler and scraper
A list of AI agents and robots to block.
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers ...
Powerful dork searcher and vulnerability scanner for windows platform
Proxy List Scrapper
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC co...
Simple robots.txt template. Keep unwanted robots out (disallow). White l...
Vietnamese text data crawler scripts for various sites (including Youtub...
hproxy - Asynchronous IP proxy pool, aims to make getting proxy as conve...
Sneakpeek is a framework that helps to quickly and conviniently develop ...
Serritor is an open source web crawler framework built upon Selenium and...