A simple proxy pool
一个简单的代理池工具
A simple proxy pool written in go
go build
# 版本 v0.3.3
wget https://github.com/phpgao/proxy_pool/releases/download/v0.3.3/proxy_pool_linux_amd64
chmod a+x proxy_pool_linux_amd64
cp config_example.json config.json
# 修改redis和端口配置
# 感谢ipip.net提供精准的IP数据(已内置)
./proxy_pool
# 打印可设定参数
./proxy_pool_linux_amd64 --help
# 命令行指定配置
./proxy_pool_linux_amd64 -host 8.8.8.8 -port 6379 -auth laogao
# 后台运行
nohup ./proxy_pool_linux_amd64 > /dev/null 2>&1 &
# 统计
curl 127.0.0.1:8088
# 随机
curl 127.0.0.1:8088/random
# 获取列表
curl 127.0.0.1:8088/get
# http
curl http://cip.cc -x 127.0.0.1:8089
# https
curl https://cip.cc -x 127.0.0.1:8089
+-------------------------+
| |
| |
+-------------+ +------------+ +-------------+ | |
| | | | | | | |
| source +------> new proxy +---------> validator +------> |
| +------> +---------> +------> |
| | | | | | | The Pool |
+-------------+ +------------+ +-------------+ | |
| |
+-----------------------------------> |
| +/- score | |
| | |
| | |
| +-------------+-----------+
| |
+-------+------+ +--------------+ |
| | | | |
| <--------+ | cron |
| old proxy <--------+ validator +<-----------------+
| | | |
| | | |
+--------------+ +--------------+
期待大家的测试和反馈!
如果你在使用scrapy,为什么不试试scrapy-random-useragent-pro呢?