pornhubをクロールしているWebHubBotプロジェクトの模倣、効率が遅すぎる、方法を探しています
pornhubをクロールしているWebHubBotプロジェクトの模倣、効率が遅すぎる、方法を探しています
Disclaimer: This project is intended to study the Scrapy Spider Framework and the MongoDB database, it cannot be used for commercial or other personal intentions. If used improperly, it will be the individuals bear.
Language: Python3.6
Environment: ubuntu, 4G RAM
Database: MongoDB
pip install -r requirements.txt
The table in the database that holds the data is PhRes. The following is a field description:
video_title: The title of the video, and as a unique.
link_url: Video jump to Website`s link
image_url: Video cover link
video_duration: The length of the video, in seconds
quality_480p: Video 480p mp4 download path
改进、未知请告知,邮件,issue,push都可。