To crawl Sina Weibo data (text, JPG/GIF images, live photos, and videos). [简单实用] 爬取新浪微博用户所发表或转发的文本、JPG/GIF图片、实况照片和视频。
This is a Python implementation of crawling Weibo data (e.g., text, images, live photos, and videos) of a Sina Weibo user from the Weibo Mobile Client. It simulates user login by a session (username and password).
Many thanks to Python Chinese Community for providing the source code SourceCode_weibocrawler.py
.
Crawling short text in original and retweeted Weibo posts.
Crawling large (preferred) or small JPG/GIF images in original and retweeted Weibo posts.
Crawling live photos (as JPG images, MOV videos, and/or GIF images) in original and retweeted Weibo posts.
Crawling HD (preferred) or SD videos in original and retweeted Weibo posts.
This code has been tested on Ubuntu 16.04 operating system. For Windows and macOS, the format of folder path should be changed accordingly.
Set S_DATA
and S_HEADER
of the session for simulating Sina Weibo user login (see comments for details).
Set USER_URL
of the target Weibo user (see comments for details).
Set the amount of pages (PAGE_AMOUNT
) for crawling (see comments for details).
Set the path (PATH_FOLDER
) and the TXT file (PATH_FILE_TXT
) for saving Weibo data.
Set the type of Weibo data (IF_IMAGE
, IF_PHOTO
, and IF_VIDEO
as 1
) for crawling.
Set IF_LIVE2GIF = True
if live photos (videos) need to be converted to GIF images.
Set TIME_DELAY
of the crawler to avoid ConnectionError 104: ('Connection aborted.')
.
If ConnectionError 104: ('Connection aborted.')
occurs:
Set IF_RECONNECT = True
for running the crawler in reconnection mode.
Set TAG_STARTCARD
as the serial number of the stopping Weibo post (according to log information).
Re-run run_WeiboCrawler.py
to continue to crawl Weibo data.
Set IF_RECONNECT = False
if run the crawler in normal mode!
Run run_WeiboCrawler.py
to crawl Weibo data of the target Sina Weibo user.
See Log_run_WeiboCrawler.txt
for log information of running the code.
The Weibo data will be saved in the pre-specified folder (e.g., Demo_WeiboData/
).
The text of Weibo posts will be saved in the TXT file (e.g., Demo_WeiboData/Demo_WeiboPost_Records.txt
).
The images, live photos, and videos will be saved in sub-folders (e.g., Demo_WeiboData/1/
, Demo_WeiboData/1_livephoto/
, and Demo_WeiboData/1_video/
).
Please report an issue if you have any question about this repository, I will respond ASAP.
Please star this repository if you found its content useful. Thank you very much. ^_^