基于httpx的一个大型项目 ,爬取黑胶唱片网站 Discogs
python3 -m pip install poetry
poetry init
poetry install
poetry run python main.py
Attention: Step1 must generate task queue first, otherwise step2 will go wrong.
Similarly, step3 must wait for step2 to generate a task queue. The call between files can be optimized according to actual business conditions.(Where step1, step2, step3 are aliases of functions)
Using some of the new features of python, you can learn some new usages, such as: typing, fstring, pydantic, dataclass How to develop asynchronous programming, the scheduling relationship between codes,the use of decorators, etc.
Of course, the level is limited, and there are many areas that need to be optimized. Welcome to discuss with me.