Web scraper project to collect the useful Hong Kong weather data from HKO website
hk0weather is an open source web scraper project using Scrapy to collect the useful weather data from Hong Kong Observatory website.
Scrapy can output collected weather data into the machine-readable formats (eg. CSV, JSON, XML).
Cloning and setup hk0weather in a Py3 virtual environment
$ git clone https://github.com/sammyfung/hk0weather.git
$ cd hk0weather
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
Activate the Py3 virtual environment once before the first running of web spiders.
$ source venv/bin/activate
$ cd hk0weather
Optionally, list all available spiders.
$ scrapy list
Run a regional weather data web crawler and export data to a JSON file.
$ scrapy crawl regional -o regional.json