python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .
东方财富、天天基金股票基金证券数据爬虫、数据分析可视化
git clone https://github.com/minicloudsky/EastMoney.git
pip3 install virtualenv
virtualenv venv
source venv/bin/activate
vim backend/eastmoney/settings.py
DATABASES = {
"default": {
'ENGINE': 'django.db.backends.mysql',
'HOST': 'your database host',
'PORT': 3306,
'USER': 'your database user',
'PASSWORD': 'your database password',
'NAME': 'your database name'
}
}
vim backend/config.ini
[CRAWL_MODE]
; 全量爬取,默认为全量爬取
crawl_mode = "ALL"
; 增量爬取
;crawl_mode = "APPEND"
pip install -r requirements.txt
python manage.py makemigrations
python manage.py migrate
python3 manage.py collectstatic
nohup
放后台执行,具体可以查看 backend/bin
下面脚本cd backend
sh bin/start_server.sh
sql
或者设置过滤聚合条件就好了metabase
docker 部署docker run -d -it -p 3000:3000 --name metabase metabase/metabase
.这里我们把metabase部署在 3000端口