Web Scraping in Python using Scrapy (with multiple examples)

This repository contains the code for the article "Web Scraping in Python using Scrapy (with multiple examples)" here - https://www.analyticsvidhya.com/blog/2017/07/web-scraping-in-python-using-scrapy/

Project Structure

.
+-- ourfirstscraper

|	+-- scrapy.cfg - scrapy configuration file

|	+-- ourfirstscraper
  
|	  +-- settings.py - contains all the settings associated with this project
   
|		  +-- spiders - contains the spiders from the tutorial
   
|			  +-- redditbot.py - Reddit Spider
 
|			  +-- shopclues.py - ShopClues Spider
 
|			  +-- techcrunch.py - Techcrunch Spider
 

Dependencies

• Python 2

• Scrapy1.4.0