Immo Feed Versions Save

• #27 Write unit tests for helpers
• #19 Add tests for scraper runner and sources
• Improved regex for price and size helpers
• Refactor sources
• Add colours to logs

BC Breaks

• Split HTMLSource and JSONSource into separate files at /scraper/src/types/html-source.ts and /scraper/src/types/json-source.ts. If you have custom sources you'll need to update your imports.

• #25 Add email notifications with Mailgun (see Setting up notifications)
• #26 Add CSV and JSON export endpoint with a download option
• #30 Fix pagination when results are hidden in the UI

• Add tests for scraper sources #18 #17
  • These are run nightly on Travis CI for every source
  • Can also be run locally (see Testing)
• Add handling of multiple pages in the runner #21
  • Now you can specify these new properties in your sources (see Adding a new source):

      // The selector for the next page button
      public nextPageSelector = '.pager-next > a'
  
      // How many pages do you want to scrape
      public pagesToScrape: number = 5
  

• Add pagination in the database, api and frontend #23
• Run the scrapers sequentially to improve stability
• Add some clearer and prettier logging to the runner #1

First version of immo-feed