Grab Versions Save

Fixed

+- Fix #346: spider does not process initial_urls +- Fix #344: raise GrabInvalidUrl for pycurl error 3

Added

• Add pylint to default test set

Fixed

• Fix #229: using deprecated response object inside Grab

Removed

• Remove spider project template and start_project script

Added

• Add --spider-transport and --grab-transport options to crawl script
• Add SOCKS5 proxy support in urllib3 transport

Fixed

• Fix #237: urllib3 transport fails without pycurl installed
• Fix bug: incorrect spider request logging when cache is enabled
• Fix bug: crawl script fails while trying to process a lock key
• Fix bug: urllib3 transport fails while trying to throw GrabConnectionError exception
• Fix bug: Spider add_task method fails while trying to log invalid URL error

Removed

• Remove obsoleted hammer_mode and hammer_timeout config options

Fixed

• Fix "error:None" in spider rps logging
• Fix race condition bug in task generator

Added

• Add original_exc attribute to GrabNetworkError (and subclasses) that points to original exception

Changed

• Remove IOError from the ancestors of GrabNetworkError
• Add default values to --spider-transport and --grab-transport options of crawl script

Fixed

• Fix bug: task generator works incorrectly
• Fix bug: pypi package misses http api html file
• Fix bug: dictionary changed size during iteration in stat logging
• Fix bug: multiple errors in urllib3 transport and threaded network service
• Fix short names of errors in stat logging
• Improve error handling in urrllib3 transport
• Fix #299: multi-added errors
• Fix bug: pypi package misses http api html file
• Fix #285: pyquery extension parses html incorrectly
• Fix #267: normalize handling of too many redirect error
• Fix #268: fix processing of utf cookies
• Fix #241: form_fields() fails on some HTML forms
• Fix normalize_unicode issue in debug post method
• Fix #323: urllib3 transport fails with UnicodeError on some invalid URLs
• Fix #31: support for multivalue form inputs
• Fix #328, fix #67: remove hard link between document and grab
• Fix #284: option headers affects content of common_headers
• Fix #293: processing non-latin chars in Location header
• Fix #324: refactor response header processing

Changed

• Refactor Spider into set of async. services
• Add certifi dependency into grab[full] setup target
• Fix #315: use psycopg2-binary package for postgres cache
• Related to #206: do not use connection_reuse=False for proxy connections in spider

Removed

• Remove cache timeout option
• Remove structured extension