Go Spider Versions Save

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

V1.2

9 years ago
  • Add header and cookies
  • Add url tag name
  • Add example "github_login_profile_page_processor" for using header and cookies
  • Add function AddRequest, AddRequests, GetByRequest, GetAllByRequest in spider and AddTargetRequestWithParams, AddTargetRequestsWithParams in Page
  • Expose moduler Request

V1.1.1

9 years ago
  1. Change capacity Scheduler to infinite

2.Scheduler alse can remove duplicate url

V1.1

9 years ago
  • Add new responce type "jsonp", "text";
  • Change the charset to utf-8 when the charset of responce data is not utf-8;
  • Add Header and Cookies to Page
  • Fix some implicit bugs

v1.0

9 years ago

1.0 release