Best 2 Robots Exclusion Standard Open Source Projects

Robots Parser

NodeJS robots.txt parser with support for wildcard (*) matching.

Jonasjacek Robots.txt

Simple robots.txt template. Keep unwanted robots out (disallow). White l...