Continuous scalable web crawler built on top of Flink and crawler-commons
A continuous scalable web crawler built on top of Flink and crawler-commons, with bits of code borrowed from bixo.
The primary goals of flink-crawler are:
See the Key Design Decisions page for more details.