Crawler is a simple Web crawler (or spider). Its main goal is to abstract the boring and error-prone code from your codebase and let you focus on crawling the site. Also, it's quite easy to use.