Orchid

Orchid is a free Python based generic multi-threaded web crawler complete with documentation.The Orchid crawler can be used to locate web pages which contained malicious code. However, the logic of what to do with the crawled pages is implemented in a separate class and therefore Orchid can easily be used for any application which requires crawling the web.
Ссылка: http://pypi.pyt..s/source/O/Orchid/orchid.tar.gz