Answer by Eric Wu:
If you're looking for a python based crawler, Scrapy is probably your best bet.
However, there are simpler python alternatives & other open source scrapers in Java, Ruby, and PHP. I've curated a list of various open source web scrapers on this Quora question:
For simple scraping without much programming effort, you might want to check out these solutions:
- uBot (http://ubotstudio.com/) is a web scraper with a fairly robust GUI interface. It's extremely versatile, but there is a learning curve.Cost = $245
- Google Spreadsheets … I know, right?! Here's an article detailing how you can use Excel to extract the data you might want using the "importxml" function and xpath (http://j.mp/gFy4p0). Cost = Free