Is there a better crawler than Scrapy?

Answer by Eric Wu:

If you're looking for a python based crawler, Scrapy is probably your best bet.

However, there are simpler python alternatives & other open source scrapers in Java, Ruby, and PHP. I've curated a list of various open source web scrapers on this Quora question: What are some good free web scrapers / scraping techniques?

For simple scraping without much programming effort, you might want to check out these solutions:

  • uBot (http://ubotstudio.com/) is a web scraper with a fairly robust GUI interface. It's extremely versatile, but there is a learning curve.Cost = $245
  • Google Spreadsheets … I know, right?! Here's an article detailing how you can use Excel to extract the data you might want using the "importxml" function and xpath (http://j.mp/gFy4p0). Cost = Free

If you need a hosted solution, I'd highly recommend 80legs (http://www.80legs.com/) or Mozenda (http://www.mozenda.com/)

Is there a better crawler than Scrapy?

Advertisements

One thought on “Is there a better crawler than Scrapy?

  1. Aw, this was an extremely good post. Taking a few minues annd actuaal effort to generate a top notch article… but
    what can I say… I put things off a whole llot and never manage to get nearly
    anything done.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s