Web Scrapers in Ruby Tools used Ruby Mechanize (Crawler and Parser) github MongoDB (Database) github Mongo Ruby Driver github Parallel (Multi threading, Multi Processing) github Typhoeus (Concurrent HTTP requests) github Delayed Job (ASync Background Job) github Samples Plain simple web scraper using mechanize Storing result in mongo Progress tracking with mongo Multi threading with parallel Concurrent HTTP requests with typhoeus Background Job with delayed_job