Created
February 11, 2015 00:27
-
-
Save devton/43dc0d75ff8564252132 to your computer and use it in GitHub Desktop.
crawler.rake
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| namespace :crawler do | |
| desc 'Start the crawler on a URL' | |
| task :start, [:url] => [:environment] do |t, args| | |
| Rails.logger.info "starting crawler on --> #{args[:url]}" | |
| links = Crawler::Web.collect_links_from args[:url] | |
| CrawledUrl.transaction do | |
| links.each do |url| | |
| CrawledUrl.persist_from url | |
| end | |
| end | |
| end | |
| end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment