Skip to content

Instantly share code, notes, and snippets.

@devton

devton/crawler.rake

Created February 11, 2015 00:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save devton/43dc0d75ff8564252132 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save devton/43dc0d75ff8564252132 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. devton created this gist Feb 11, 2015.
    12 changes: 12 additions & 0 deletions crawler.rake
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,12 @@
    namespace :crawler do
    desc 'Start the crawler on a URL'
    task :start, [:url] => [:environment] do |t, args|
    Rails.logger.info "starting crawler on --> #{args[:url]}"
    links = Crawler::Web.collect_links_from args[:url]
    CrawledUrl.transaction do
    links.each do |url|
    CrawledUrl.persist_from url
    end
    end
    end
    end