Skip to content

Instantly share code, notes, and snippets.

@erwanlr
Last active December 14, 2015 14:18
Show Gist options
  • Save erwanlr/5099652 to your computer and use it in GitHub Desktop.
Save erwanlr/5099652 to your computer and use it in GitHub Desktop.

Revisions

  1. erwanlr revised this gist May 17, 2013. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions Gemfile
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,6 @@
    source "http://rubygems.org"

    gem 'typhoeus', '>=0.6.2'
    #gem 'typhoeus', :git => 'git://github.com/typhoeus/typhoeus.git'
    gem 'webmock', '>=1.9.3'
    gem 'rspec', :require => 'spec'
  2. erwanlr revised this gist May 17, 2013. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion test_spec.rb
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,7 @@

    context "when concurrency = #{concurrency}" do
    it 'returns all the files' do
    foo.retrieve_files(concurrency).should == foo.urls
    foo.retrieve_files(concurrency).sort.should == foo.urls.sort
    end
    end
    end
  3. erwanlr revised this gist Mar 6, 2013. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions test.rb
    Original file line number Diff line number Diff line change
    @@ -40,3 +40,7 @@ def self.files
    %w{file1.php file2.php file3.php file4.php}
    end
    end

    # uncomment this for real case
    # foo = Foo.new('http://google.com')
    # p foo.retrieve_files
  4. erwanlr created this gist Mar 6, 2013.
    5 changes: 5 additions & 0 deletions Gemfile
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,5 @@
    source "http://rubygems.org"

    gem 'typhoeus', '>=0.6.2'
    gem 'webmock', '>=1.9.3'
    gem 'rspec', :require => 'spec'
    4 changes: 4 additions & 0 deletions spec_helper.rb
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,4 @@
    # spec/spec_helper.rb

    require 'webmock/rspec'
    require File.expand_path(File.dirname(__FILE__) + '/../test')
    42 changes: 42 additions & 0 deletions test.rb
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,42 @@
    require "rubygems"
    require "bundler/setup"
    require 'typhoeus'

    class Foo

    attr_reader :target_url

    def initialize(target_url)
    @target_url = target_url
    end

    def retrieve_files(max_concurrency = 2)
    hydra = Typhoeus::Hydra.new(:max_concurrency => max_concurrency)
    found = []

    urls.each do |url|
    request = Typhoeus::Request.new(url)

    request.on_complete do |response|
    found << response.request.base_url
    end

    hydra.queue(request)
    end

    hydra.run
    found
    end

    def urls
    urls = []

    Foo.files.each {|file| urls << "#@target_url/#{file}"}

    urls
    end

    def self.files
    %w{file1.php file2.php file3.php file4.php}
    end
    end
    21 changes: 21 additions & 0 deletions test_spec.rb
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,21 @@
    # spec/test_spec.rb

    require 'spec_helper'

    describe Foo do
    let(:url) { 'http://google.com' }
    subject(:foo) { Foo.new(url) }

    before { stub_request(:get, /.*/).to_return(status: 200) }

    describe '#retrieve_files' do
    (1..4).each do |concurrency|

    context "when concurrency = #{concurrency}" do
    it 'returns all the files' do
    foo.retrieve_files(concurrency).should == foo.urls
    end
    end
    end
    end
    end