Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save kudrin/a00dd0f55baf72933a33 to your computer and use it in GitHub Desktop.
Save kudrin/a00dd0f55baf72933a33 to your computer and use it in GitHub Desktop.

Revisions

  1. @xpepper xpepper revised this gist Apr 28, 2014. 1 changed file with 1 addition and 1 deletion.
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    The goal: Upload several files (virtually big files) to a Rails application without compromising the user experience.

    Architecture
    ## Architecture

    jQuery File Upload + Nginx + Rails (Carrierwave) + Sidekiq

  2. @xpepper xpepper revised this gist Apr 28, 2014. 1 changed file with 1 addition and 1 deletion.
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq
    # Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq

    The goal: Upload several files (virtually big files) to a Rails application without compromising the user experience.

  3. @xpepper xpepper renamed this gist Apr 28, 2014. 1 changed file with 0 additions and 0 deletions.
  4. @xpepper xpepper created this gist Apr 28, 2014.
    122 changes: 122 additions & 0 deletions Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,122 @@
    Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq

    The goal: Upload several files (virtually big files) to a Rails application without compromising the user experience.

    Architecture

    jQuery File Upload + Nginx + Rails (Carrierwave) + Sidekiq

    1) jQuery File Upload (http://blueimp.github.io/jQuery-File-Upload/)

    2) Nginx

    3) Rails (Carrierwave)

    Add an asset_upload_controller.rb with a base upload action:

    class AssetsUploadController < ApplicationController
    def upload
    end
    end

    Let's now modify `routes.rb` to route each request for `assets_upload/upload` to our controller:

    post "assets_upload/upload" => 'assets_upload#upload'

    Once the controller and the route have been created, we can create a factory that will instantiate a new asset processor given the content type.

    module AssetProcessors
    class Factory

    SUPPORTED_CONTENT_TYPES = %w{ audio/mp3 video/mp4 }
    PROCESSORS = {
    'audio/mp3' => AudioAssetProcessor,
    'video/mp4' => VideoAssetProcessor
    }

    def self.supports?(content_type)
    SUPPORTED_CONTENT_TYPES.include?(content_type)
    end

    def self.create(content_type, file_name, tmp_path)
    processor = PROCESSORS.fetch(content_type, NullAssetProcessor)
    processor.new(file_name, tmp_path)
    end

    end
    end

    Where VideoAssetProcessor is:

    module AssetProcessors
    class VideoAssetProcessor < AssetProcessor

    def process
    resource = File.open(src_path)
    video = VideoAsset.create(resource: resource)
    end

    end
    end


    class VideoUploader < CarrierWave::Uploader::Base

    storage :file

    def store_dir
    upload_dir_root = Rails.env.test? ? "test" : "uploads"
    "#{upload_dir_root}/#{model.class.to_s.underscore}/video/#{model.id}"
    end

    def extension_white_list
    %w( mp4 )
    end
    end

    4) Sidekiq

    What is Sidekiq? Sidekiq is a background processing framework for Ruby (in alternative you can pick Delayed::Job or Resque). Sidekiq is easily to be integrated into your Rails application and was build with a particular aim at performances.

    In order to integrate Sidekiq into our Rails application, we have to add it to our Gemfile:

    gem 'sidekiq'

    And install the new dependency:

    bundle install

    Once Sidekiq is integrated, we can start implementing the worker that will carry out the job in background. The first step to take in this sense consists in creating the directory `workers` in our Rails `app` directory. In such folder we will create the file upload_worker.rb with the actual implementation of the job.

    class UploadWorker
    include Sidekiq::Worker

    def perform(content_type, file_name, tmp_path)
    processor = AssetProcessors::Factory.create(content_type, file_name, tmp_path)
    processor.execute
    end
    end

    Where `AssetProcessors::Factory` will instantiate a new asset processor according to file content type calling execute on it; so for example for a file with content type mp4 VideoUploader will be called.

    Let's now update `assets_upload_controller.rb` so that it will use our Sidekiq worker.

    class AssetsUploadController < ApplicationController

    def upload
    assets_params.each do |params|
    UploadWorker.perform_async(params[:content_type], params[:file_name], params[:tmp_path])
    end
    end

    private

    def assets_params
    params.require(:assets)
    end

    end

    Run Sidekiq:

    bundle exec sidekiq