| require "async" | |
| CONCURRENCY = 1000 | |
| ITERATIONS = 100 | |
| def work | |
| Async do |task| | |
| CONCURRENCY.times do | |
| task.async do | |
| sleep 1 |
FWIW: I (@rondy) am not the creator of the content shared here, which is an excerpt from Edmond Lau's book. I simply copied and pasted it from another location and saved it as a personal note, before it gained popularity on news.ycombinator.com. Unfortunately, I cannot recall the exact origin of the original source, nor was I able to find the author's name, so I am can't provide the appropriate credits.
| require 'net/ftp' | |
| def scan(ftp, dir) | |
| ftp.chdir(dir) | |
| puts ftp.pwd + "/." | |
| entries = ftp.list('*') | |
| entries.each do |entry| | |
| if entry.split(/\s+/)[0][0,1] == "d" then | |
| scan(ftp, entry.split.last) | |
| else |
First, add pry-rails to your Gemfile:
https://github.com/rweng/pry-rails
gem 'pry-rails', group: :developmentThen you'll want to rebuild your Docker container to install the gems
| # Intention to dump from live redis. Use rdb-tools to dump from a file. | |
| # To know the usage run: >> ruby redis_debug.rb | |
| require 'redis' # needs redis gem to work | |
| require 'csv' | |
| class RedisDebug | |
| DEFAULT_OPTS = { redis: {host: 'localhost', port: '6379', db: 0, timeout: 100}, | |
| batch_size: 2000, |
| class ParseEntries | |
| def process(message) | |
| entry = JSON.parse(message.value) | |
| [entry] | |
| end | |
| end | |
| class CombineEntries | |
| def initialize | |
| @open_transactions = {} |
When developing a program in Ruby, you may sometimes encounter a memory leak. For a while now, Ruby has a facility to gather information about what objects are laying around: ObjectSpace.
There are several approaches one can take to debug a leak. This discusses a time-based approach, where a full memory dump is generated every, say, 5 minutes, during a time that the memory leak is showing up. Afterwards, one can look at all the objects, and find out which ones are staying around, causing the
| def split_into_chunks(string, chunk_length=2): | |
| chunks = [] | |
| while len(string) > 0: | |
| chunks.append(string[:chunk_length]) | |
| string = string[chunk_length:] | |
| return chunks | |
| def to_oracle_raw16(string, strip_dashes=True, dashify_result=False): | |
| oracle_format_indices = [3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15] | |
| if strip_dashes: |
My largest Sidekiq application had a memory leak and I was able to find and fix it in just few hours spent on analyzing Ruby's heap. In this post I'll show my profiling setup.
As you might know Ruby 2.1 introduced a few great changes to ObjectSpace, so now it's much easier to find a line of code that is allocating too many objects. Here is great post explaining how it's working.
I was too lazy to set up some seeding and run it locally, so I checked that test suite passes when profiling is enabled and pushed debugging to production. Production environment also suited me better since my jobs data can't be fully random generated.
So, in order to profile your worker, add this to your Sidekiq configuration:
if ENV["PROFILE"]
| module Storage | |
| module Models | |
| class Brand < ActiveRecord::Base; end | |
| end | |
| end |