Finding a Ruby memory leak using a time analysis
================================================

When developing a program in [Ruby](http://ruby-lang.org), you may sometimes encounter a memory leak.
For a while now, Ruby has a facility to gather information about what objects are laying around:
[ObjectSpace](http://ruby-doc.org/core/ObjectSpace.html).

There are several approaches one can take to debug a leak. This discusses a time-based approach, where
a full memory dump is generated every, say, 5 minutes, during a time that the memory leak is showing up.
Afterwards, one can look at all the objects, and find out which ones are staying around, causing the
memory leak.


Gather
------

Setup your Ruby application to dump all objects to a file. If you have an event loop, something like this would work:

```ruby
require 'objspace'

def heap_dump
  GC.start
  
  i = Time.now.strftime('%s')

  open("/tmp/ruby-heap-#{i}.dump", "w") do |io|
    ObjectSpace.dump_all(output: io)
  end
  
  # On Heroku you'll need to push it elsewhere, like S3
  #s3 = AWS::S3.new(access_key_id: ENV['S3_ACCESS_KEY'], secret_access_key: ENV['S3_SECRET_KEY'])
  #bucket = s3.buckets['qm-import-export']
  #obj = bucket.objects["ruby-heap-#{i}.jsonl"]
  #obj.write(IO.binread(path))
end

ObjectSpace.trace_object_allocations_start
mainloop do
  # assuming your mainloop does the work, and calls this block every 5 minutes
  heap_dump
end
```

Or, if you're having a Rails app, do this in a controller that you visit every 5 minutes

```ruby
# app/controllers/heap_dumps_controller.rb
class HeapDumpsController < ActionController::Metal

  def heap_dump
    if ENV['HEAP_DUMP'] == '1' && params[:token].to_s == ENV['HEAP_DUMP_TOKEN']
      heap_dump
      self.response_body = 'Dumped heap'
    else
      self.status = 401
      self.response_body = 'Invalid token'
    end
  end
end

# add to config/routes.rb
get "/heap_dump", to: HeapDumpsController.action(:heap_dump)

# config/initializers/heap_dump_tracing.rb
if ENV['HEAP_DUMP'] == 1
  require 'objspace'
  ObjectSpace.trace_object_allocations_start
end
```


Install
-------

- Having [Ruby](http://ruby-lang.org/), install the dependencies with `bundle install`.
- Having [PostgreSQL](http://postgresql.org/), create the database with `createdb mem_analysis`.
- When getting dumps from Amazon S3, [s3cmd](https://github.com/s3tools/s3cmd) may come in handy.


Import
------

If stored on S3, get the dump list. Update the bucket and date in the grep command to reflect your case. This stores filenames and dates in _index.txt_.

    S3_URL=s3://qm-import-export/
    s3cmd ls $S3_URL | grep '^2015-11-23' | sed 's/[0-9]*\+\s\+s3:.*\///' >index.txt

Then download them:

    for file in `cat index.txt | awk '{print $3}'`; do s3cmd get $S3_URL/$file $file; done

Initialize the database:

    bundle exec ruby createdb.rb

Because importing can take quite a while, this is split into two steps: converting each file to SQL, and loading all into the database:

    bundle exec ruby gencsv.rb
    sh genimport.sh | psql mem_analysis


Analyse
-------

Now that the database is loaded, we're ready to gather information.
To find out what is causing a memory leak, we can look at graphs plotting memory usage over time in different dimensions.
This is done by `graph.rb`. Let's start with the object type.

    bundle exec ruby graph.rb type-mem

This will create the file _graph-type-mem.png_ showing the total size of objects by type. If there's one thing leaking,
you'll probably have a number of somewhat flat lines, and one with a positive slope, which is the culprit.

Then create a similar graph for that object type only, and plot lines by file, for example. This gives one an idea in which
gem the leaking objects may be created. If it's a string, run

    bundle exec ruby graph.rb string-mem

If it's something else, edit _graph.rb_ and expand the `case`-block. In this way you may be able to zoom in on the cause.


Sample
------

![graph-type-count](https://cloud.githubusercontent.com/assets/503804/11392637/56f47762-935b-11e5-8122-a7bfd16cbec8.png)