Skip to content

Instantly share code, notes, and snippets.

@excalq

excalq/rails.rb

Last active September 18, 2018 13:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save excalq/0c5023fce8af90089040daa6404525f2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save excalq/0c5023fce8af90089040daa6404525f2 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. excalq renamed this gist Dec 24, 2016. 1 changed file with 0 additions and 0 deletions.
    0 rails → rails.rb
    View file
    File renamed without changes.
  2. excalq renamed this gist Dec 24, 2016. 1 changed file with 0 additions and 0 deletions.
    0 rails.grok.md → rails
    View file
    File renamed without changes.
  3. excalq created this gist Dec 24, 2016.
    42 changes: 42 additions & 0 deletions rails.grok.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,42 @@
    ####################################
    ### Rais3 Log Grok (Unicorn and Thin supported at :info level)
    ## There are many non-capturing groups. For debugging, change them to named groups and use http://grokconstructor.appspot.com/do/match
    #
    ## Assumes Rails is prepending :uuid to log lines, with one log per unicorn worker.
    ## Assumes the whole request is sent multiline (by Filebeat, not Logstash mulitline plugin)
    #
    ## Add this to the Filebeat.yml prospector
    ## multiline:
    # pattern: '\[[a-fA-F0-9]{32}\]( Started |$)'
    ## negate: true
    ## match: after
    #
    # To use this, save this gist to /etc/logstash/patterns.d/rails, and add this to your Logstash config:
    # grok {
    # patterns_dir => ["/etc/logstash/patterns.d/rails"]
    # match => { "message" => "%{RAILS3}" }
    # }
    ####################################

    # Fragment patterns
    RUUID \[\h{32}\]
    RTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE}
    RAILSCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
    RAILSEXCLASS (?<exception>\w+((?:\:\:)?\w+)?)
    RAILSPROFILE %{NUMBER:duration}ms(?: \((?:Views: %{NUMBER:view}ms \| )?ActiveRecord: %{NUMBER:db}ms\))?

    # Whole-line/multiline patterns (faster to parse)
    RPARAMETERS (?:^%{RUUID} Parameters: {%{DATA:params}}$\n)?
    # In case rack_timer gem is used.
    RAILSRACK (^(?:%{RUUID} )?(?:Rack Timer|source=rack-timeout).*?$\n)+
    RAILSEXCEPTION (?:^%{RUUID}\s*\n%{RAILSEXCLASS} (- (?<error_message>.*)|\((?<message>.*)\))\:\n\s+(?<backtrace>(?m)%{GREEDYDATA})\n{2,3})
    RAILSOTHER (?:^(?!(?:%{RUUID} )?Completed %{NUMBER}).*?$\n)
    RAILSLOGCONTENT ^(?:(%{RAILSEXCEPTION}|%{RAILSRACK}|%{RAILSOTHER})*)

    # Highest level patterns
    RAILSHEAD ^%{RUUID} Started %{WORD:verb} "%{URIPATHPARAM:path}" for %{IPORHOST:remote_ip} at %{RTIMESTAMP:timestamp}$\n
    RAILSPROCESSING ^%{RUUID} Processing by %{RAILSCONTROLLER} as %{WORD:format}$\n
    RAILSFOOT ^%{RUUID} Completed %{NUMBER:status}%{DATA} in %{RAILSPROFILE}%{DATA}$\n

    # Finally, use this as the master pattern which you'll grok in your Logstash conf
    RAILS3 \A%{RAILSHEAD}%{RAILSPROCESSING}?%{RPARAMETERS}?%{RAILSLOGCONTENT}?%{RAILSFOOT}?%{RAILSRACK}?