Skip to content

Instantly share code, notes, and snippets.

@jordansissel
Created July 11, 2012 07:03
Show Gist options
  • Select an option

  • Save jordansissel/3088552 to your computer and use it in GitHub Desktop.

Select an option

Save jordansissel/3088552 to your computer and use it in GitHub Desktop.

Revisions

  1. jordansissel revised this gist Jul 14, 2012. 1 changed file with 1 addition and 5 deletions.
    6 changes: 1 addition & 5 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -18,14 +18,10 @@ To get to that point, I need a faster feedback cycle as well as strong incentive

    ## Hire me.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth. This will also include research, prototype, and deployment of whatever helps logstash provide more value for you.
    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth.

    Choosing logstash and hiring one or two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.

    What you get out of it: A super passionate engineer improving your logging practices and infrastructure with the goal of providing strong business value from your logs and event data.

    What I get out of it: A tight feedback loop of experience managing logstash in production which leads to improved logstash features and performance.

    ## Join me.

    Not hiring? The logstash community is growing super super fast and has some already amazing members who help out on IRC and the mailing list. Everyone is welcome.
  2. jordansissel revised this gist Jul 14, 2012. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -18,10 +18,14 @@ To get to that point, I need a faster feedback cycle as well as strong incentive

    ## Hire me.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth.
    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth. This will also include research, prototype, and deployment of whatever helps logstash provide more value for you.

    Choosing logstash and hiring one or two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.

    What you get out of it: A super passionate engineer improving your logging practices and infrastructure with the goal of providing strong business value from your logs and event data.

    What I get out of it: A tight feedback loop of experience managing logstash in production which leads to improved logstash features and performance.

    ## Join me.

    Not hiring? The logstash community is growing super super fast and has some already amazing members who help out on IRC and the mailing list. Everyone is welcome.
  3. jordansissel revised this gist Jul 12, 2012. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -41,10 +41,11 @@ This is my rough roadmap for things to improve in logstash. Many (analytics, thr
    * Pick up working on the [vxin](https://github.com/jordansissel/vxin) project again.
    * Port kibana to ruby to help ship it with logstash as well as gain more contributors (simply by logstash already being ruby)

    ## Improve Throughput
    ## Reduce Cost of Operation

    * Improve agent pipeline speeds
    * Find pipeline bottlenecks. Fix.
    * Improve on-disk data sizes when using elasticsearch.

    ## Improve Releases

  4. jordansissel revised this gist Jul 11, 2012. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -32,7 +32,9 @@ Can't join because you're the competition? Better get your shit together, becaus

    Gauntlet thrown, but not for violence. Let's do some awesome!

    # Roadmap
    # Roadmap

    This is my rough roadmap for things to improve in logstash. Many (analytics, throughput, etc) are super important and will directly benefit anyone using logstash.

    ## Search, Analytics, and Visuals

  5. jordansissel revised this gist Jul 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -10,7 +10,7 @@ I'm embarrassed to answer user questions like "How do I optimize elasticsearch's

    It's bullshit, really. That changes now.

    Right now, the only competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).
    Right now, the big competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).

    In 3-5 years, I want logstash to dominate everything in the logging space not simply by cost, but in form and function.

  6. jordansissel revised this gist Jul 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -30,7 +30,7 @@ Not hiring? The logstash community is growing super super fast and has some alre

    Can't join because you're the competition? Better get your shit together, because logstash will be winning the race very soon. It will be far more fun if we push each other, through competition, to solve problems in this area.

    Gauntlet thrown. Let's do some awesome.
    Gauntlet thrown, but not for violence. Let's do some awesome!

    # Roadmap

  7. jordansissel revised this gist Jul 11, 2012. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -76,6 +76,7 @@ Gauntlet thrown. Let's do some awesome.
    * Logstash may be too fat to run in some places. Provide a light-weight log forwarding tool (similar to my syslog-shipper project)
    * Ship RPM and Deb packages
    * Ship upstart, systemd, smf, and sysv init configurations.
    * Puppet modules, cfengine thingies, chef cookbooks, vagrant vms, etc.

    ## Grow the Community

    @@ -88,4 +89,4 @@ Gauntlet thrown. Let's do some awesome.

    ## Research

    * Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
    * Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
  8. jordansissel revised this gist Jul 11, 2012. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -53,6 +53,7 @@ Gauntlet thrown. Let's do some awesome.

    * Run-time API managing.
    * Make the agent monitorable (metrics!)
    * Make external plugins super easy to include.

    ## Improve Documentation

  9. jordansissel revised this gist Jul 11, 2012. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -83,6 +83,7 @@ Gauntlet thrown. Let's do some awesome.
    * Grow active IRC users
    * Grow active mailing list users
    * Grow active ticket users.
    * Build a community culture of hugs and support.

    ## Research

  10. jordansissel revised this gist Jul 11, 2012. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -59,6 +59,11 @@ Gauntlet thrown. Let's do some awesome.
    * Push hard for new cookbook recipes
    * Write more cookbook recipes

    ## Improve Integration

    * Find relevant open source and commercial targets for input, filter, and output integration.
    * Find relevant targets for search/analytics integration.

    ## Improve Agility

    * Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
  11. jordansissel revised this gist Jul 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion readme.md
    Original file line number Diff line number Diff line change
    @@ -37,7 +37,7 @@ Gauntlet thrown. Let's do some awesome.
    ## Search, Analytics, and Visuals

    * Pick up working on the [vxin](https://github.com/jordansissel/vxin) project again.
    * Port kibana to ruby (the author of said project wants this anyway, iirc)
    * Port kibana to ruby to help ship it with logstash as well as gain more contributors (simply by logstash already being ruby)

    ## Improve Throughput

  12. jordansissel revised this gist Jul 11, 2012. 1 changed file with 5 additions and 3 deletions.
    8 changes: 5 additions & 3 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -1,10 +1,12 @@
    # logstash needs full time love.

    I've been working on logstash for about 2 years. All as a hobby. About 70% of the current content (commits, 'git blame' lines, etc) are by me.
    There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.

    I've never used logstash in production. (Embarrassing, I know!)
    I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.

    I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop"
    Unfortunately, I've never used logstash in production. (Embarrassing, I know!)

    I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."

    It's bullshit, really. That changes now.

  13. jordansissel revised this gist Jul 11, 2012. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -69,6 +69,14 @@ Gauntlet thrown. Let's do some awesome.
    * Ship RPM and Deb packages
    * Ship upstart, systemd, smf, and sysv init configurations.

    ## Grow the Community

    * If a newbie has a bad time, it's a bug.
    * If something doesn't work towards expectations, it's a bug - in documentation, code, somewhere.
    * Grow active IRC users
    * Grow active mailing list users
    * Grow active ticket users.

    ## Research

    * Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
  14. jordansissel revised this gist Jul 11, 2012. 1 changed file with 6 additions and 3 deletions.
    9 changes: 6 additions & 3 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -16,9 +16,9 @@ To get to that point, I need a faster feedback cycle as well as strong incentive

    ## Hire me.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it.
    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth.

    In general, choosing logstash and hiring two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.
    Choosing logstash and hiring one or two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.

    ## Join me.

    @@ -28,7 +28,9 @@ Not hiring? The logstash community is growing super super fast and has some alre

    Can't join because you're the competition? Better get your shit together, because logstash will be winning the race very soon. It will be far more fun if we push each other, through competition, to solve problems in this area.

    # Roadmap of Seriously Major Awesome
    Gauntlet thrown. Let's do some awesome.

    # Roadmap

    ## Search, Analytics, and Visuals

    @@ -59,6 +61,7 @@ Can't join because you're the competition? Better get your shit together, becaus

    * Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
    * Interactive multiline filter tool for the same.
    * Find any other things that slow folks down building logstash deployments, and fix them.

    ## Improve Deployment

  15. jordansissel revised this gist Jul 11, 2012. No changes.
  16. jordansissel revised this gist Jul 11, 2012. 1 changed file with 30 additions and 10 deletions.
    40 changes: 30 additions & 10 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -8,44 +8,64 @@ I'm embarrassed to answer user questions like "How do I optimize elasticsearch's

    It's bullshit, really. That changes now.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it.

    Right now, the only competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).

    In 3-5 years, I want logstash to dominate everything in the logging space not simply by cost, but in form and function.

    ## Roadmap of Seriously Major Awesome
    To get to that point, I need a faster feedback cycle as well as strong incentive to improve the project. To that end...

    ## Hire me.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it.

    In general, choosing logstash and hiring two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.

    ## Join me.

    ### Search, Analytics, and Visuals
    Not hiring? The logstash community is growing super super fast and has some already amazing members who help out on IRC and the mailing list. Everyone is welcome.

    ## Compete with me.

    Can't join because you're the competition? Better get your shit together, because logstash will be winning the race very soon. It will be far more fun if we push each other, through competition, to solve problems in this area.

    # Roadmap of Seriously Major Awesome

    ## Search, Analytics, and Visuals

    * Pick up working on the [vxin](https://github.com/jordansissel/vxin) project again.
    * Port kibana to ruby (the author of said project wants this anyway, iirc)

    ### Improve Throughput
    ## Improve Throughput

    * Improve agent pipeline speeds
    * Find pipeline bottlenecks. Fix.

    ### Improve Releases
    ## Improve Releases

    * Make testing easier.
    * Release frequently.

    ### Improve Management
    ## Improve Management

    * Run-time API managing.
    * Make the agent monitorable (metrics!)

    ### Improve Documentation
    ## Improve Documentation

    * Push hard for new cookbook recipes
    * Write more cookbook recipes

    ### Improve Agility
    ## Improve Agility

    * Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
    * Interactive multiline filter tool for the same.

    ### Research
    ## Improve Deployment

    * Logstash may be too fat to run in some places. Provide a light-weight log forwarding tool (similar to my syslog-shipper project)
    * Ship RPM and Deb packages
    * Ship upstart, systemd, smf, and sysv init configurations.

    ## Research

    * Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
  17. jordansissel created this gist Jul 11, 2012.
    51 changes: 51 additions & 0 deletions readme.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,51 @@
    # logstash needs full time love.

    I've been working on logstash for about 2 years. All as a hobby. About 70% of the current content (commits, 'git blame' lines, etc) are by me.

    I've never used logstash in production. (Embarrassing, I know!)

    I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop"

    It's bullshit, really. That changes now.

    I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it.

    Right now, the only competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).

    In 3-5 years, I want logstash to dominate everything in the logging space not simply by cost, but in form and function.

    ## Roadmap of Seriously Major Awesome

    ### Search, Analytics, and Visuals

    * Pick up working on the [vxin](https://github.com/jordansissel/vxin) project again.
    * Port kibana to ruby (the author of said project wants this anyway, iirc)

    ### Improve Throughput

    * Improve agent pipeline speeds
    * Find pipeline bottlenecks. Fix.

    ### Improve Releases

    * Make testing easier.
    * Release frequently.

    ### Improve Management

    * Run-time API managing.
    * Make the agent monitorable (metrics!)

    ### Improve Documentation

    * Push hard for new cookbook recipes
    * Write more cookbook recipes

    ### Improve Agility

    * Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
    * Interactive multiline filter tool for the same.

    ### Research

    * Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.