Forked from wildahunden/2013-Austin-DevOpsDays-Notes-BothDays
Created
October 8, 2016 17:56
-
-
Save andragabr/b4581565fa2db9e9bcb034a66ae574dd to your computer and use it in GitHub Desktop.
2013 Austin DevOps Days Notes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 30 April 2013 | |
| https://gist.github.com/anonymous/5491171 | |
| DevOps Days | |
| Keynote Speaker -- John Willis | |
| (CAMS) | |
| Culture | |
| Continuous deployment needs a culture change | |
| Culture wins war of recruitment | |
| Command & Control | |
| Culture Coaches | |
| Slogan Based | |
| Incentive Alignment | |
| Corporate values | |
| Enron example - integrity | |
| Chad Dickonson ETSY CEO | |
| Drucker | |
| Don't change corporate culture - use it | |
| Figure out what it is and work with it | |
| Nurture it | |
| Simon Sinek "Start with Why" Golden Circle Concept | |
| Why get involved and work for a company | |
| Why people buy product - when matches why people work for company - magic | |
| W Edward Deming, "The New Economics" | |
| System of Profound Knowledge | |
| system thinking | |
| knowledge of psychology | |
| theory of knowledge | |
| Louis V Gerstner, Jr, "Who says Elephants can't dance?" | |
| "Culture isn't just one aspect of the game -- it is the game. | |
| Case studies | |
| netflix | |
| etsy | |
| github social coding | |
| "Culture is the edge - how you win the hiring war." | |
| Netflix | |
| Don't apologize for being tough | |
| Github | |
| Culture of collaboration | |
| no managers | |
| structure is not structure | |
| Etsy | |
| code as craft (artisan) | |
| enable people to make a living making things | |
| blameless culture | |
| tools to reinforce your culture | |
| benfit corporation | |
| LiveRebel deployment automation | |
| UC4 - application release engineering | |
| DYN - Pete Cheslock | |
| How to retain best employees | |
| Recruiting - Sisyphus | |
| doomed to pushing rock uphill repeatedly | |
| 2 weeks notice | |
| 2 weeks into to company | |
| 3 moths ramp-up | |
| four months total | |
| probably like 6 months | |
| "The Phoenix Project" book. Constraints | |
| "The Goal" | |
| "An hour lost on the bottleneck is an hour lost on the entire system." | |
| "An hour gained on a non-bottleneck is a mirage." | |
| "Common sense is not so common" Voltaire | |
| Monetary adjustments = short term fixes (at bat) | |
| STPR - Stuff to Pay Ratio | |
| What drives us: | |
| Autonomy | |
| Mastery | |
| Purpose | |
| Don Pink | |
| Align work with an owner | |
| Back off and let them work on it | |
| Offer support as needed | |
| Filter unnecessary distractions | |
| Mastery | |
| Continuous Learning | |
| Skill improvement | |
| Purpose | |
| being a part of something greater | |
| fulfilling, important work that matters | |
| sensu replaced nagios | |
| Open source projects | |
| gives people ownership | |
| can make it easier to recruit later | |
| costs can be minimal | |
| benefits can be massive | |
| Unlimited Vacations | |
| you have to have the culture back it up | |
| not a way to avoid burnout | |
| need work/life balance | |
| Remote employees | |
| hipchat | |
| irc | |
| Skype | |
| Conferences | |
| Unscheduled Time | |
| High levels of trust across the organization, personal ownership & responsibility | |
| Challenging Meaningful Work | |
| Employee ownership | |
| High levels of trust | |
| Treat them like adults | |
| Lee Thomson | |
| CTO MorphLabs | |
| Chief Techologist E*Trade | |
| DevOps assessments, training & toolchain implementations for clients | |
| Dev2Ops blog | |
| DevOps Toolchaing | |
| Control | |
| Provisioning | |
| Release | |
| Model | |
| Monitoring | |
| Sources | |
| dto Soluctions slide | |
| Service Delivery Platform | |
| Develop, commit, build, Package, Register Promote, Install Configure | |
| Application | |
| Infrastructure | |
| SDP strategy | |
| Quick Syntax utilization Survey | |
| 45 syntaxes on an E*Trade project | |
| Syntax Proficiency | |
| well advanced in an art, occupation or branch of knowledge | |
| lee's def: ability to understand, test, correct and release someone else's code | |
| Systemic Viewpoint | |
| does your org understand convergence, imperative vs. declarative, topographic sort, etc | |
| if not, do you have the budget and foresight to train…. | |
| This? | |
| biz | |
| Good architecture removes complexity | |
| Bash - rerun | |
| everyone know it | |
| poor modularity | |
| rerun helps make is more modular | |
| mvp | |
| Compatible | |
| puppet, chef, rundeck, ansible, salt stack, many more | |
| Very similar to powershell | |
| To the break…. | |
| nothing wrong with well-tested imperative code | |
| better participation across organizational boundaries | |
| easy to switch between linux/windows | |
| people are part of the solution | |
| mvp | |
| Chris - Hippie Hacker | |
| H eart | |
| I eye seeing each other | |
| mea nui - the big thing | |
| maunga teitei - lofty mountain | |
| te rourou - with your food basket & mine, the people with thrive | |
| programming for people | |
| continuously deliver something meaningful | |
| who, what, why | |
| Mr. Rogers, cpb congress funding for cpb | |
| Look for the helpers | |
| ii.can.cd/ | |
| you.can.cd/ | |
| we.can.cd/pacific_hope | |
| aroha nui -- big live | |
| tena koe i see you | |
| you.can.cd on github | |
| 1 fork | |
| 2 commit change | |
| help.thiis.is | |
| Open Space | |
| starts when it starts | |
| ends when it ends | |
| whoever comes is the right people | |
| whatever happens is the only thing that could have happened | |
| law of 2 feet, law of mobility | |
| Day 2 - Patrick DeBois | |
| Godfather of DevOps | |
| (CAMS) | |
| Culture | |
| Automatation | |
| Measurement | |
| Sharing | |
| Measure Culture | |
| Everyone in the culture wants to remain in the culture. That's how you measure culture success | |
| Stages of societies (primitive -> civilized) | |
| (the real) memes | |
| "an idea behavior or style that spreads from person to person within a culture" | |
| unit of cultural transmission | |
| memeplexes comprise groups of memes that replicate together and coadapt | |
| I. replications | |
| 2. variation | |
| 3. Adapt to changes | |
| http://devopsreactions.tumblr.com | |
| meme map | |
| Unable to trademark "DevOps" because of previous prior art. | |
| TEMES = 3rd replicator - for technology | |
| Viruses | |
| parasites, whose presence lowers the fitness of their host | |
| commensals, whose presence is neutral | |
| mutuals | |
| Encourage diversity | |
| Collaboration | |
| Strategic flexibility | |
| #devops branches are good for diversity | |
| Adapt | |
| DataDog - | |
| UC4 - udeploy | |
| Gene Kim | |
| How do we better sell DevOps | |
| We're the ones who have learned about DevOPs. The rest of the IT world are in state of learned failure. | |
| Act I: IT Ops Fixing Fragile Artifacts | |
| The Product Managers don't understand what technology can/can't do. They broadcast commitments to customers | |
| Act 2: The developers | |
| To make the date we have to ignore deployability | |
| Acrue technical debt | |
| Deploys take longer and longer to complete | |
| We do painful things less frequently | |
| IT Ops and Dev at War | |
| Nothing left for Infosec | |
| compliance work | |
| The downward spiral | |
| Eric Passmore, Former SVP Global Engineering, AOL (2007) | |
| http://www.slideshare.net/realgenekim/how-can-we-better-sell-devops | |
| Operations sees… | |
| Fragile applications prointe to failure | |
| Long time require to figure out thich bit bot flipped | |
| Marketshare goes down | |
| Dev Sees | |
| More urgent, date-driven projects put into the queue | |
| Even more fragile code put into production | |
| More releases have increasingly turbulent installs | |
| Release cycles lengthen to amortize "cost of deployments" | |
| Ever increasing backlog of infrastructure projects that could fix root cause and reduce costs | |
| Value to InfoSec | |
| Integrate security testing into daily dev work | |
| reduce | |
| Desired Future reality | |
| installs are predictable | |
| High performing DevOps Teams | |
| They're more agile | |
| deploy code 30 times o=morerequently | |
| 8000 time faster than peer | |
| They're more reliable | |
| double change succès rate | |
| restore service 12 times faster | |
| How organization achieve high performance | |
| 89% have infrastrue artifacts in version conrol | |
| 82% have automated process to create environments | |
| Orgs that implemented DevOps practes over 12 months ago were 5x more likely to be high performing than organziaatios that were't implementing DevOPs at all. See PuppletLabs survey. | |
| Head of Dev & Head of Ops have shared goals | |
| Frustration for downstream operations caused by upstream decisions. | |
| Goal Persuasion -- use stories for communication | |
| "The Goal," Eliyahu Goldratt | |
| See slide share for links to free pdf at phoenix-project website. | |
| Jez Humble, "Continuous Delivery" | |
| Sponsor shoutouts | |
| HP | |
| Collabnet | |
| Nick Galbreath | |
| Care & Feeding of Large Scale Graphite Installations | |
| http://client9.com/20130501 | |
| Etsy collected over One Billion Points collected Daily | |
| Continuous Deployment | |
| Worked at Etsy | |
| MS Security Development | |
| Positive Hack Days | |
| Strive for fastest way for deployments possible | |
| Graphite? | |
| Making it easy to create, analyze and share data can change your organization | |
| Making a data-driven culture | |
| Empowering developers, operations, qa, security and business to be more confident in the changes they make | |
| Your job is likely invisible to the rest of the organization | |
| invisible things aren't valued | |
| so make what you do visible | |
| …and celebrate your wins | |
| Why Graphite? | |
| Many innovations in eau part of the stack | |
| But, it's th efull stack that really makes it special | |
| On-disk layout to UI and API to…the community around it | |
| Sharing is Caring | |
| allow data to be easily accessed | |
| and easily shared. This makes it different than many monitoring solutions. | |
| It's your own in-house mashup generator | |
| combine operations and business metrics in the same pile | |
| Big Picture | |
| It's a database | |
| Not ACID | |
| w/o db tools | |
| Installation | |
| Don't use the OS default version | |
| newer point release of graphite have significant improvements | |
| Documentation | |
| http://graphite.readthedocs.org | |
| Whisper storage Engines | |
| 12 bytes / point stored | |
| Store 1 minute points for 1 month and 15 minutes for 11 months | |
| Disk Layout | |
| Each metric create a directory tree | |
| makes 4 directories | |
| very branchy directory | |
| MIddleware | |
| carbon-cache | |
| Metrics go in | |
| csv file | |
| TCP | |
| write buffer don't go directly to disk | |
| The cache | |
| once data is written, its out of the cash | |
| reboot causes data loss | |
| New Metrics | |
| Very expensive | |
| MAX_CREATES_PER_MINUTE = 50 | |
| Saves your disk, but new metrics will "pile up" | |
| FALLOCATE | |
| WHISPER_FALLOCATE_CREATE = True | |
| linux kernel >= 2.6.23 | |
| then can bump up max creates per minute | |
| MAX_CACHE_SIZE = inf (no.set it to some limit) | |
| CARBON_METRIC_PREFIX = carbon | |
| another one to turn on | |
| carbon-agrregator | |
| sum or average metrics based on wildcards 7 regexps | |
| helps eliminate | |
| destination | |
| carbon-relay | |
| consistent hashing | |
| or more rule-based routing | |
| StatsD | |
| udp | |
| good for embedding | |
| not so good for batch | |
| can use stated and aggregator together | |
| a different statsd that may be good | |
| https://github.com/armon/statsite | |
| Backups, do you really need them? | |
| naive backup caused graphite performance goes to crap | |
| ionice is better | |
| monitoring | |
| url mistypes cause | |
| kill old metrics - they kill disk | |
| cpu is stable | |
| except for apache usage | |
| consider moving apache to separate machine | |
| Tune apache | |
| unlimited by default. tune it down | |
| /version/ | |
| /metrics/expand/ | |
| /events/ | |
| Granularity | |
| like RRDTool, | |
| rapid spikes can be averaged awash | |
| vertical line technology | |
| easy to make horizontal lines | |
| How to make vertical line | |
| draw as Infinite | |
| PyPy accelerate pything | |
| Ceres storage engine | |
| Mine apache log to see what metrics are being queried | |
| Write Docs | |
| Opscode - chef conference videos will be on youtube | |
| RetailMeNot.com - largest provider of discounts | |
| AppDynamics - Application Performance Monitoring | |
| BazaarVoice - | |
| Google+ devops days photos | |
| Jeff Hackert (Talk) | |
| Exploiting Conway's Law for Underpants and Profit or | |
| The wrong way to be right | |
| Miscommunication, Misalignment happen often and need to be addressed | |
| Ken Schwaber: SCRUM | |
| Noise/Variation | |
| technology | |
| requirements | |
| people - "a project with certain technology and agreed upon functionalyis now categorized a complected rather than simple because of the people vector." | |
| Conway's Law | |
| "Organization which design systems..are constained to produce design which are copies of the communication structures of these organizations." | |
| Contention - | |
| our position in a discussion | |
| heated disagreement | |
| Example of curly braces online with invocation or on line below | |
| "I'm surrounded by morons" Ontological Arogance | |
| Personalize problem on the other side. Naive realism | |
| Assigning motives to others. Not only are you wrong, but I know why you're wrong. You intentions are not good. | |
| Are we or are we not our work? | |
| Creating a table leg that isn't right doesn't mean we are defective. If we discuss an idea we have, we take it personally if the idea is attacked. Disagreement = attack. We are wired to flee in danger. Better to flee than fight. | |
| At work, the best idea wins. We are not all ready for that. Rejection in the present moment is about who I am and can cause difficulty to accept feedback. | |
| Before aware of contention we feel it in our body. Tension, lean forward, hot, a lot of thought action. | |
| Psychology. Sound travels to spine then direct to brain. Sound is fast. | |
| Amigdala becomes very active fast. Neo-cortex function stops functioning. | |
| We let the emotion take over. | |
| Vulnerability - fear of disconnection | |
| Communication is the basis for misunderstanding | |
| Fear | |
| Peopleware "The right to be right is irrelevant; it's only the right to be wrong that makes you free." | |
| Rule of the hole: "When in a hole, stop digging." | |
| We want others to understand. Give to others what we want for ourselves. | |
| 1) My perspective is not the only perspective | |
| 2) Understand that misunderstandings often arise | |
| J. McCarthy "I will never to anything dumb on purpose." | |
| Stop digging is to stop doing something stupid. Stop investing in my "rightness." | |
| Don't label someone because they don't understand my idea. | |
| When I undermine your understanding, take advantage of your sense of unawareness. | |
| Next conflict you need to give the other person understanding. | |
| When you see the emotion of the other person, acknowledge it. | |
| Stop the amigdala hijack. | |
| Understand that you might be wrong. | |
| "As human beings, we are basically awake and we can understand reality. We are not enslaved by sour lives; we are free. Being free…means simply that we have a body a mind, and WE CAN UPLIFT OURSELVES IN ORDER TO WORK WITH REALITH IN A DIGNIFIED AND HUMOUROUS WAY…we can handle our universe properly and fully" CHOGYAM TRUNGPA RINPOCHE | |
| Sponsor minutes | |
| Zenos | |
| unified monitoring solutions through discovery | |
| physical infrastructure | |
| applications (rabbit mq, mysql) | |
| Confio | |
| Ignite db performance analysis focus on response time | |
| Ignites | |
| Karthik Gaekwad (Ignite) | |
| SEXY Dashboards with Twitter Bootstrap | |
| web developers hard to find | |
| n(12)x12 grid | |
| JQuery plugins rolled in | |
| Use an online editor-divsite | |
| use Font awesome for icons | |
| use bootsnipp | |
| Alex Corley (Ignite) | |
| Chef Recipes, Gitflow, Developer/Staging/Production Environments and Chef Server | |
| Gitflow - A successful git branching model | |
| Foodcritic | |
| Isaac Salpeter (Ignite) | |
| Inside S3 Operations, or How (not) to Store Trillions of Objects | |
| Big workload | |
| 200+ tps | |
| Key Naming is key amenable to prefixing | |
| don't have keys with same beginning strings | |
| or add bash 16 has at beginning | |
| distribute loads across key | |
| aws.amazoncom/support | |
| Jesse Gonzalez (Ignite) | |
| NIH syndrome - Pitfall or Paradise | |
| Not Invented Here | |
| Motivation | |
| excuses | |
| doesn't work in our cloud | |
| documentation stinks | |
| language war | |
| Time to Deliver | |
| Never reach MVP | |
| Duplication of effort | |
| Improved features | |
| Open source? | |
| Ownership & Pride | |
| weigh your options | |
| How much value does it really add | |
| support the existing community | |
| learn that new language | |
| improve broken or missing documentation | |
| Neil Matatall (Ignite) | |
| Dig up stupid | |
| Need to get dev, ops & security together | |
| don't need to yell at others "you need to fix this…" | |
| write tools | |
| don't rely on vendors alone | |
| keep it tailored. Write the tools and make them do what you want | |
| code review - static analyze tools | |
| Peco Karayanev (Ignite) | |
| Why does monitoring suck? Because it is rooted in detecting failure, and not preventing it | |
| Riverbed | |
| Wall of contusion | |
| devOPS (that is crappy code) | |
| DEVops (your system such) | |
| MANops (fix it or I will fire you) | |
| Reactiveness = run a test send an alert | |
| More stimulus than you can handle | |
| Generating 30% of our traffic with monitoring | |
| If it is not broken, it may be breaking. | |
| Track close close. | |
| Tracking vs monitoring | |
| wired article on fas preventative fixes | |
| Allow time for your team to do more analysis work | |
| equip them with good data | |
| Victor Trac (Ignite) | |
| 9 AWS Infrastructure Hacks You Should Do | |
| 1) CloudFormation | |
| 3) Iam | |
| 4) Security groups | |
| 8) Internal DNS for network | |
| Route53 for External DNS | |
| 9) Team tagging | |
| Open spaces | |
| devopsdays.rockappearalshop.com | |
| Austin Cloud User Group | |
| Agile Austin DevOps SIG Last Wed each month BazaarVoice | |
| Meetup.com | |
| Dell Crowbar Demo - bare metal provisioning system | |
| Designed to take your naked hardware install o/s, networking, raid, bios, etc | |
| crowbar was born because dell was tired of showing up at a client to install openStack and not have everything and have to delay. ruby/rails for state machine. Chef for configuration management. | |
| Using a bootable cd that installs an o/s | |
| turn on computers, pxi-boot with discover using sledgehammer. Ohai runs to discover new hardware. Displays in crowbar. | |
| sledge-hammer can re-configure changes through the interface with crowbar | |
| devStack - openstack cloud | |
| crowbar | |
| admin | |
| 2 other each with 2 gb |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment