Skip to content

Instantly share code, notes, and snippets.

@ascdso2020

ascdso2020/AWSDevOpsStudyNotes

Forked from ejlp12/AWSDevOpsStudyNotes
Created May 7, 2024 19:06
Show Gist options
  • Save ascdso2020/33abae9da40ec77d420f08b68d7ae4d1 to your computer and use it in GitHub Desktop.
Save ascdso2020/33abae9da40ec77d420f08b68d7ae4d1 to your computer and use it in GitHub Desktop.
Download ZIP
AWS DevOps Engineer Professional Study Notes
Raw
AWSDevOpsStudyNotes
CI & CD:
========
2 core software development processes
CI process of automating regular code commits followed by an automated build and test process designed to highlight intergration issues early.
Additional tooling and functionality provided by Bamboo, CruiseControl, Jenkins, Go and TeamCity etc.
workflow based
CD takes the form of a workflow based process which accepts a tested software build payload from a CI server. Automates the deployment into a working QA, Pre-prod or Prod environment.
AWS CodeDeploy and CodePipeline provide CI/CD services
Elasticbeanstalk and CFN provide functionality which can be utilized by CI/CD servers.
Deployment Types:
=================
1. Single Target Deployment - small dev projects, legacy or non-HA infrastructure; outage occurs in case of failure, testing opportunity is limited.
2. All-at-Once Deployment - deployment happens on multiple targets, requires Orchestration tools, suitable for non critical apps in 5-10 range.
3. Minimum in-service Deployment - keeps min in-service targets and deploy in multiple stages, suitable for large environments, allow automated testing, no downtime
4. Rolling Deployments - x targets per stage, happens in multiple stages, after completion of stage 1, next stage begins, orchestration and health check required, can be least efficient if x is smaller, allow automated testing, no downtime if x is not large to impact application, can be paused, allowing multi-version testing.
5. Blue Green Deployment - Deploy to seperate Green environment, update the code on Green, extra cost due to duplicate env during deployment, Deployment is rapid, cutover and migration is clean(DNS Change), Rollback easy(DNS regression), can be fully automates using CFN etc.
Binary, No Traffic Split, not used to feature test
A/B Testing - distribution traffic between blue/green, allows gradual performance/stability/health analysis, allows new feature testing, rollback is quick, end goal of A/B testing is not migration, Uses Route 53 for DNS resolution, 2 records one pointing A, other pointing B, weighted/round robin.
Bootstraping: process starts with base image (ISO/AMI) and via automation build on it to create more complex object.
Cloud Init, CFN Init (complex bootstraping engine)
AMI (Baking) based approach large number of AMIs even for a small system change. Ex: Quickstart
Bootstraping - every option like OS, patches, dependencies, applications can be tweaked. Ex: Quick launch
Immutable Architecture - Replace infra instead of upgrading or repairing faulty components, treat servers as unchangeable objects, don't diagnose and fix, throw away and re-create, Nothing bootstraped except AMI.
Example: Pets vs Cattles
Container/Docker: contains only application and dependencies, achieves higher density, and improved portability by removing per container Guest OS.
Escape for dependency hell, Consistent progression from DEV->TEST->QA->PROD, Isolation, resource scheduling at micro level, Extreme code portability, Micro-services; Docker Image(Read Only), Docker container (hold everything required to run), Layers/UFS (combines layers using UFS), Dockerfile, Docker Daemon/Engine, Docker Client (Interface between user and Docker Engine), Docker Registry/Hub (Private/Public store for Docker images)
CloudFormation:
================
Building block service to provision infra within AWS, Elastic Beanstalk uses CFN, JSON format, Stack (CFN unit of grouping infra), Template (JSON doc giving instructions for CFN on how to act and what to create/update), Stack Policy (IAM style policy governs what/who can change, can be added via CLI or UI, updated but can not be removed)
Create CFN template -> Add template to CFN -> Create CFNStack -> Resources (200 per template) -> Update template/Stack -> Delete Stack
Template Anatomy: Parameters (pass variables into a template), Mappings (allow processing of hash's by CFN template), Resources, Outputs (results from template); Only Resources is mandatory. CFN can run scripts within instances, expand files within instances, stack id(unique).
Use cases:
1. Template for manual deployment of bespoke infra.
. create repeatable patterned environment
3. run automated testing for CI/CD environments
4. define an environment once, deployed
o5. manage infra config using software development style versioning and testing concepts.
IMPORTANT: Template should be designed to work 1 or 1000 app in one or more regions.
CloudFormation Structure:
=========================
CloudWatch:
============
Metric gathering service, Monitoring/alerting service, A graphing service
CMetrics data will be available only for 2 weeks. Anything over 2 weeks should be stored outside.
Namespaces - Ex: EBS, EC2, Check CloudWatch Developer Guide for all Namespaces. Enable detailed monitoring for additional metrics under Namespaces.
Metrics can be aggregated across Autoscaling Services.
Custom Metrics:
Create an IAM role -> CloudWatch -> EC2 -> CloudWatch Full Access
Create EC2 instance with CloudWatch Role -> SSH -> Install Python, Pip, Git and AWS CLI -> git clone https://github.com/ACloudGuru/resources; cd resources;
cat time.sh; chmod a+x time.sh -> setup a cron job (crontab -e) */1 * * * * /home/ubuntu/resources/time.sh -> save -> CloudWatch Min granualrity 1 min ->
Check the CloudWatch for new namespace with Unixtime metrics
CloudWatch Alarms:
Initiate actions (SNS notification) based on parameters specified against metrics available. Alarm period equal to metric frequency (5 min or 1 min).
Alarm can't invoke actions unless state change. Alarms actions must be in same region as alarm. AWS resource wont send data under certain
conditions (ex: EBS not attached to EC2 wont send data to CloudWatch)
Alarm States: OK (metric match thresold) ALARM (metric outside thresold) INSUFFICIENT_DATA (metric not available/not enough)
Limits: 5000 alarms per AWS account, create or update alarm using mon-put-metric-alarm, enable/disable using mon-[enable/disable]-alarm,
Describe alarm using mon-describe-alarm. Alarm can be created before metrics created.
date +%s -> Get Unix Time
AWS Alarm can be created using CLI aws cloudwatch put-metric-alarm.
Autoscale based on CloudWatch Metrics: CPU/Network/Memory too high/low.
Attach EC2 Full access policy to CloudWatch IAM role.
Using aws autoscaling CLI, create autoscaling launch config and Auto Scaling group.
Use aws autoscaling put-scaling-policy to scale in and out.
Use aws cloudwatch put-metric-alarm to create an alarm for Add capacity using Scale out ARN.
Use aws cloudwatch put-metric-alarm to create an alarm for Remove capacity using Scale in ARN.
sudo apt-get install stress -> to stress CPU
stress --cpu 2 --timeout 600
CloudWatch Logs: Monitor your existing system, application and custom logs in real time.
Send your existing logs to CloudWatch; Create patterns to look for in your logs; Alert based on finding of these patterns.
Free agents for Ubuntu, Amazon Linux, Windows.
Main Purpose
1. Monitor logs from EC2 instances in realtime. (track number of errors in application logs and send notification if exceed thresold)
2. Monitor AWS CloudTrail logged events (API Activity such as manual EC2 instance termination)
3. Archive log data (change log retention setting to automatically delete)
Log events - A record sent to CloudWatch Logs to be stored. Timestamp and Message.
Log Streams - this is a sequence of log events that share the same resource (Apache access logs, automatically deleted after 2 months).
Log Groups - group of log stream that share same retention, monitoring and access control settings.
Metric Filters - define how a service would extract metric observations from events and turn them into data points for a CloudWatch metric.
Retention Settings - how long events are kept; Expired logs are deleted automatically.
Download the logs agent (cloudwatchlogs-commands.txt) and run the command.
Log Group Retention can be set to 1 day to 10 years.
CloudWatch Log Filters: filter log data pushed to CloudWatch; won't work on existing log data, only work after log filter created, only returns
first 50 results. Metric contains 1. Filter Pattern 2. Metric Name 3. Metric NameSpace 4. Metric value
Modify rsyslog (/etc/rsyslog.d/50-default.conf) and remove auth on line number 9, sudo service rsyslog restart
while [ 1 ] ; do ssh xx.xx.xx.xx ; sleep 2 ; done
Real-Time Log processing: Subscription Filters for RTLP - Kinesis Streams, Lambda, Kinesis Firehouse
Use aws kinesis command to create/describe stream, get stream ARN. Update permissions.json with stream and role ARN.
Run aws logs put-subscription-filter to start real time log processing.
Check GitHub: awslabs/cloudwatch-logs-subscription-consumer ==> one click deploy for ELK;
CloudTrail:
service records all AWS API calls on account and delivers in a log. Log contain identity of whom made the API call, time of call, source
ip of call, request parameters, response elements returned by AWS service.
Purpose - to enable security analysis, track changes to your account, provide compliance auditing
Types - All regions and one region (logs delivered to S3 bucket and stored using SSE, logs delivered in 15 mins, new logs files every 5 min)
CloudWatch Events: similar to CloudTrail but faster, central nervous system of AWS. Components 1. Events - Created when State Change or API call or
Own code generated application level events. 2. Rules - match incoming events and route them for processing, not ordered. 3. Targets -
where events are processed. i) Lambda functions ii) Kinesis streams iii) SNS topics iv) built-in targets
Go to AWS Lambda from console and copy cloudwatchevents.txt in configure function.
Go to CloudWatch, Create rule -> EC2 state change notification(shutting down), Lambda Function(), Configure rule details -> Create rule
Delegation & Federation:
=======================
Identity Federation - Own IdP - IAM, allow users in other AWS accounts access to resources - Delegation,
allows users from external IdP - Federation.
2 Types of Federation -
1. Corporate/Enterprise Identity Federation (LADP,AD,SAML,AWS Directory Service)
2. Web Identity Federation (Amazon, Facebook, Google, Twitter, OpenID Connect) allow app or access to your AWS resources.
ROLES: object which contains 2 policy documents. TRUST Policy (who granted - ARN) ACCESS policy (what entity - Action)
SESSIONS: set of temporary credentials; access and secret key with expiration; obtained by STS;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment