Skip to content

Instantly share code, notes, and snippets.

@prog893
Last active April 10, 2018 07:08
Show Gist options
  • Save prog893/2d0debe3ffd84d0710b18d2b1b19f94a to your computer and use it in GitHub Desktop.
Save prog893/2d0debe3ffd84d0710b18d2b1b19f94a to your computer and use it in GitHub Desktop.

Revisions

  1. Torgayev Tamirlan revised this gist Apr 10, 2018. 2 changed files with 2 additions and 4 deletions.
    1 change: 0 additions & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,6 @@ Inspired by: https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-p
    Ignores ecs-agent and dd-agent when counting running containers. You can add more in `containers_running` in the script below.

    ## Prerequisites
    - awk
    - jq
    - awscli
    - run as root
    5 changes: 2 additions & 3 deletions ecs_instance_scalein_protector.sh
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    #!/bin/bash

    # inspired by https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations
    # needs awk, jq, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection
    # needs jq, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection

    echo "--------------------------------"
    echo "[$(date)] Starting script "
    @@ -18,11 +18,10 @@ region=${az:0:${#az} - 1}
    asg_name=$(aws autoscaling describe-auto-scaling-instances --region $region --instance-ids $instance_id | jq .AutoScalingInstances[].AutoScalingGroupName -r)

    ## number of containers running (except ecs-agent and dd-agent)
    containers_running=$(docker ps | awk '{print $2}' | tail -n+2 | grep -v "amazon/amazon-ecs-agent" | grep -v "datadog/docker-dd-agent" | wc -l)
    containers_running=$(docker ps --format "{{.Image}}" | grep -v "amazon/amazon-ecs-agent" | grep -v "datadog/docker-dd-agent" | wc -l)

    ## current protection state
    scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ${region} --instance-ids ${instance_id} | jq '.AutoScalingInstances[].ProtectedFromScaleIn' -r)
    # scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ap-northeast-1 --instance-ids i-0d24aa0f9d1b5558f --output text | awk '{print tolower($8)}')

    if [ ${containers_running} -ge 1 ]; then
    if [ ${scale_protection} == "true" ]; then
  2. Torgayev Tamirlan revised this gist Apr 10, 2018. 2 changed files with 2 additions and 1 deletion.
    1 change: 1 addition & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -7,6 +7,7 @@ Ignores ecs-agent and dd-agent when counting running containers. You can add mor

    ## Prerequisites
    - awk
    - jq
    - awscli
    - run as root
    - cron (opt.)
    2 changes: 1 addition & 1 deletion ecs_instance_scalein_protector.sh
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    #!/bin/bash

    # inspired by https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations
    # needs awk, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection
    # needs awk, jq, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection

    echo "--------------------------------"
    echo "[$(date)] Starting script "
  3. Torgayev Tamirlan created this gist Mar 6, 2018.
    20 changes: 20 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,20 @@
    # ECS Container Instance scale-in protector

    Protect container instance with containers running from scale-in. Uses aws-cli [set-instance-protection](https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/as-instance-termination.html#instance-protection).
    Inspired by: https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations

    Ignores ecs-agent and dd-agent when counting running containers. You can add more in `containers_running` in the script below.

    ## Prerequisites
    - awk
    - awscli
    - run as root
    - cron (opt.)
    - Permissions for EC2 IAM role
    - autoscaling:DescribeAutoScalingInstances
    - autoscaling:SetInstanceProtection

    ## Example crontab
    ```
    */5 * * * * root /usr/local/bin/scale_in_protection.sh
    ```
    47 changes: 47 additions & 0 deletions ecs_instance_scalein_protector.sh
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,47 @@
    #!/bin/bash

    # inspired by https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations
    # needs awk, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection

    echo "--------------------------------"
    echo "[$(date)] Starting script "

    set -x
    ## self EC2 instance ID
    instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

    ## self region
    az=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
    region=${az:0:${#az} - 1}

    ## self ASG ID
    asg_name=$(aws autoscaling describe-auto-scaling-instances --region $region --instance-ids $instance_id | jq .AutoScalingInstances[].AutoScalingGroupName -r)

    ## number of containers running (except ecs-agent and dd-agent)
    containers_running=$(docker ps | awk '{print $2}' | tail -n+2 | grep -v "amazon/amazon-ecs-agent" | grep -v "datadog/docker-dd-agent" | wc -l)

    ## current protection state
    scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ${region} --instance-ids ${instance_id} | jq '.AutoScalingInstances[].ProtectedFromScaleIn' -r)
    # scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ap-northeast-1 --instance-ids i-0d24aa0f9d1b5558f --output text | awk '{print tolower($8)}')

    if [ ${containers_running} -ge 1 ]; then
    if [ ${scale_protection} == "true" ]; then
    echo "Containers are running and scale protection is on. Doing nothing."
    else
    aws autoscaling set-instance-protection --region ${region} --instance-ids ${instance_id} --auto-scaling-group-name ${asg_name} --protected-from-scale-in
    echo "Containers are running and scale protection is off. Enabling scale-in protection"
    fi
    elif [ ${containers_running} == 0 ]; then
    if [ ${scale_protection} == "true" ]; then
    echo "No containers running and scale protection is on. Disabling scale-in protection."
    aws autoscaling set-instance-protection --region ${region} --instance-ids ${instance_id} --auto-scaling-group-name ${asg_name} --no-protected-from-scale-in
    else
    echo "No containers running and scale protection is off. Doing nothing."
    fi
    fi

    set +x

    echo "[$(date)] Script finished running successfully."
    echo "--------------------------------"
    echo