Skip to content

Instantly share code, notes, and snippets.

@RafaelAMello

RafaelAMello/DEPLOYING_AIRFLOW_TURBINE.md

Last active April 14, 2020 00:14
Show Gist options
  • Save RafaelAMello/c7749bc0e7cff2c3278c4729cdb8dec3 to your computer and use it in GitHub Desktop.
Save RafaelAMello/c7749bc0e7cff2c3278c4729cdb8dec3 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. RafaelAMello revised this gist Apr 14, 2020. 1 changed file with 0 additions and 1 deletion.
    1 change: 0 additions & 1 deletion DEPLOYING_AIRFLOW_TURBINE.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,7 +19,6 @@ zip aws-airflow-stack/functions/package.zip aws-airflow-stack/functions/load_met

    aws s3 cp aws-airflow-stack s3://$DEPLOYMENT_BUCKET/quickstart-turbine-airflow --recursive

    # Production
    aws cloudformation create-stack \
    --stack-name airflow-$(python -c "import uuid; print(str(uuid.uuid4()))") \
    --template-url https://$DEPLOYMENT_BUCKET.s3.amazonaws.com/quickstart-turbine-airflow/templates/turbine-master.template \
  2. RafaelAMello revised this gist Apr 14, 2020. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions DEPLOYING_AIRFLOW_TURBINE.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,12 +19,11 @@ zip aws-airflow-stack/functions/package.zip aws-airflow-stack/functions/load_met

    aws s3 cp aws-airflow-stack s3://$DEPLOYMENT_BUCKET/quickstart-turbine-airflow --recursive

    python -c "import random, string; print(''.join(random.choice(string.ascii_letters + string.digits) for i in range(48)))" >> dbpassword.txt

    # Production
    aws cloudformation create-stack \
    --stack-name airflow-$(python -c "import uuid; print(str(uuid.uuid4()))") \
    --template-url https://$DEPLOYMENT_BUCKET.s3.amazonaws.com/quickstart-turbine-airflow/templates/turbine-master.template \
    --parameters ParameterKey=DbMasterPassword,ParameterValue=$(cat dbpassword.txt) ParameterKey=QSS3BucketName,ParameterValue=$DEPLOYMENT_BUCKET \
    --parameters ParameterKey=QSS3BucketName,ParameterValue=$DEPLOYMENT_BUCKET ParameterKey=WorkerInstanceType,ParameterValue=t3.2xlarge ParameterKey=SchedulerInstanceType,ParameterValue=t3.small \
    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

    CREATE_IN_PROGRESS="CREATE_IN_PROGRESS"
    @@ -39,6 +38,7 @@ echo opening http://$AIRFLOW_UI:8080
    open http://$AIRFLOW_UI:8080

    echo "Create a Pipeline here https://$DEPLOYMENT_REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines"
    open https://$DEPLOYMENT_REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines
    ```
    # Create Password User Inside the EC2
    ```bash
  3. RafaelAMello created this gist Mar 12, 2020.
    78 changes: 78 additions & 0 deletions DEPLOYING_AIRFLOW_TURBINE.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,78 @@
    # Deploying Airflow With Turbine Cloud Formation Steps

    Dependencies:
    1. aws cli - https://aws.amazon.com/cli/
    2. jq - https://stedolan.github.io/jq/

    ```bash
    # Change this if you have a different S3 bucket
    export DEPLOYMENT_BUCKET=
    export DEPLOYMENT_REGION=

    aws configure set default.region $DEPLOYMENT_REGION

    aws s3 mb s3://$DEPLOYMENT_BUCKET

    rm -r -f aws-airflow-stack && git clone --recursive [email protected]:villasv/aws-airflow-stack.git

    zip aws-airflow-stack/functions/package.zip aws-airflow-stack/functions/load_metric.py

    aws s3 cp aws-airflow-stack s3://$DEPLOYMENT_BUCKET/quickstart-turbine-airflow --recursive

    python -c "import random, string; print(''.join(random.choice(string.ascii_letters + string.digits) for i in range(48)))" >> dbpassword.txt

    aws cloudformation create-stack \
    --stack-name airflow-$(python -c "import uuid; print(str(uuid.uuid4()))") \
    --template-url https://$DEPLOYMENT_BUCKET.s3.amazonaws.com/quickstart-turbine-airflow/templates/turbine-master.template \
    --parameters ParameterKey=DbMasterPassword,ParameterValue=$(cat dbpassword.txt) ParameterKey=QSS3BucketName,ParameterValue=$DEPLOYMENT_BUCKET \
    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

    CREATE_IN_PROGRESS="CREATE_IN_PROGRESS"
    STACK_PROGRESS="CREATE_IN_PROGRESS"
    printf 'Creating Stack'
    while [ "$STACK_PROGRESS" = "$CREATE_IN_PROGRESS" ]; do printf '.'; sleep 5; STACK_PROGRESS=$(aws cloudformation describe-stacks | jq '.Stacks[] | select(.StackId | test("airflow.")) | select(.ParentId == null and .DeletionTime == null) .StackStatus' -r); done

    export AIRFLOW_UI=$(
    aws ec2 describe-instances --filters Name=tag:Name,Values=turbine-webserver Name=instance-state-code,Values=16 --query 'Reservations[].Instances[].PublicDnsName'| jq '.[0]' -r)
    echo
    echo opening http://$AIRFLOW_UI:8080
    open http://$AIRFLOW_UI:8080

    echo "Create a Pipeline here https://$DEPLOYMENT_REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines"
    ```
    # Create Password User Inside the EC2
    ```bash
    export $(xargs </etc/sysconfig/airflow) && python3
    ```
    ```python
    from airflow import models, settings
    from airflow.contrib.auth.backends.password_auth import PasswordUser
    user = PasswordUser(models.User())
    user.username = 'insights'
    user.email = '[email protected]'
    user.superuser = True
    user.password =
    session = settings.Session()
    session.add(user)
    session.commit()
    session.close()
    exit()
    ```

    # Bringing it down
    ```bash
    export STACKID=$(aws cloudformation describe-stacks | jq '.Stacks[] | select(.StackId | test("airflow.")) | select(.ParentId == null and .DeletionTime == null) .StackId' -r)
    echo shutting down $STACKID
    aws cloudformation delete-stack --stack-name $STACKID && rm dbpassword.txt
    ```

    # Running Commands
    ```bash
    export $(xargs </etc/sysconfig/airflow)
    airflow [COMMAND]
    ```

    # Seeing Logs
    ```bash
    sudo journalctl -u airflow --follow
    ```