Skip to content

Instantly share code, notes, and snippets.

@alexamies
Created March 12, 2020 18:13
Show Gist options
  • Save alexamies/b24f99cbeaf2cd50b2642bc182a91aa8 to your computer and use it in GitHub Desktop.
Save alexamies/b24f99cbeaf2cd50b2642bc182a91aa8 to your computer and use it in GitHub Desktop.

Revisions

  1. @alexpamies alexpamies created this gist Mar 12, 2020.
    51 changes: 51 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,51 @@
    # How to debug an unresponsive App Engine Flex app

    You can debug an unhealthy app using Google Cloud Logging and other GCP tools
    even if the app has become unresponsive. A test app is provided that simulates
    the problem in an experiment for the purpose of explaining the debugging process
    with App Engine Flex.

    Prerequisites:
    1. GCP Project with billing enabled
    2. Google Cloud SDK installed

    The app randomly stalls on incoming requests by going to sleep.

    Follow these steps to deploy and run the app.

    Set your project as the default in gcloud

    ```shell
    GOOGLE_CLOUD_PROJECT=[Your project]
    gcloud config set project $GOOGLE_CLOUD_PROJECT
    ```

    Enable App Engine

    ```shell
    gcloud app create
    ```

    Deploy the app

    ```shell
    gcloud app deploy
    ```

    Browse to the app

    ```shell
    gcloud app browse
    ```

    Set up Apache Bench In the Cloud Shell or Linux command line

    ```shell
    sudo apt-get install apache2-utils
    ```

    Send some requests to your app

    ```shell
    ab -n 1000 -c 5 https://${GOOGLE_CLOUD_PROJECT}.appspot.com/
    ```
    3 changes: 3 additions & 0 deletions an_unhealthy_app_demo.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    # How to debug an unresponsive App Engine Flex app

    See the README.md for details on how to run this
    18 changes: 18 additions & 0 deletions app.yaml
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,18 @@
    runtime: python
    env: flex
    entrypoint: gunicorn -b :$PORT main:app
    runtime_config:
    python_version: 3
    manual_scaling:
    instances: 1
    resources:
    cpu: 1
    memory_gb: 0.5
    disk_size_gb: 10
    readiness_check:
    path: "/readiness_check"
    check_interval_sec: 5
    timeout_sec: 4
    failure_threshold: 2
    success_threshold: 2
    app_start_timeout_sec: 300
    75 changes: 75 additions & 0 deletions main.py
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,75 @@
    # Copyright 2020 Google Inc. All Rights Reserved.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    """Simulate an unhealthy app for demonstration of troubleshooting"""

    import logging
    import random
    import sys
    import time
    from flask import Flask, g


    app = Flask(__name__)


    def get_app_healthy() -> bool:
    """Get the application health"""
    if 'app_healthy' not in g:
    g.app_healthy = True
    return g.app_healthy


    def set_app_healthy(value: bool):
    """Set the application health"""
    if 'app_healthy' not in g:
    g.app_healthy = value
    g.app_healthy = value


    @app.route('/')
    def home():
    """1 out of 20 requests change the app to an unhealthy state"""
    msg = 'Welcome to the home page!\n'
    if not get_app_healthy():
    msg = 'Changing to an unhealthy state'
    logging.error(msg)
    #sys.exit(1) # exit the process
    time.sleep(90) # seconds
    return msg
    r = random.uniform(0.0, 1.0)
    if r > 0.95:
    msg = 'App is unhealthy'
    logging.error(msg)
    set_app_healthy(False)
    time.sleep(90) # seconds
    return msg


    @app.route('/readiness_check')
    def readiness_check():
    #Respond to the load balancer's readiness check
    msg = 'OK\n'
    r = random.uniform(0.0, 1.0)
    if r > 0.9:
    msg = 'App is refuses to start'
    logging.error(msg)
    sys.exit(1)
    return msg


    if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    logging.info('Starting up')
    app.run(host='127.0.0.1', port=8080)
    2 changes: 2 additions & 0 deletions requirements.txt
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,2 @@
    Flask==1.1.1
    gunicorn==20.0.4