Skip to content

Instantly share code, notes, and snippets.

@alexamies

alexamies/README.md

Created March 12, 2020 18:13
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save alexamies/b24f99cbeaf2cd50b2642bc182a91aa8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save alexamies/b24f99cbeaf2cd50b2642bc182a91aa8 to your computer and use it in GitHub Desktop.
Download ZIP
How to debug an unresponsive App Engine Flex app
Raw
README.md

How to debug an unresponsive App Engine Flex app

You can debug an unhealthy app using Google Cloud Logging and other GCP tools even if the app has become unresponsive. A test app is provided that simulates the problem in an experiment for the purpose of explaining the debugging process with App Engine Flex.

Prerequisites:

  1. GCP Project with billing enabled
  2. Google Cloud SDK installed

The app randomly stalls on incoming requests by going to sleep.

Follow these steps to deploy and run the app.

Set your project as the default in gcloud

GOOGLE_CLOUD_PROJECT=[Your project]
gcloud config set project $GOOGLE_CLOUD_PROJECT

Enable App Engine

gcloud app create

Deploy the app

gcloud app deploy

Browse to the app

gcloud app browse

Set up Apache Bench In the Cloud Shell or Linux command line

sudo apt-get install apache2-utils

Send some requests to your app

ab -n 1000 -c 5 https://${GOOGLE_CLOUD_PROJECT}.appspot.com/
Raw
an_unhealthy_app_demo.md

How to debug an unresponsive App Engine Flex app

See the README.md for details on how to run this

Raw
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
readiness_check:
path: "/readiness_check"
check_interval_sec: 5
timeout_sec: 4
failure_threshold: 2
success_threshold: 2
app_start_timeout_sec: 300
Raw
main.py
# Copyright 2020 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Simulate an unhealthy app for demonstration of troubleshooting"""
import logging
import random
import sys
import time
from flask import Flask, g
app = Flask(__name__)
def get_app_healthy() -> bool:
"""Get the application health"""
if 'app_healthy' not in g:
g.app_healthy = True
return g.app_healthy
def set_app_healthy(value: bool):
"""Set the application health"""
if 'app_healthy' not in g:
g.app_healthy = value
g.app_healthy = value
@app.route('/')
def home():
"""1 out of 20 requests change the app to an unhealthy state"""
msg = 'Welcome to the home page!\n'
if not get_app_healthy():
msg = 'Changing to an unhealthy state'
logging.error(msg)
#sys.exit(1) # exit the process
time.sleep(90) # seconds
return msg
r = random.uniform(0.0, 1.0)
if r > 0.95:
msg = 'App is unhealthy'
logging.error(msg)
set_app_healthy(False)
time.sleep(90) # seconds
return msg
@app.route('/readiness_check')
def readiness_check():
#Respond to the load balancer's readiness check
msg = 'OK\n'
r = random.uniform(0.0, 1.0)
if r > 0.9:
msg = 'App is refuses to start'
logging.error(msg)
sys.exit(1)
return msg
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
logging.info('Starting up')
app.run(host='127.0.0.1', port=8080)
Raw
requirements.txt
Flask==1.1.1
gunicorn==20.0.4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment