Created
March 12, 2020 18:13
-
-
Save alexamies/b24f99cbeaf2cd50b2642bc182a91aa8 to your computer and use it in GitHub Desktop.
Revisions
-
alexpamies created this gist
Mar 12, 2020 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,51 @@ # How to debug an unresponsive App Engine Flex app You can debug an unhealthy app using Google Cloud Logging and other GCP tools even if the app has become unresponsive. A test app is provided that simulates the problem in an experiment for the purpose of explaining the debugging process with App Engine Flex. Prerequisites: 1. GCP Project with billing enabled 2. Google Cloud SDK installed The app randomly stalls on incoming requests by going to sleep. Follow these steps to deploy and run the app. Set your project as the default in gcloud ```shell GOOGLE_CLOUD_PROJECT=[Your project] gcloud config set project $GOOGLE_CLOUD_PROJECT ``` Enable App Engine ```shell gcloud app create ``` Deploy the app ```shell gcloud app deploy ``` Browse to the app ```shell gcloud app browse ``` Set up Apache Bench In the Cloud Shell or Linux command line ```shell sudo apt-get install apache2-utils ``` Send some requests to your app ```shell ab -n 1000 -c 5 https://${GOOGLE_CLOUD_PROJECT}.appspot.com/ ``` This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,3 @@ # How to debug an unresponsive App Engine Flex app See the README.md for details on how to run this This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,18 @@ runtime: python env: flex entrypoint: gunicorn -b :$PORT main:app runtime_config: python_version: 3 manual_scaling: instances: 1 resources: cpu: 1 memory_gb: 0.5 disk_size_gb: 10 readiness_check: path: "/readiness_check" check_interval_sec: 5 timeout_sec: 4 failure_threshold: 2 success_threshold: 2 app_start_timeout_sec: 300 This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,75 @@ # Copyright 2020 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Simulate an unhealthy app for demonstration of troubleshooting""" import logging import random import sys import time from flask import Flask, g app = Flask(__name__) def get_app_healthy() -> bool: """Get the application health""" if 'app_healthy' not in g: g.app_healthy = True return g.app_healthy def set_app_healthy(value: bool): """Set the application health""" if 'app_healthy' not in g: g.app_healthy = value g.app_healthy = value @app.route('/') def home(): """1 out of 20 requests change the app to an unhealthy state""" msg = 'Welcome to the home page!\n' if not get_app_healthy(): msg = 'Changing to an unhealthy state' logging.error(msg) #sys.exit(1) # exit the process time.sleep(90) # seconds return msg r = random.uniform(0.0, 1.0) if r > 0.95: msg = 'App is unhealthy' logging.error(msg) set_app_healthy(False) time.sleep(90) # seconds return msg @app.route('/readiness_check') def readiness_check(): #Respond to the load balancer's readiness check msg = 'OK\n' r = random.uniform(0.0, 1.0) if r > 0.9: msg = 'App is refuses to start' logging.error(msg) sys.exit(1) return msg if __name__ == '__main__': logging.basicConfig(level=logging.INFO) logging.info('Starting up') app.run(host='127.0.0.1', port=8080) This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,2 @@ Flask==1.1.1 gunicorn==20.0.4