Skip to content

Instantly share code, notes, and snippets.

@czarneckid

czarneckid/Zero downtime deploys with gunicorn.markdown

Last active August 31, 2016 17:42
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save czarneckid/e657f2e2c8059b9a1395 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save czarneckid/e657f2e2c8059b9a1395 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. czarneckid renamed this gist Jul 31, 2014. 1 changed file with 0 additions and 0 deletions.
    0 Notes.markdown → Zero downtime deploys with gunicorn.markdown
    View file
    File renamed without changes.
  2. czarneckid created this gist Jul 31, 2014.
    8 changes: 8 additions & 0 deletions Notes.markdown
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,8 @@
    ## Zero downtime deploys with gunicorn

    Below are the actual files we use in one of our latest applications at [Agora Games](http://www.agoragames.com) to achieve zero downtime deploys with gunicorn. I hope these files and notes help. I am happy to update these files or these notes if there are comments/questions. YMMV (of course).

    Salient points for each file:

    * `gunicorn.py`: The `pre_fork` function looks for gunicorn's old PID file in the proper file and sends the proper QUIT signal to the old process once the new process is running.
    * `sv-gunicorn-run.jinja`: This is the runit template we use in our Salt-managed infrastructure for handling the application process management. You could just as easily convert this to a non-templatized version.
    88 changes: 88 additions & 0 deletions gunicorn.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,88 @@
    import multiprocessing
    import os, os.path
    import signal

    # The Access log file to write to. [None]
    accesslog='{}/logs/unicorn-access.log'.format(os.getcwd())

    # The Access log format . [%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"]
    # access_log_format='"%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'

    # Logging config file.
    # logconfig="/opt/example/log.conf"

    # The Error log file to write to. [-]
    errorlog='{}/logs/unicorn-error.log'.format(os.getcwd())

    # The granularity of Error log outputs. [info]
    loglevel='debug'

    # The logger you want to use to log events in gunicorn. [simple]
    # logger_class='simple'

    # A base to use with setproctitle for process naming. [None]
    proc_name='your-app-name'

    # Load application code before the worker processes are forked. [False]
    preload_app=True

    # forked. [False]
    daemon=False
    # daemon=True

    # A filename to use for the PID file. [None]
    # pidfile='/var/run/example.pid'
    pidfile='{}/pids/unicorn.pid'.format(os.getcwd())

    # Switch worker processes to run as this user. [0]
    # user="example"

    # Switch worker process to run as this group. [0]
    # group="example"

    # A bit mask for the file mode on files written by Gunicorn. [0]
    umask=0002

    # The socket to bind. [127.0.0.1:8000]
    bind='0.0.0.0:8000'
    # bind='unix:{}/sockets/unicorn.sock'.format(os.getcwd())

    # The maximum number of pending connections. [2048]
    # - Amazon Linux default=1024 ($ sysctl net.ipv4.tcp_max_syn_backlog)
    backlog=2048

    # The number of worker process for handling requests. [1]
    workers=multiprocessing.cpu_count() * 2 + 1
    # workers=1

    # The type of workers to use. [sync]
    worker_class='gevent'

    # The maximum number of simultaneous clients. [1000]
    worker_connections=4098

    # The maximum number of requests a worker will process before restarting. [0]
    max_requests=4098

    # Workers silent for more than this many seconds are killed and restarted. [30]
    timeout=120

    #The number of seconds to wait for requests on a Keep-Alive connection. [2]
    keepalive=2

    # Timeout for graceful workers restart.
    graceful_timeout=30

    def pre_fork(server, worker):
    old_pid_file = '{}/pids/unicorn.pid.oldbin'.format(os.getcwd())

    if os.path.isfile(old_pid_file):
    with open(old_pid_file, 'r') as pid_contents:
    try:
    old_pid = int(pid_contents.read())
    if old_pid != server.pid:
    os.kill(old_pid, signal.SIGQUIT)
    except Exception as err:
    pass

    pre_fork=pre_fork
    118 changes: 118 additions & 0 deletions sv-gunicorn-run.jinja
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,118 @@
    #!/bin/bash
    #
    # This file is managed by salt, using the unicorn formula template.
    # Editing this file by hand is highly discouraged!
    #
    exec 2>&1

    #
    # Since unicorn creates a new pid on restart/reload, it needs a little extra
    # love to manage with runit. Instead of managing unicorn directly, we simply
    # trap signal calls to the service and redirect them to unicorn directly.
    #

    RUNIT_PID=$$
    APPLICATION_NAME={{ unicorn_application }}
    APPLICATION_PATH=/var/www/{{ unicorn_application }}/current
    UNICORN_CMD={{ unicorn_command }}
    CUR_PID_FILE=/var/www/{{ unicorn_application }}/shared/pids/unicorn.pid
    OLD_PID_FILE=$CUR_PID_FILE.oldbin

    echo "Runit service restarted (PID: $RUNIT_PID)"

    function is_unicorn_alive {
    set +e
    if [ -n $1 ] && kill -0 $1 >/dev/null 2>&1; then
    echo "yes"
    fi
    set -e
    }

    if [ -e $OLD_PID_FILE ]; then
    OLD_PID=$(cat $OLD_PID_FILE)
    echo "Old master detected (PID: $OLD_PID), waiting for it to quit"
    while [ -n "$(is_unicorn_alive $OLD_PID)" ]; do
    sleep 5
    done
    fi

    if [ -e $CUR_PID_FILE ]; then
    CUR_PID=$(cat $CUR_PID_FILE)
    if [ -n "$(is_unicorn_alive $CUR_PID)" ]; then
    echo "Detected running Unicorn instance (PID: $CUR_PID)"
    RUNNING=true
    fi
    fi

    function start {
    unset ACTION
    if [ $RUNNING ]; then
    restart
    else
    echo 'Starting new unicorn instance'

    RUNASUID=$(getent passwd {{ user }} | cut -d: -f3)
    RUNASGROUPS=$(id -G {{ group }} | tr ' ' ':')
    ulimit -n 65535
    export HOME={{ home }}
    export PATH={{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin:$PATH
    source {{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin/activate

    cd $APPLICATION_PATH

    exec chpst -u :$RUNASUID:$RUNASGROUPS $UNICORN_CMD -c /var/www/{{ unicorn_application }}/current/config/gunicorn.py wsgi:app
    sleep 3
    CUR_PID=$(cat $CUR_PID_FILE)
    fi
    }

    function stop {
    unset ACTION
    echo 'Initializing graceful shutdown'
    kill -QUIT $CUR_PID

    while [ -n "$(is_unicorn_alive $CUR_PID)" ]; do
    echo '.'
    sleep 2
    done

    echo 'Unicorn stopped, exiting Runit process'
    kill -9 $RUNIT_PID
    }

    function restart {
    unset ACTION
    echo "Restart request captured, swapping old master (PID: $CUR_PID) for new master with USR2"
    kill -USR2 $CUR_PID

    sleep 2
    echo 'Restarting Runit service to capture new master PID'
    exit
    }

    function alarm {
    unset ACTION
    echo 'Unicorn process interrupted'
    }

    trap 'ACTION=stop' STOP TERM KILL
    trap 'ACTION=restart' QUIT USR2 INT
    trap 'ACTION=alarm' ALRM

    [ $RUNNING ] || ACTION=start

    if [ $ACTION ]; then
    echo "Performing \"$ACTION\" action and going into sleep mode until new signal captured"
    elif [ $RUNNING ]; then
    echo "Going into sleep mode until new signal captured"
    fi

    if [ $ACTION ] || [ $RUNNING ]; then
    while true; do
    [ "$ACTION" == 'start' ] && start
    [ "$ACTION" == 'stop' ] && stop
    [ "$ACTION" == 'restart' ] && restart
    [ "$ACTION" == 'alarm' ] && alarm
    sleep 2
    done
    fi