Skip to content

Instantly share code, notes, and snippets.

@procrastinatio

procrastinatio/haproxy_rate_limiting.md

Created October 25, 2017 06:04
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save procrastinatio/6b6579230d99be5bfa26d04acd788e7a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save procrastinatio/6b6579230d99be5bfa26d04acd788e7a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. procrastinatio created this gist Oct 25, 2017.
    660 changes: 660 additions & 0 deletions haproxy_rate_limiting.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,660 @@


    Introduction
    ============

    So HAProxy is primalery a load balancer an proxy for TCP and HTTP. But it may act as a traffic
    regulator. It may also be used as a protection against DDoS and service abuse, by maintening a wide variety
    of statistics (IP, URL, cookie) and when abuse is happening, action as denying, redirecting to other backend
    may undertaken ([haproxy ddos config], [haproxy ddos])




    The various HAProxy configuration files are in this repository:
    <https://github.com/procrastinatio/haproxy-docker/tree/more_examples>


    Simple application
    ==================

    Simple `haproxy.cfg` proxying everything on a single backend, _mf-chsdi3.int.bgdi.ch_, only setting a custom header:

    defaults
    mode http
    timeout connect 4000ms
    timeout client 50000ms
    timeout server 50000ms

    stats enable
    stats refresh 5s
    stats show-node
    stats uri /stats/haproxy

    global
    log 127.0.0.1:1514 local0 debug
    user haproxy
    group haproxy

    frontend fe_app
    log global
    log-format "%ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]} %{+Q}r"

    reqadd Referer:\ http://zorba.geo.admin.ch

    bind 0.0.0.0:8080 name http
    default_backend be_app_stable

    backend be_app_stable
    log global
    http-request replace-value Host .* mf-chsdi3.int.bgdi.ch
    server upstream_server mf-chsdi3.int.bgdi.ch:80 maxconn 512



    Source IP based limit
    =====================

    In this example, we modified the config, to track by origin IP for overenthusiastic users.

    https://blog.serverfault.com/2010/08/26/1016491873/

    * New backend `be_429_slow_down` which will respond after 2s with an http code be_429_slow_down
    * If an source IP is making more than 3 requests in a period of 10 seconds, it will be labeld as aubsive
    * User has to stay quiet for 30seconds to be unbanned.

    See also [better rate limiting]

    ~~~~~~~
    frontend fe_app
    [...]
    # table used to store behaviour of source IPs (type is ip)
    stick-table type ip size 200k expire 30s store gpc0,conn_rate(10s),http_req_rate(10s)
    # IPs that have gpc0 > 0 are blocked until the go away for at least 30 seconds
    acl source_is_abuser src_get_gpc0 gt 0
    # Instead of redirecting to slowing down backend, we may also reject any request
    #tcp-request connection reject if source_is_abuser
    # connection rate abuses get blocked (3 requests in 10s, then blocked for
    # 30s)
    acl conn_rate_abuse sc1_conn_rate gt 3
    acl mark_as_abuser sc1_inc_gpc0 ge 0
    tcp-request connection track-sc1 src
    # Same as above, we are nice, we do not reject all request,
    # but these count also as access, so counter is not reset
    #tcp-request connection reject if conn_rate_abuse mark_as_abuser
    reqadd Referer:\ http://zorba.geo.admin.ch
    use_backend be_429_slow_down if conn_rate_abuse mark_as_abuser source_is_abuser
    bind 0.0.0.0:8080 name http
    default_backend be_app_stable
    backend be_429_slow_down
    timeout tarpit 2s
    errorfile 500 /var/local/429.http
    http-request tarpit
    ~~~~~~~
    The important line is:

    stick-table type ip size 200k expire 30s store gpc0,conn_rate(10s),http_req_rate(10s)

    It declares a talbe of type *ip* holding up to _200'000 entries_. Each entry consists of a source *ip* (50 bytes), the *connection rate* and the *http request rate*, both about 12 bytes each.
    Hence the table size will be at maximum 200'000 ⋅ 74 bytes, 14.7 Mbytes of storage for this table (in memory).
    An entry is deleted after 30 seconds if not added or updated. As a rule of a thumb, it should be at least twice the longest rate argument, for a smooth average.
    The argument for connection rate and http request rate are how long to calculate the average over (10 seconds).

    Tesing with one request every 4 seconds:

    for i in {0..10}; do sleep 4; curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:8080/rest/services/height?easting=600000&northing=200000" | ts ; done

    Oct 15 19:00:23 200
    Oct 15 19:00:27 200
    Oct 15 19:00:31 200
    Oct 15 19:00:35 200
    Oct 15 19:00:39 200
    Oct 15 19:00:43 200
    Oct 15 19:00:47 200
    Oct 15 19:00:51 200
    Oct 15 19:00:55 200
    Oct 15 19:00:59 200
    Oct 15 19:01:03 200

    With with one request every second in two parallel threads, the fourth and all subsequent requests are discard

    seq 30 | parallel -n0 -j2 "sleep 1; curl -s -o \/dev\/null -w \"%{http_code}\n\" 'http://localhost:8080/rest/services/height?easting=600000&northing=200000' | ts"
    Oct 15 21:28:54 200
    Oct 15 21:28:54 200
    Oct 15 21:28:55 200
    Oct 15 21:28:57 429
    Oct 15 21:28:58 429
    Oct 15 21:29:00 429
    Oct 15 21:29:01 429
    Oct 15 21:29:03 429
    Oct 15 21:29:04 429
    Oct 15 21:29:06 429
    Oct 15 21:29:07 429
    Oct 15 21:29:09 429
    Oct 15 21:29:10 429
    Oct 15 21:29:12 429




    Entrée log:

    2017/10/11 12:21:01 <local0,info> 172.17.0.1:51974 [11/Oct/2017:12:21:01.485] fe_app blocked-proxy-ua/<NOSRV> 0/-1/-1/-1/0 503 444 - - SC-- 0/0/0/0/0 0/0 {-,"",""} "HEAD / HTTP/1.1"
    2017/10/11 12:24:53 <local0,info> 172.17.0.1:51978 [11/Oct/2017:12:24:53.497] fe_app be_app_stable/upstream_server 0/0/1/3/4 200 511 - - ---- 1/1/0/1/0 0/0 {-,"",""} "HEAD / HTTP/1.1"



    seq 100 | parallel -n0 -j4 "curl -s -o \/dev\/null -w \"%{http_code}\n\" 'curl -X POST --data @profile.json 'http://localhost:8080/rest/services/profile.json'"

    Same goes for all requests:

    ~~~~~~~
    seq 500 | parallel -n0 -j4 "curl -s -o \/dev\/null -w \"%{http_code}\n\" -X
    POST --data @profile.json 'http://localhost:8080/rest/services/profile.json'"
    seq 500 | parallel -n0 -j12 "curl -s -o \/dev\/null -w
    \"%{http_code}\n\" -X POST --data @profile.json
    'http://localhost:8080/rest/services/profile.json'"
    ~~~~~~~


    Backend rate limit
    ==================

    Another way to limit the traffic is to define per backend the session rate and number of connections it may
    accept.

    ~~~~~~~
    backend be_app_stable
    #
    acl too_fast be_sess_rate gt 10
    acl too_many be_conn gt 10
    tcp-request inspect-delay 3s
    tcp-request content accept if ! too_fast or ! too_many
    tcp-request content accept if WAIT_END
    ~~~~~~~

    With this setting, 10 concurrent connection are still allowed (response time is about 7-8 ms)

    ab -n 500 -c 10 'http://localhost:8080/rest/services/height?easting=600000&northing=200000'


    Server Software: Apache/2.2.22
    Server Hostname: localhost
    Server Port: 8080

    Document Path: /rest/services/height?easting=600000&northing=200000
    Document Length: 18 bytes

    Concurrency Level: 10
    Time taken for tests: 0.550 seconds
    Complete requests: 500
    Failed requests: 0
    Total transferred: 346280 bytes
    HTML transferred: 9000 bytes
    Requests per second: 908.99 [#/sec] (mean)
    Time per request: 11.001 [ms] (mean)
    Time per request: 1.100 [ms] (mean, across all concurrent requests)
    Transfer rate: 614.78 [Kbytes/sec] received

    Connection Times (ms)
    min mean[+/-sd] median max
    Connect: 0 0 0.3 0 2
    Processing: 5 10 11.0 8 116
    Waiting: 5 9 11.0 7 116
    Total: 5 10 11.0 8 116

    Percentage of the requests served within a certain time (ms)
    50% 8
    66% 9
    75% 10
    80% 11
    90% 13
    95% 15
    98% 18
    99% 104
    100% 116 (longest request)
    Going with 12 concurent connections, many connections are rejected

    ab -n 500 -c 12 'http://localhost:8080/rest/services/height?easting=600000&northing=200000'


    Server Software: Apache/2.2.22
    Server Hostname: localhost
    Server Port: 8080

    Document Path: /rest/services/height?easting=600000&northing=200000
    Document Length: 18 bytes

    Concurrency Level: 12
    Time taken for tests: 54.715 seconds
    Complete requests: 500
    Failed requests: 0
    Total transferred: 345500 bytes
    HTML transferred: 9000 bytes
    Requests per second: 9.14 [#/sec] (mean)
    Time per request: 1313.171 [ms] (mean)
    Time per request: 109.431 [ms] (mean, across all concurrent requests)
    Transfer rate: 6.17 [Kbytes/sec] received

    Connection Times (ms)
    min mean[+/-sd] median max
    Connect: 0 0 0.3 0 2
    Processing: 4 1313 1489.5 27 3186
    Waiting: 4 1312 1489.6 27 3186
    Total: 4 1313 1489.5 27 3186

    Percentage of the requests served within a certain time (ms)
    50% 27
    66% 3006
    75% 3007
    80% 3008
    90% 3017
    95% 3042
    98% 3082
    99% 3118
    100% 3186 (longest request)


    Using Headers
    =============


    Sometimes **HAProxy** may not have access to source IP of requests, because it lays behing a CDN or an AWS ELB. So, if these elements
    are correctly setting an **X-Forwarded-For** header, we may eventually use it. Let see


    Configuration file *haproxy_x_forwarded_for.cfg*


    ~~~~~
    frontend fe_app
    [...]
    # Check restricted network
    acl restricted_network src 90.80.70.60 # Home network
    acl restricted_network hdr_ip(X-Forwarded-For) -f /etc/haproxy/acl_restricted_network
    use_backend blocked-proxy-ua if !restricted_network
    ~~~~~


    In this case we restrict access to one ip address (90.80.70.60) or to requests originating from '1.2.3.4/32' or '5.6.7.8/32' (_acl_restricted_network_)

    Requesting from this machine is not allowed:

    [root@ip-10-220-5-68] curl -I http://localhost:8080
    HTTP/1.0 503 Service Unavailable
    Cache-Control: no-cache
    Connection: close
    Content-Type: text/html

    But faking the *X-Forwared-For* header makes it a valid request:

    [root@ip-10-220-5-68] curl -I -H "X-Forwarded-For: 1.2.3.4" http://localhost:8080
    HTTP/1.1 200 OK
    Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token
    Access-Control-Allow-Methods: POST, GET, OPTIONS
    Access-Control-Allow-Origin: *
    Age: 0





    Rate limit based on header values
    =================================

    See <https://github.com/dschneller/haproxy-http-based-rate-limiting/blob/master/haproxy.cfg>

    Configuration file *haproxy_x_forwarded_for.cfg*

    ~~~~~
    stick-table type string size 100k expire 30s store gpc0_rate(3s)
    acl document_request path_beg -i /rest/services/height
    acl too_many_uploads_by_user sc0_gpc0_rate() gt 2
    acl mark_seen sc0_inc_gpc0 gt 0
    tcp-request content track-sc0 hdr(X-Forwarded-For) if METH_GET document_request
    reqadd Referer:\ http://zorba.geo.admin.ch
    use_backend be_429_slow_down if mark_seen too_many_uploads_by_user
    ~~~~~

    In this case, we must use a *stick-table* of type *string* to store the *X-Forwarded-For* header (or is there a way to convert it to ip?).
    We also restrict the rate limiting to request **GET** on path begining with **/rest/services/height**


    ab -n 500 -c 12 'http://localhost:8080/rest/services/height?easting=600000&northing=200000'



    ab -n 500 -c 12 -H "X-Forwared-For: 6.6.6.6" 'http://localhost:8080/rest/services/height?easting=600000&northing=200000'


    for i in {0..10}; do sleep 2; curl -s -o /dev/null -H "X-Forwarded-For: 1.2.3.4" -w "%{http_code}\n" "http://localhost:8080/rest/services/height?easting=600000&northing=200000"; done
    200
    200
    200
    200
    200
    200
    200
    200
    200
    200
    200


    for i in {0..10}; do curl -s -o /dev/null -H "X-Forwarded-For: 1.2.3.4" -w "%{http_code}\n" "http://localhost:8080/rest/services/height?easting=600000&northing=200000"; done
    200
    200
    429
    429
    429
    200
    200
    429
    429
    429
    200

    ab -n 100 -c 4 -H "X-Forwarded-For: 6.6.6.6" 'http://localhost:8080/rest/services/height?easting=600000&northing=200000'

    Server Hostname: localhost
    Server Port: 8080

    Document Path: /rest/services/height?easting=600000&northing=200000
    Document Length: 18 bytes

    Concurrency Level: 4
    Time taken for tests: 50.104 seconds
    Complete requests: 100
    Failed requests: 98
    (Connect: 0, Receive: 0, Length: 98, Exceptions: 0)
    Total transferred: 15500 bytes
    HTML transferred: 36 bytes
    Requests per second: 2.00 [#/sec] (mean)
    Time per request: 2004.173 [ms] (mean)
    Time per request: 501.043 [ms] (mean, across all concurrent requests)
    Transfer rate: 0.30 [Kbytes/sec] received

    Connection Times (ms)
    min mean[+/-sd] median max
    Connect: 0 0 0.2 0 1
    Processing: 6 1964 281.0 2004 2006
    Waiting: 6 1963 281.0 2004 2006
    Total: 7 1964 280.9 2004 2006

    Percentage of the requests served within a certain time (ms)
    50% 2004
    66% 2005
    75% 2005
    80% 2005
    90% 2005
    95% 2005
    98% 2005
    99% 2006
    100% 2006 (longest request)
    2017/10/11 17:29:22 <local0,info> 172.17.0.1:55866 [11/Oct/2017:17:29:22.777] fe_app be_app_stable/upstream_server 0/0/1/4/5 200 694 - - ---- 0/0/0/0/0 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:22 <local0,info> 172.17.0.1:55872 [11/Oct/2017:17:29:22.783] fe_app be_app_stable/upstream_server 0/0/1/4/5 200 694 - - ---- 0/0/0/0/0 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:24 <local0,info> 172.17.0.1:55878 [11/Oct/2017:17:29:22.789] fe_app be_429_slow_down/<NOSRV> -1/2003/-1/-1/2002 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:26 <local0,info> 172.17.0.1:55882 [11/Oct/2017:17:29:24.792] fe_app be_429_slow_down/<NOSRV> -1/2003/-1/-1/2002 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:28 <local0,info> 172.17.0.1:55886 [11/Oct/2017:17:29:26.795] fe_app be_429_slow_down/<NOSRV> -1/2002/-1/-1/2001 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:28 <local0,info> 172.17.0.1:55890 [11/Oct/2017:17:29:28.797] fe_app be_app_stable/upstream_server 0/0/1/3/4 200 694 - - ---- 0/0/0/0/0 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:28 <local0,info> 172.17.0.1:55896 [11/Oct/2017:17:29:28.802] fe_app be_app_stable/upstream_server 0/0/1/4/5 200 694 - - ---- 0/0/0/0/0 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:30 <local0,info> 172.17.0.1:55902 [11/Oct/2017:17:29:28.808] fe_app be_429_slow_down/<NOSRV> -1/2003/-1/-1/2002 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:32 <local0,info> 172.17.0.1:55906 [11/Oct/2017:17:29:30.811] fe_app be_429_slow_down/<NOSRV> -1/2003/-1/-1/2002 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"
    2017/10/11 17:29:33 <local0,info> 172.17.0.1:55910 [11/Oct/2017:17:29:32.814] fe_app be_429_slow_down/<NOSRV> -1/891/-1/-1/891 500 144 - - PT-- 0/0/0/0/3 0/0 {-,"",""} "GET /rest/services/height?easting=600000&northing=200000 HTTP/1.0"



    GeoIP with source IP/Header
    ===========================

    We may try to get the MaxMind [GeoIP](http://dev.maxmind.com/geoip/legacy/geolite/) database (legacy format) and use it with HAProxy ([geolocation and haproxy])

    Using whitelist/blacklist
    -------------------------


    We define an hypothetic *whitelist*, with a single network (a bluewin IP)

    $ cat CH.txt
    85.5.53.0/21

    and in the **haproxy.cfg**

    ~~~~~~~
    acl acl_CH src -f /usr/local/etc/haproxy/CH.txt
    http-request deny if acl_CH
    ~~~~~~~

    Let's test it.

    From home, checking my IP is really 85.5.53.104:

    marco at ultrabook in ~
    $ curl ifconfig.co
    85.5.53.104

    it will be denied:

    marco at ultrabook in ~
    $ curl -I "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.0 403 Forbidden
    Cache-Control: no-cache
    Connection: close
    Content-Type: text/html

    And from **docker0**, which is an AWS instance in Ireland, everything is fine:

    2d [ltmom@ip-10-220-4-246:~] $ curl ifconfig.co
    54.194.151.117



    2d [ltmom@ip-10-220-4-246:~] $ curl -I "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.1 200 OK
    Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token
    Access-Control-Allow-Methods: POST, GET, OPTIONS
    Access-Control-Allow-Origin: *
    Age: 0

    Checking an entry in the GeoIP database:


    # /usr/bin/mmdblookup --file GeoLite2-Country.mmdb --ip 52.48.106.118 registered_country names en
    "Ireland" <utf8_string>


    Blocking whole countries
    ------------------------

    Considering a *geoip.txt* file as:

    ~~~~~~~
    $ head geoip.txt
    1.0.0.0/24 AU
    1.0.1.0/24 CN
    1.0.2.0/23 CN
    1.0.4.0/22 AU
    1.0.8.0/21 CN
    1.0.16.0/20 JP
    1.0.32.0/19 CN
    1.0.64.0/18 JP
    1.0.128.0/17 TH
    1.1.0.0/24 CN
    ~~~~~~~


    We may define the follwing *acl*, allowing only Iran, sorry Ireland and Switzerland

    # For source ip
    acl acl_geoloc_ch_ie src,map_ip(/usr/local/etc/haproxy/geoip.txt) -m reg -i (IE|CH)
    # For x-forwarded-for
    acl acl_geoloc_ch_ie hdr(X-Forwarded-For),map_ip(/usr/local/etc/haproxy/geoip.txt) -m reg -i (IE|CH)

    http-request deny if !acl_geoloc_ch_ie



    GeoIP set by CloudFlare
    -----------------------

    Not tested, but it looks like some CDN like Cloudflare are setting GeoIP headers([haproxy cloudflare]) which be used by HAProxy

    It looks like **ClouFlare** is adding some custom header:

    _SERVER["HTTP_CF_IPCOUNTRY"] CN
    _SERVER["HTTP_CF_RAY"] 17da8155355b0520-SEA
    _SERVER["HTTP_CF_VISITOR"] {"scheme":"http"}
    _SERVER["HTTP_CF_CONNECTING_IP"] XX.YY.ZZ.00



    Accessing the sticky-table
    ==========================





    #<i class="fa fa-map" aria-hidden="true"></i> Colophon

    This document was created with [pandoc](http://pandoc.org), with the following commands:

    /usr/local/bin/pandoc --smart --standalone -f markdown --latex-engine=xelatex --number-sections --variable mainfont="Latin Modern Roman" \
    -V fontsize=9pt -V papersize:a4 --toc -F pandoc-citeproc --metadata link-citations -o "haproxy_rate_limiting.pdf" "haproxy_rate_limiting.md"


    [better rate limiting]: https://blog.serverfault.com/2010/08/26/1016491873/
    "Better Rate Limiting For All with HAProxy"


    [haproxy ddos config]: https://www.codementor.io/peerasan780/haproxy-ddos-protection-config-co0dbs7ua
    "Haproxy DDoS Protection config"



    [haproxy ddos]: https://github.com/analytically/haproxy-ddos
    "DDOS and attack resilient HAProxy configuration"

    [haproxy cloudflare]: https://roman.pertl.org/2016/02/cloudflare-and-haproxy-lodbalancer/
    "Cloudflare and Haproxy Lodbalancer"

    [geolocation and haproxy]: https://icicimov.github.io/blog/devops/Haproxy-GeoIP/
    "Geo Location with HAProxy "




    head GeoIPCountryWhois.csv
    "1.0.0.0","1.0.0.255","16777216","16777471","AU","Australia"
    "1.0.1.0","1.0.3.255","16777472","16778239","CN","China"
    "1.0.4.0","1.0.7.255","16778240","16779263","AU","Australia"
    "1.0.8.0","1.0.15.255","16779264","16781311","CN","China"
    "1.0.16.0","1.0.31.255","16781312","16785407","JP","Japan"
    "1.0.32.0","1.0.63.255","16785408","16793599","CN","China"



    $ cat CH.txt
    85.5.53.0/21



    acl acl_CH src -f /usr/local/etc/haproxy/CH.txt
    http-request deny if acl_CH


    marco at ultrabook in ~
    $ curl ifconfig.co
    85.5.53.104
    marco at ultrabook in ~
    $ curl -I "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.0 403 Forbidden
    Cache-Control: no-cache
    Connection: close
    Content-Type: text/html


    2d [ltmom@ip-10-220-4-246:~] $ curl ifconfig.co
    54.194.151.117
    2d [ltmom@ip-10-220-4-246:~] $ curl -I "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.1 200 OK
    Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token
    Access-Control-Allow-Methods: POST, GET, OPTIONS
    Access-Control-Allow-Origin: *
    Age: 0


    marco at ultrabook in ~
    $ curl -I -H "X-Forwarded-For: 52.48.106.118" "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.1 200 OK
    Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token
    Access-Control-Allow-Methods: POST, GET, OPTIONS
    Access-Control-Allow-Origin: *
    Age: 0
    Cache-Control: max-age=0, must-revalidate, no-cache, no-store
    Content-Type: application/json; charset=UTF-8
    Date: Fri, 13 Oct 2017 19:08:18 GMT
    Expires: Fri, 13 Oct 2017 19:08:18 GMT
    Last-Modified: Fri, 13 Oct 2017 19:08:18 GMT
    Pragma: no-cache
    Server: Apache/2.2.22 (Debian)
    Vary: Accept-Encoding
    Via: 1.1 varnish-v4
    X-Cache: MISS
    X-UA-Compatible: IE=Edge
    X-Varnish: 827075497

    marco at ultrabook in ~
    $ curl -I -H "X-Forwarded-For: 8.8.8.8" "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.0 403 Forbidden
    Cache-Control: no-cache
    Connection: close
    Content-Type: text/html

    marco at ultrabook in ~
    $ curl -I "http://haproxy.dubious.cloud/rest/services/height?easting=600000&northing=200000"
    HTTP/1.0 403 Forbidden
    Cache-Control: no-cache
    Connection: close
    Content-Type: text/html