Skip to content

Instantly share code, notes, and snippets.

@omnibs

omnibs/phoenix showdown rackspace onmetal io.md

Last active June 10, 2024 17:47
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save omnibs/e5e72b31e6bd25caf39a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save omnibs/e5e72b31e6bd25caf39a to your computer and use it in GitHub Desktop.
Download ZIP
Phoenix Showdown Comparative Benchmarks @ Rackspace
Raw
phoenix showdown rackspace onmetal io.md

Comparative Benchmark Numbers @ Rackspace

I've taken the benchmarks from Matthew Rothenberg's phoenix-showdown, updated Phoenix to 0.13.1 and ran the tests on the most powerful machines available at Rackspace.

Results

Framework Throughput (req/s) Latency (ms) Consistency (σ ms)
Plug 198328.21 0.63ms 2.22ms
Phoenix 0.13.1 179685.94 0.61ms 1.04ms
Gin 176156.41 0.65ms 0.57ms
Play 171236.03 1.89ms 14.17ms
Phoenix 0.9.0-dev 169030.24 0.59ms 0.30ms
Express Cluster 92064.94 1.24ms 1.07ms
Martini 32077.24 3.35ms 2.52ms
Sinatra 30561.95 3.50ms 2.53ms
Rails 11903.48 8.50ms 4.07ms

Environment

Environment Version
Erlang 17.5
Elixir 1.1.0-dev
Ruby 2.2.0
Node 0.12.4
Go 1.3.3
Java 8u45

Server

Rackspace OnMetal IO
CPU Dual 2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	128 GB
System Disk	32 GB
Data Disk	Dual 1.6 TB PCIe flash cards
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$2.4658 / hr

Client

Rackspace OnMetal Compute
CPU	2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	32 GB
System Disk	32 GB
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$0.7534 / hr

Detailed Results

PHOENIX 0.9.0-dev (original)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   595.50us  398.08us  62.05ms   99.05%
    Req/Sec     8.50k   304.25     9.20k    70.34%
  5087667 requests in 30.10s, 10.42GB read
Requests/sec: 169030.24
Transfer/sec:    354.47MB

PHOENIX 0.13.1 (current)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.13.193:4000/showdown"
Running 30s test @ http://10.184.13.193:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   611.61us    1.04ms  95.57ms   99.05%
    Req/Sec     9.04k   656.30    10.98k    71.84%
  5408349 requests in 30.10s, 11.08GB read
Requests/sec: 179685.94
Transfer/sec:    376.82MB

PLUG

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   631.83us    2.22ms 217.98ms   99.05%
    Req/Sec     9.99k   761.02    18.66k    78.99%
  5969613 requests in 30.10s, 12.00GB read
Requests/sec: 198328.21
Transfer/sec:    408.34MB

PLAY

server:~# ./activator clean stage
server:~# ./target/universal/stage/bin/play-scala -Dhttp.port=5000
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:5000/showdown"
Running 30s test @ http://10.184.11.239:5000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.89ms   14.17ms 349.18ms   98.57%
    Req/Sec     8.80k     1.58k   23.45k    86.04%
  5154091 requests in 30.10s, 9.82GB read
Requests/sec: 171236.03
Transfer/sec:    334.12MB

RAILS

server:~# PUMA_WORKERS=20 MIN_THREADS=1 MAX_THREADS=80 RACK_ENV=production bundle exec puma
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.50ms    4.07ms  61.71ms   70.94%
    Req/Sec   598.10     40.81   740.00     71.03%
  357320 requests in 30.02s, 788.91MB read
Requests/sec:  11903.48
Transfer/sec:     26.28MB

SINATRA

server:~# RACK_ENV=production bundle exec puma -t 1:16 -w 20 --preload
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:9292/showdown"
Running 30s test @ http://10.184.11.239:9292/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.50ms    2.53ms  56.23ms   74.62%
    Req/Sec     1.54k   414.73     5.33k    65.96%
  919897 requests in 30.10s, 1.85GB read
Requests/sec:  30561.95
Transfer/sec:     63.04MB

GIN

server:~# GOMAXPROCS=20 GIN_MODE=release go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   655.59us  573.14us  10.11ms   89.21%
    Req/Sec     8.86k   759.03    15.02k    82.84%
  5302230 requests in 30.10s, 10.31GB read
Requests/sec: 176156.41
Transfer/sec:    350.61MB

MARTINI

server:~# GOMAXPROCS=20 MARTINI_ENV=production go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.35ms    2.52ms  32.73ms   60.45%
    Req/Sec     1.61k    99.91     1.98k    68.77%
  962648 requests in 30.01s, 1.87GB read
Requests/sec:  32077.24
Transfer/sec:     63.84MB

EXPRESS (20-node cluster)

server:~# NODE_ENV=production node server.js -w 20
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.24ms    1.07ms  26.92ms   83.39%
    Req/Sec     4.63k     2.37k   13.34k    74.60%
  2771185 requests in 30.10s, 5.65GB read
Requests/sec:  92064.94
Transfer/sec:    192.37MB

Bonus Track

ASP.Net 4.5.1 MVC 5 on IIS 8

On a Rackspace I/O v1 120GB instance listed as:

CPU:	32 vCPUs
RAM:	120 GB
System Disk:	40 GB
Data Disk:	1.2 TB (4 disks)
Network:	10 Gb / s
Disk I/O:	Best

Results:

root@client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.223.241.211/showdown"
Running 30s test @ http://10.223.241.211/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    62.98ms  104.13ms 962.77ms   81.84%
    Req/Sec     1.02k   435.75     2.92k    70.76%
  581748 requests in 30.02s, 1.39GB read
Requests/sec:  19381.05
Transfer/sec:     47.37MB
@zepplock
Copy link

zepplock commented Jul 9, 2015

Would be great to see updated .NET results ;)

@omnibs
Copy link
Author

omnibs commented Jul 20, 2015

I've been busy lately, but I plan on running Windows + Linux benches on Amazon. Amazon has that whole deal with network drivers for 10gbit to work so I've been putting it off heh

I did run linux benchmarks on the same machine that got me 19k on ASP.Net and results were not amazing at all. Top performers did around 80k req/s. I guess virtualization really sucks (yay for OnMetal).

@bbhoss
Copy link

bbhoss commented Sep 15, 2015

I think benchmarking things on AWS is pretty pointless. You never know what hardware you're going to get or how utilized it is. No test run has a chance at ever being even similar to others, unless you're running on dedicated hardware.

@TheCrott
Copy link

TheCrott commented Jan 15, 2016

Please update Phoenix to the latest stable, since this benchmark is using beta version of Phoenix

@slashmili
Copy link

slashmili commented Feb 28, 2016

What does exactly the Consistency column mean?

@peteygao
Copy link

peteygao commented Apr 3, 2016

Consistency refers to Standard Deviation (as per the Greek letter Sigma: σ). Roughly speaking, 68% of all requests fall within the stated "deviation" in terms of milliseconds. More about Standard Deviation on Wikipedia.

@naterh
Copy link

naterh commented Jul 3, 2016

I would be curious to see the same tests ran on Rackspace's new-ish OnMetal v2 servers that come with better virt networking.

@omnibs
Copy link
Author

omnibs commented Jul 13, 2016 •
edited
Loading

Didn't know they had better network on v2, interesting. @naterh do you have a source on that? All I found was about some VPN-ish hybrid cloud stuff.

Honestly I just saw the specs had lower clocks and didn't bother playing with them haha

@wbednarski
Copy link

wbednarski commented Aug 5, 2016

Can you use a monospace font in the table for numbers? Thanks!

@kioqq
Copy link

kioqq commented Sep 26, 2016

try iris for golang https://github.com/kataras/iris :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment