Skip to content

Instantly share code, notes, and snippets.

@omnibs

omnibs/phoenix showdown rackspace onmetal io.md

Last active June 10, 2024 17:47
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save omnibs/e5e72b31e6bd25caf39a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save omnibs/e5e72b31e6bd25caf39a to your computer and use it in GitHub Desktop.
Download ZIP
Phoenix Showdown Comparative Benchmarks @ Rackspace
Raw
phoenix showdown rackspace onmetal io.md

Comparative Benchmark Numbers @ Rackspace

I've taken the benchmarks from Matthew Rothenberg's phoenix-showdown, updated Phoenix to 0.13.1 and ran the tests on the most powerful machines available at Rackspace.

Results

Framework Throughput (req/s) Latency (ms) Consistency (σ ms)
Phoenix 0.9.0-dev 169030.24 0.59ms 0.30ms
Phoenix 0.13.1 179685.94 0.61ms 1.04ms
Plug 198328.21 0.63ms 2.22ms
Play 171236.03 1.89ms 14.17ms
Rails 11903.48 8.50ms 4.07ms
Sinatra 30561.95 3.50ms 2.53ms
Gin 176156.41 0.65ms 0.57ms
Martini 32077.24 3.35ms 2.52ms
Express Cluster 92064.94 1.24ms 1.07ms

Environment

Environment Version
Erlang 17.5
Elixir 1.1.0-dev
Ruby 2.2.0
Node 0.12.4
Go 1.3.3
Java 8u45

Server

Rackspace OnMetal IO
CPU Dual 2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	128 GB
System Disk	32 GB
Data Disk	Dual 1.6 TB PCIe flash cards
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$2.4658 / hr

Client

Rackspace OnMetal Compute
CPU	2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	32 GB
System Disk	32 GB
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$0.7534 / hr

Detailed Results

PHOENIX 0.9.0-dev (original)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   595.50us  398.08us  62.05ms   99.05%
    Req/Sec     8.50k   304.25     9.20k    70.34%
  5087667 requests in 30.10s, 10.42GB read
Requests/sec: 169030.24
Transfer/sec:    354.47MB

PHOENIX 0.13.1 (current)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.13.193:4000/showdown"
Running 30s test @ http://10.184.13.193:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   611.61us    1.04ms  95.57ms   99.05%
    Req/Sec     9.04k   656.30    10.98k    71.84%
  5408349 requests in 30.10s, 11.08GB read
Requests/sec: 179685.94
Transfer/sec:    376.82MB

PLUG

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   631.83us    2.22ms 217.98ms   99.05%
    Req/Sec     9.99k   761.02    18.66k    78.99%
  5969613 requests in 30.10s, 12.00GB read
Requests/sec: 198328.21
Transfer/sec:    408.34MB

PLAY

server:~# ./activator clean stage
server:~# ./target/universal/stage/bin/play-scala -Dhttp.port=5000
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:5000/showdown"
Running 30s test @ http://10.184.11.239:5000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.89ms   14.17ms 349.18ms   98.57%
    Req/Sec     8.80k     1.58k   23.45k    86.04%
  5154091 requests in 30.10s, 9.82GB read
Requests/sec: 171236.03
Transfer/sec:    334.12MB

RAILS

server:~# PUMA_WORKERS=20 MIN_THREADS=1 MAX_THREADS=80 RACK_ENV=production bundle exec puma
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.50ms    4.07ms  61.71ms   70.94%
    Req/Sec   598.10     40.81   740.00     71.03%
  357320 requests in 30.02s, 788.91MB read
Requests/sec:  11903.48
Transfer/sec:     26.28MB

SINATRA

server:~# RACK_ENV=production bundle exec puma -t 1:16 -w 20 --preload
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:9292/showdown"
Running 30s test @ http://10.184.11.239:9292/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.50ms    2.53ms  56.23ms   74.62%
    Req/Sec     1.54k   414.73     5.33k    65.96%
  919897 requests in 30.10s, 1.85GB read
Requests/sec:  30561.95
Transfer/sec:     63.04MB

GIN

server:~# GOMAXPROCS=20 GIN_MODE=release go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   655.59us  573.14us  10.11ms   89.21%
    Req/Sec     8.86k   759.03    15.02k    82.84%
  5302230 requests in 30.10s, 10.31GB read
Requests/sec: 176156.41
Transfer/sec:    350.61MB

MARTINI

server:~# GOMAXPROCS=20 MARTINI_ENV=production go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.35ms    2.52ms  32.73ms   60.45%
    Req/Sec     1.61k    99.91     1.98k    68.77%
  962648 requests in 30.01s, 1.87GB read
Requests/sec:  32077.24
Transfer/sec:     63.84MB

EXPRESS (20-node cluster)

server:~# NODE_ENV=production node server.js -w 20
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.24ms    1.07ms  26.92ms   83.39%
    Req/Sec     4.63k     2.37k   13.34k    74.60%
  2771185 requests in 30.10s, 5.65GB read
Requests/sec:  92064.94
Transfer/sec:    192.37MB

Bonus Track

ASP.Net 4.5.1 MVC 5 on IIS 8

On a Rackspace I/O v1 120GB instance listed as:

CPU:	32 vCPUs
RAM:	120 GB
System Disk:	40 GB
Data Disk:	1.2 TB (4 disks)
Network:	10 Gb / s
Disk I/O:	Best

Results:

root@client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.223.241.211/showdown"
Running 30s test @ http://10.223.241.211/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    62.98ms  104.13ms 962.77ms   81.84%
    Req/Sec     1.02k   435.75     2.92k    70.76%
  581748 requests in 30.02s, 1.39GB read
Requests/sec:  19381.05
Transfer/sec:     47.37MB
@zepplock
Copy link

zepplock commented Jul 9, 2015

Would be great to see updated .NET results ;)

@omnibs
Copy link
Author

omnibs commented Jul 20, 2015

I've been busy lately, but I plan on running Windows + Linux benches on Amazon. Amazon has that whole deal with network drivers for 10gbit to work so I've been putting it off heh

I did run linux benchmarks on the same machine that got me 19k on ASP.Net and results were not amazing at all. Top performers did around 80k req/s. I guess virtualization really sucks (yay for OnMetal).

@bbhoss
Copy link

bbhoss commented Sep 15, 2015

I think benchmarking things on AWS is pretty pointless. You never know what hardware you're going to get or how utilized it is. No test run has a chance at ever being even similar to others, unless you're running on dedicated hardware.

@TheCrott
Copy link

TheCrott commented Jan 15, 2016

Please update Phoenix to the latest stable, since this benchmark is using beta version of Phoenix

@slashmili
Copy link

slashmili commented Feb 28, 2016

What does exactly the Consistency column mean?

@peteygao
Copy link

peteygao commented Apr 3, 2016

Consistency refers to Standard Deviation (as per the Greek letter Sigma: σ). Roughly speaking, 68% of all requests fall within the stated "deviation" in terms of milliseconds. More about Standard Deviation on Wikipedia.

@naterh
Copy link

naterh commented Jul 3, 2016

I would be curious to see the same tests ran on Rackspace's new-ish OnMetal v2 servers that come with better virt networking.

@omnibs
Copy link
Author

omnibs commented Jul 13, 2016 •
edited
Loading

Didn't know they had better network on v2, interesting. @naterh do you have a source on that? All I found was about some VPN-ish hybrid cloud stuff.

Honestly I just saw the specs had lower clocks and didn't bother playing with them haha

@wbednarski
Copy link

wbednarski commented Aug 5, 2016

Can you use a monospace font in the table for numbers? Thanks!

@kioqq
Copy link

kioqq commented Sep 26, 2016

try iris for golang https://github.com/kataras/iris :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment