Pete Projects
To give the child a name, Pete simply stands for PErformance TEst.
Why
Learning new languages like Elixir alone is already exciting but make it a bit more exciting by trying new frameworks and compare their speed against each other. Because I'm a PHP (mostly Drupal) guy and just discovered Elixir I want to take it as opportunity to learn functional programming paradigms but also see how Elixir's (Erlang) concurrency model speeds out our traditional languages such as PHP. Therefore I took the Elixir web framework Phoenix Framework and try to compare it in a similar structure and setup to Slim Framework (PHP) and Phalcon (PHP as C extension) and see how the performance differs.
What
I choose to do a very simplistic gallery app, with no model and database backend. Only router, controller, view and templates are used.
Requriements
Image gallery:
- Headline
- 4 images (filenames as array/list with copyright info)
- 3 Templates/partials (base, gallery, image)
- CSS
- No model and database backend (only controller / view / templates)
Framworks
How
I'm not familar with any of the frameworks I have chosen here, so bare with me if some structure is a complete mess :)
See subdirectories for framework specific information.
Test Results
I made multiple test runs on Raspberry Pi 2 (4 cores) and some Digitalocean VMs with 2 and 12 cores to check how the frameworks deal with multiple cores and concurrency.
Update 2016-01-05:
After changing php.ini settings (see configuration settings below) PHP frameworks where
able to improve by 10 - 40%. Updated test results. Would be great if we could figure out why Phoenix
is not using all system ressources on 12-core and it get's back to an interesting first place race with Phalcon at the top.
Update 2016-01-06:
Ok, the history has to be rewritten.
Because I used a slow way to
include the images in the template file the performance was really bad. We tracked that issue down
and now we really see the performance of Phoenix (Elixir/Erlang). I expected Phoenix to be faster
than PHP frameworks but that much of lead, I did not expect. Compared to the last test Phoenix speedup was 4x to 14x!
But take a look below yourself, unbelievable fast. And the craziest thing is, no matter how much load you throw at it, it seems
to never return a bad request (3xx or 5xx) status code. (At least bandwidth seems to be maxed out so unable to throw more load at it :))
Testing using wrk
Using wrk as benchmarking tool with this or similar command:
$ wrk -t4 -c100 -d60s --timeout 2000 http://ip-or-host.tld/gallery
Language versions used for frameworks
PHP 5: PHP 5.6.14-0-deb8u1 as PHP-FPM, OPcache enabled, Nginx 1.6.2
PHP 7: PHP 7.0.1 (self compiled) as PHP-FPM, OPcache enabled, Nginx 1.6.2
Phoenix: Erlang 18.2.1 (SMP + hipe enabled), Elixir 1.2.0, Phoenix 1.1.0
Testing summary: Raspberry Pi 2 (Model B)
One nice thing about the Raspberry Pi is that the hardware is cheap and easy to get and test results can (hopefully) be reproduced and compared easier.
Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
---|---|---|---|
Phoenix | 2247.92 | 47.92 | 36.19 |
Phalcon | 586.41 | 170.22 | 18.68 |
Slim (PHP 5.6) | 132.75 | 748.62 | 104.27 |
Slim (PHP 7.0) | 27.71 | >3500.00 | 553.34 |
Detailed results and specs: results--raspberry-pi2.md
Testing summary: Virtual Machine with 2 cores
Basic digitalocean VM with SSD and 2 cores, 2gb ram
Pretty interesting that with fewer cores Phoenix is close but does not take the crown this time.
Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
---|---|---|---|
Phalcon | 1716.16 | 58.28 | 16.74 |
Slim (PHP 7.0) | 1394.63 | 71.26 | 13.56 |
Phoenix | 1347.11 | 54.47 | 16.32 |
Slim (PHP 5.6) | 693.11 | 143.40 | 20.78 |
Detailed results and specs: results--2-core-vm.md
But now time to get really into multi core business and see who can handle concurrency best:
Testing summary: Virtual Machine with 12 cores
App VM: Digitalocean VM with SSD and 12 cores, 32gb RAM
Benchmark VM: Digitalocean VM with SSD and 4 cores, 8 gb RAM
Connected over 1 Gb/s
We do the same number of connections but with more threads first, then let's try something insane and quadruple the number of connections and see what happens.
# ./wrk -t12 -c100 -d60s --timeout 1000
Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
---|---|---|---|
Phoenix | 38399.63 | 2.56 | 2.28 |
Phalcon | 9669.83 | 11.25 | 14.32 |
Slim (PHP 7.0) | 7162.05 | 13.44 | 3.70 |
Slim (PHP 5.6) | 3419.78 | 28.11 | 6.80 |
Ok, get a bit extreme and throw 12 threads with 400 connections onto the app server:
# ./wrk -t12 -c400 -d180s --timeout 1000
Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
---|---|---|---|
Phoenix | ** | ** | ** |
Phalcon | - | - | - |
Slim (PHP 7.0) | - | - | - |
Slim (PHP 5.6) | - | - | - |
** 2016-01-06: will need to run this test sometime again, old value was 3000 req/s but it handles now way more. See insane test below :)
Ok, so I really want to make Phoenix throw some errors now and do something insane. 48 treads 500 connections each.
# ./wrk -t48 -c500 -d60s --timeout 1000
Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
---|---|---|---|
Phoenix | 41162.79 | 43.55 | 111.75 |
Phalcon | - | - | - |
Slim (PHP 7.0) | - | - | - |
Slim (PHP 5.6) | - | - | - |
This is just outstanding, take a look at the details below and look at the response times etc.
Detailed results, specs and comments: results--12-core-vm.md
Configuration
Some configs for PHP and Nginx can be found in configs
directory:
Raspberry PI + 2 core VM configs for Nginx + PHP-FPM pool:
configs/raspberry-pi2/nginx.conf
configs/raspberry-pi2/php-fpm_pool.d_www.conf
12 Core VM
configs/big-machine/nginx.conf
configs/big-machine/php-fpm_pool.d_www.conf
Common configs (Nginx vhosts + php.ini used an all testsystems):
configs/common/phalcon_nginx_vhost
configs/common/slim_nginx_vhost
configs/common/php.ini
Adding the following php.ini opcache-options improved above mentioned throughput for the PHP frameworks
about 10 - 20 % of improvement (+ ~40% for Slim on 12-core!). Thanks to @andresgutierrez for suggesting these tweaks.
opcache.revalidate_freq = 300
opcache.enable_file_override = On
(Keep in mind that this revalidates opcache code only every 5 minutes, so you definetely want that option on
production when you want to squeeze everything out and code does not change)
configs/common/phpinfo.html
Credits / inspiration
I was inspired by these great guys but wanted to do it my way and see the results with a slightly more complex testapp.
http://blog.onfido.com/using-cpus-elixir-on-raspberry-pi2/
(title is a bit misleading because also PHP with FPM uses all CPUs pretty much, but more to that in a blog post)
Comparison from Chris McCord (creator of Phoenix) with Rails:
http://www.littlelines.com/blog/2014/07/08/elixir-vs-ruby-showdown-phoenix-vs-rails/
Followup to above mentioned comparison but extended to Go, Ruby, NodeJS frameworks:
https://github.com/mroth/phoenix-showdown
License: MIT