Benchmark

system · December 4, 2008, 1:55pm

Hi,

I just run benchmarks on a dev server, with APC. The main differences are:

I warm up APC cache with a little apache bench vefore main execution
I added zend-1.6.2.autoload

zend-1.6.2.autoload is the same codebase than zend-1.6.2 with:

all require_* commented out
index.php changed to use Zend_Loader::autoload:



<?php





set_include_path(dirname(__FILE__).'/library');





require_once 'Zend/Loader.php';


Zend_Loader::registerAutoload();





//require_once 'Zend/Controller/Front.php';


Zend_Controller_Front::run(dirname(__FILE__)."/application/controllers");

All benchmark codes were tested before, to see if they return correct HTML code, and I have these results:

[table]

[tr][td]/cakephp-1.1.20/webroot/index.php [/td][td]Requests per second: [/td][td]717.15 [#/sec] (mean)[/td][/tr]

[tr][td]/codeigniter-1.6.3/index.php [/td][td]Requests per second: [/td][td]573.19 [#/sec] (mean)[/td][/tr]

[tr][td]/prado-3.1.2/index.php [/td][td]Requests per second: [/td][td]140.83 [#/sec] (mean)[/td][/tr]

[tr][td]/symfony-1.1.4/web/index.php [/td][td]Requests per second: [/td][td]170.19 [#/sec] (mean)[/td][/tr]

[tr][td]/yii-1.0-beta/index.php [/td][td]Requests per second: [/td][td]507.33 [#/sec] (mean)[/td][/tr]

[tr][td]/yii-1.0-beta/index2.php [/td][td]Requests per second: [/td][td]314.67 [#/sec] (mean)[/td][/tr]

[tr][td]/zend-1.6.2/index.php [/td][td]Requests per second: [/td][td]380.26 [#/sec] (mean)[/td][/tr]

[tr][td]/zend-1.6.2.autoload/index.php [/td][td]Requests per second: [/td][td]692.56 [#/sec] (mean)[/td][/tr]

[/table]

If we sort them, the results are quite different than the ones given on http://www.yiiframew…m/performance/

qiang · December 4, 2008, 2:45pm

Thank you for taking the effort to re-run the benchmark.

I think you should stop and start Apache server before running the test for each framework. Could you please let us know the results then?

Below is the shell script that we used to do the benchmark:



#!/bin/bash


HOSTNAME='www.yiiframework.com'


PORT=81


BASE_URI='/benchmark'


 


CONFIG_FILE='readme.txt'


TIME=`date "+%Y%m%d-%H%M%S"`


OUTPUT_FILE="benchmark-$TIME.txt"


 


AB_EXEC='/usr/bin/ab'


BENCH_EXEC="$AB_EXEC -t 30 -c 10"


BENCH_INIT="$AB_EXEC -n 5";


HTTP_RESTART='sudo /etc/init.d/httpd restart'


 


while read LINE


do


    # Find lines with "framework name: /uri", ignoring lines startings with "#"


    if echo $LINE |  grep -v "^#" | grep -q ": +/"


    then


        TEST=${LINE%:*}


        URI=`echo ${LINE#*:} | sed 's/^ *//' | sed 's/ *$//'`


        TEST_URL="http://$HOSTNAME:$PORT$BASE_URI$URI"


        echo "Testing $TEST_URL"


 


        # Restart apache, warm up the target url, hit it and sleep.


        $HTTP_RESTART


        sleep 2


        $BENCH_INIT $TEST_URL > /dev/null 2>&1


        echo "~~~ Testing $TEST : $TEST_URL" >> $OUTPUT_FILE


        $BENCH_EXEC $TEST_URL >> $OUTPUT_FILE


        sleep 3


    fi


done < $CONFIG_FILE

system · December 4, 2008, 3:38pm

After adding zend with autoload:



# cat benchmark-20081204-163734.txt  | grep -e Path -e Request


Document Path:          /benchmark/baseline/index.html


Requests per second:    4234.32 [#/sec] (mean)


Document Path:          /benchmark/baseline/index.php


Requests per second:    4226.27 [#/sec] (mean)


Document Path:          /benchmark/cakephp-1.1.20/webroot/index.php


Requests per second:    717.33 [#/sec] (mean)


Document Path:          /benchmark/codeigniter-1.6.3/index.php


Requests per second:    545.33 [#/sec] (mean)


Document Path:          /benchmark/prado-3.1.2/index.php


Requests per second:    139.55 [#/sec] (mean)


Document Path:          /benchmark/symfony-1.1.4/web/index.php


Requests per second:    169.40 [#/sec] (mean)


Document Path:          /benchmark/yii-1.0-beta/index.php


Requests per second:    519.26 [#/sec] (mean)


Document Path:          /benchmark/yii-1.0-beta/index2.php


Requests per second:    300.81 [#/sec] (mean)


Document Path:          /benchmark/zend-1.6.2/index.php


Requests per second:    380.03 [#/sec] (mean)


Document Path:          /benchmark/zend-1.6.2.autoload/index.php


Requests per second:    700.55 [#/sec] (mean)

Ludo

qiang · December 4, 2008, 4:18pm

Is this result obtained by stopping and starting apache before running the test for each framework?

One thing I observed in your result is that index2.php is slower than index.php for Yii. This does not seem reasonable to me because index2.php should benefit from APC much more than index.php. (index2.php includes only a big PHP file while index.php includes many files). I verified this on several other platforms.

I just re-ran the benchmarking again on this project site. The result is still very similar to the one we obtained a month ago (with a little degradation overall considering the project site is live functioning).

Let's see what other people report about their results.

system · December 4, 2008, 4:44pm

In fact, APC was misconfigured on first tests, here are the last results (APC working correctly), generated with your script, with a restart between each test:



Document Path:          /benchmark/baseline/index.html


Requests per second:    4266.89 [#/sec] (mean)


Document Path:          /benchmark/baseline/index.php


Requests per second:    4215.87 [#/sec] (mean)


Document Path:          /benchmark/cakephp-1.1.20/webroot/index.php


Requests per second:    851.45 [#/sec] (mean)


Document Path:          /benchmark/codeigniter-1.6.3/index.php


Requests per second:    1385.54 [#/sec] (mean)


Document Path:          /benchmark/prado-3.1.2/index.php


Requests per second:    570.93 [#/sec] (mean)


Document Path:          /benchmark/symfony-1.1.4/web/index.php


Requests per second:    477.23 [#/sec] (mean)


Document Path:          /benchmark/yii-1.0-beta/index.php


Requests per second:    1411.16 [#/sec] (mean)


Document Path:          /benchmark/yii-1.0-beta/index2.php


Requests per second:    1541.51 [#/sec] (mean)


Document Path:          /benchmark/zend-1.6.2/index.php


Requests per second:    865.49 [#/sec] (mean)


Document Path:          /benchmark/zend-1.6.2.autoload/index.php


Requests per second:    701.66 [#/sec] (mean)

And you're right, yii is really impressive. Not in the same proportions than yours, but it's pretty fast.

qiang · December 4, 2008, 4:53pm

Again, thank you for running the benchmarks.

Yes, the server configuration (both software and hardware) really affects the results, sometimes dramatically. I also couldn't get the same proportion between index.php and index2.php when running them on Windows. But since people were requesting to run the tests on a Linux box, Wei and I took the effort to re-do it, which turned out index2.php was much faster.

MetaYii · December 4, 2008, 5:17pm

luddic, could you please post your APC configuration?

system · December 4, 2008, 5:30pm



[apc]


extension=apc.so


apc.enabled=1


apc.enabled_cli=1


apc.mmap_file_mask=/tmp/apc.XXXXXX


apc.max_file_size=2M





# warning performance:


apc.stat=0


apc.slam_defense=10


apc.write_lock=0


apc.ttl=0





apc.shm_size=128


apc.num_files_hint=4096


apc.gc_ttl=0 





apc.rfc1867=on





apc.localcache=on


apc.localcache.size=2048

jonah · December 4, 2008, 6:38pm

First off, the version of CakePHP you are benching is very old. I heard the newer version of cake is supposed to be even faster than the old one. Even so, I find Cakephp to be very slow. I've been playing around with cakephp for about half a year now. When playing with my cakephp sites a get a good feel of it's speed.

When playing around with my Yii setup, I have to admit it seems a whole lot faster than my Cakephp setup. A lot faster. So I am very happy with Yii's speed and question these results.

system · December 18, 2008, 11:02am

I have made some benchmarks using yii to do a simple database querie on a table without key and with 15 regs. The numbers are:

yii.php - 18.25 rps

yiilite.php - 18.85 rps

With pgpool

aprox. 40 rps

Python web tools like werkzeug + sqlalchemy + jinja2 get 400 rps on the same situation.

How to improve this ?

qiang · December 18, 2008, 12:12pm

Did you turn on APC?

Are you using AR? If so, did you turn on schema caching?

system · December 18, 2008, 12:20pm

Quote

Did you turn on APC?

Yes

Quote

Are you using AR? If so, did you turn on schema caching?

I have created the list with model and crud shell operations. Turned off the debug option and :

ab -c 10 -t 60 http://localhost/~me…p?r=module/list

About the schema caching, i will read the docs to learn how to do this and repeat the test.

Ty.

system · December 25, 2008, 7:30pm

Quote

First off, the version of CakePHP you are benching is very old. I heard the newer version of cake is supposed to be even faster than the old one. Even so, I find Cakephp to be very slow. I've been playing around with cakephp for about half a year now. When playing with my cakephp sites a get a good feel of it's speed.

When playing around with my Yii setup, I have to admit it seems a whole lot faster than my Cakephp setup. A lot faster. So I am very happy with Yii's speed and question these results.

Yes! If Yii makes a benchmark with CakePHP 1.2 RC4 is great to compare. In pratical, CakePHP 1.2 is used in prodution without problems…

I maked some tests with frameworks (including Yii) and the result in http://blog.cakephp-…frameworks-php/ (in portuguese).

qiang · December 25, 2008, 7:51pm

Good to see your benchmarking results. Could you explain how you did those database-related tests?

The upcoming Yii 1.0.1 has some more optimizations which should make it even faster. We will update our tests when it is released.

Yes, CakePHP 1.2 RC4 is much slower than 1.1.20. Considering it is still not in a stable release, we are not including it in our tests.

system · December 26, 2008, 12:17am

My table (contacts) have one field 'id' pk, auto increment with others 2 fields: 'name' and 'phone'. Name is varchar(150) and phone varchar(30). The value of name is a UUID and phone a rand between 1000 and 3000000 as string.

When have Yii 1.0.1 I will update my tests too.

Reading the Yii docs, I see that focus is develop fast and execute fast! This is great! I look forward for version 1.1 for use with automated tests in the company. Do not forget the code coverage.

Congratulations for the work in Yii.

jonah · December 26, 2008, 6:42am

@qiang

CakePHP 1.2.0 final is now out, you may already be on this but thought you should know - http://bakery.cakeph…ft-of-1-2-final

qiang · December 26, 2008, 12:51pm

Thanks!

system · December 27, 2008, 3:04pm

One question: Have you enabled the APC cache in CakePHP (app/config/core.php) when did the tests of APC?

mikl · January 22, 2009, 10:18am

Just an addition to this old topic:

It would also be interesting to add a third test run for each framework with apc.stat=0. This is a very useful setting for production servers, as APC will not check the modification times for every cached file:

[quote=PHP manual]On a production server where you rarely change the code, turning stats off can produce a significant performance boost.
[/quote]

qiang · January 22, 2009, 12:12pm

Yes, we did try that, and Yii got much better results.