Do you remember a magnitude?
Do you remember a magnitude?
Can someone help me get APC setup correctly? It’s for a good cause. In this blog post I announced a PHP framework comparison I have been working on for a few weeks. Qiang pointed out that the benchmark part of the comparison would be more fair if it was done in my local developer box using APC. So I want to do just that.
I have Ubuntu. I have successfully installed Apache, MySQL, PHP and APC and APC is running. I suspect that APC is poorly configured because the benchmark runs slower with yiilite.php than with yii.php. I have tried to follow the advice on this thread:
sudo /etc/init.d/apache2 restart ab -n 5 'localhost/Benchmark/Yii/index.php?r=greeting' ab -t 30 -c 10 'localhost/Benchmark/Yii/index.php?r=greeting'
[table]
[tr][td]CodeIgniter[/td][td]709.68 [#/sec][/td][/tr]
[tr][td]yii.php[/td][td]557.81 [#/sec][/td][/tr]
[tr][td]yiilite.php[/td][td]421.41 [#/sec][/td][/tr]
[/table]
You can tell that something is wrong because the program runs slower with yiilite.php. I am confident that APC itself is running and is reading the config file because if I set 'apc.enabled=0' in the config files, the test speed drops by almost a factor of 10.
I don't know the right way to do it, so this is what I did:
cd .../yii-1.0.1/framework # Backup yii.php cp yii.php orig.yii.php # Switch to Yii Lite: cp yiilite.php yii.php # Switch to Yii: cp orig.yii.php yii.php
extension=apc.so apc.enabled=1 apc.enabled_cli=1 apc.mmap_file_mask=/tmp/apc.XXXXXX apc.max_file_size=2M apc.stat=0 # warning performance: apc.stat=0 apc.slam_defense=10 apc.write_lock=0 apc.ttl=0 apc.shm_size=128 apc.num_files_hint=4096 apc.gc_ttl=0 apc.rfc1867=on apc.localcache=on apc.localcache.size=204
Any ideas? Does anyone know what I'm doing wrong?
Below are the two bootstrap files that we use to test yii.php and yiilite.php:
http://code.google.c…trunk/yii-1.0.1
You may refer to the benchmarking shell script we use:
http://code.google.c…nchmark.sh?r=10
Make sure you shutdown Apache and then start it before testing a new framework setup. I think APC has some issue with Apache restart.
Below is the APC configuration we used:
apc.enabled=1 apc.shm_segments=1 apc.optimization=0 apc.shm_size=32 apc.ttl=7200 apc.user_ttl=7200 apc.num_files_hint=1024 apc.mmap_file_mask=/tmp/apc.XXXXXX apc.enable_cli=1 apc.cache_by_default=1
I switched to Apache stop+start, and I used your APC configuration. No change in the results.
Could you tell me how to download your entire benchmark directory with SVN? I'd like to try running your exact same benchmark on my computer. If I get results similar to yours, we'll know that the problem is not with APC.
Nevermind, I found it:
svn checkout http://phpmark.googlecode.com/svn/trunk/ phpmark-read-only
Ok. I ran Yii's benchmark. The results are below. Some things to notice:
So we can still see some odd behaviour. Btw, my own tests were done with Yii 1.0.1.r473.
/phpmark/baseline/index.html 3878.31 [#/sec] (mean) /phpmark/baseline/index.php 2710.24 [#/sec] (mean) /phpmark/cakephp-1.1.20/webroot/index.php 609.36 [#/sec] (mean) /phpmark/cakephp-1.2.0/benchmark 43.28 [#/sec] (mean) /phpmark/codeigniter-1.6.3/index.php 791.16 [#/sec] (mean) /phpmark/codeigniter-1.7.0/index.php 776.76 [#/sec] (mean) /phpmark/prado-3.1.3/index.php 164.41 [#/sec] (mean) /phpmark/symfony-1.1.6/web/index.php 215.39 [#/sec] (mean) /phpmark/symfony-1.2.1/web/index.php 184.28 [#/sec] (mean) /phpmark/yii-1.0.0/index.php 610.32 [#/sec] (mean) /phpmark/yii-1.0.0/index2.php 464.42 [#/sec] (mean) /phpmark/yii-1.0.1/index.php 2232.30 [#/sec] (mean) /phpmark/yii-1.0.1/index2.php 2191.26 [#/sec] (mean) /phpmark/zend-1.6.2/index.php 301.96 [#/sec] (mean) /phpmark/zend-1.7.2/index.php 315.22 [#/sec] (mean)
Fudge. I think I found a bug in the testing program.
When I visited http://localhost/php…1.0.1/index.php I got the error “Could not open phpmark/yii-1.0.1/framework/yii.php no such file or directory”.
I checked and indeed, phpmark/yii-1.0.1/framework/yii.php does not exist. The right path is phpmark/yii-1.0.1/framework/framework/yii.php. Ditto for index2.php and yiilite.php. I corrected the paths in index.php and index2.php. I ran the benchmark again and now the results for Yii 1.0.1 are way down. It looks like only reason Yii 1.0.1 looked so fast in my previous test is that PHP was just giving an error rather than running Yii. 
/phpmark/yii-1.0.0/index.php 587.11 [#/sec] (mean) /phpmark/yii-1.0.0/index2.php 393.40 [#/sec] (mean) /phpmark/yii-1.0.1/index.php 640.18 [#/sec] (mean) /phpmark/yii-1.0.1/index2.php 417.63 [#/sec] (mean)
Could one of the Yii developers take a look at the test and double check this? Check that the location of yii.php matches the require line in index.php and that the hello world program actually works.
Sorry, the SVN external dependency of yii-1.0.1 was not set correctly. I just fixed it.
Did you use the shell script I recommended to automate the test? A few warm-up requests are needed for each framework so that APC can properly set up the cache.
It seems you have really powerful machine. I don't know why yiilite is slower in your case. Perhaps you may contact luddic (he started this thread) because he had similar results at the beginning and later he found out some configuration error.
Welcome to the mud of performance tests. 
Quote
Did you re-run the test with the SVN version? I just want to confirm that you still get similar results to what’s posted on the performance page. I just updated from SVN and I got similar results as before:
/phpmark/cakephp-1.1.20/webroot/index.php 561.36 [#/sec] (mean) /phpmark/cakephp-1.2.0/benchmark 42.19 [#/sec] (mean) /phpmark/codeigniter-1.6.3/index.php 760.13 [#/sec] (mean) /phpmark/codeigniter-1.7.0/index.php 833.71 [#/sec] (mean) /phpmark/yii-1.0.0/index.php 589.34 [#/sec] (mean) /phpmark/yii-1.0.0/index2.php 423.90 [#/sec] (mean) /phpmark/yii-1.0.1/index.php 602.10 [#/sec] (mean) /phpmark/yii-1.0.1/index2.php 426.85 [#/sec] (mean)
Quote
Yes. I used the same script from SVN. I just ran sh benchmark.sh localhost/phpmark/
Quote
Actually, my machine is very average. It’s just a MacBook, not new. I’ll PM luddic to see if he has any ideas. When I’m confident I have the test right, I’ll update my framework comparison. It’s ok if Yii is not the fastest. It is still the second fastest after CodeIgniter, and my framework review makes it clear that there are many important factors besides speed.
Yes, those results on the performance page were repeated several times.
I just changed the benchmark scripts and replaced those "die" statements with "echo". I also reran the tests on the same machine and got the following results, which is very close to the one shown on the performance page:
Document Path: /phpmark/baseline/index.html Requests per second: 3170.89 [#/sec] (mean) Document Path: /phpmark/baseline/index.php Requests per second: 2928.81 [#/sec] (mean) Document Path: /phpmark/cakephp-1.2.1/benchmark Requests per second: 70.89 [#/sec] (mean) Document Path: /phpmark/codeigniter-1.7.0/index.php Requests per second: 140.78 [#/sec] (mean) Document Path: /phpmark/prado-3.1.3/index.php Requests per second: 52.77 [#/sec] (mean) Document Path: /phpmark/symfony-1.2.2/web/index.php Requests per second: 34.47 [#/sec] (mean) Document Path: /phpmark/yii-1.0.1/index.php Requests per second: 324.54 [#/sec] (mean) Document Path: /phpmark/yii-1.0.1/index2.php Requests per second: 447.78 [#/sec] (mean) Document Path: /phpmark/zend-1.7.3/index.php Requests per second: 51.60 [#/sec] (mean)
One thing that I forgot to mention is that this performance variance across different machines is probably caused by different apache server setting (e.g. fork/children, caching). I observed your tests gave much much better RPS than mine, even though your machine may not be as powerful.
Anyway, I agree with you that performance is not (far from) the whole story. People would be more interested at seeing your in-depth comparisons on different features of each framework.
mikl, below are the results when apc.stats=0
Document Path: /phpmark/baseline/index.html Requests per second: 3066.57 [#/sec] (mean) Document Path: /phpmark/baseline/index.php Requests per second: 2811.26 [#/sec] (mean) Document Path: /phpmark/cakephp-1.2.1/benchmark Requests per second: 79.16 [#/sec] (mean) Document Path: /phpmark/codeigniter-1.7.0/index.php Requests per second: 202.28 [#/sec] (mean) Document Path: /phpmark/prado-3.1.3/index.php Requests per second: 72.78 [#/sec] (mean) Document Path: /phpmark/symfony-1.2.2/web/index.php Requests per second: 48.59 [#/sec] (mean) Document Path: /phpmark/yii-1.0.1/index.php Requests per second: 652.97 [#/sec] (mean) Document Path: /phpmark/yii-1.0.1/index2.php Requests per second: 496.30 [#/sec] (mean) Document Path: /phpmark/zend-1.7.3/index.php Requests per second: 82.16 [#/sec] (mean)
Ah, thanks. Looks like yii.php takes much more advantage from apc.stat=0 and even outperforms yiilite.php.
When I did my tests I think I had apc.stat=0 so maybe that's the reason why my yii.php was outperforming yiilite.php. I'll test this hypothesis later.
Yes, I guess the reason that yiilite doesn't improve much when stats=0 is because it has much less files involved and stats=0 is mainly to reduce the cost spent in checking file timestamps.
DanielCarrera, yesterday we ran the tests on two different machines. One is a very powerful physical machine, one is a virtual machine using the default setup bundled with this image: http://www.thoughtpo…vmware/#fedora9 and we fixed its CPU frequency to 1.5GHz.
Below are the results when apc is enabled with stats=0. It seems to me that these two sets of data agree very well.
Document Path: /benchmark/phpmark/baseline/index.html Requests per second: 11173.07 [#/sec] (mean) Document Path: /benchmark/phpmark/baseline/index.php Requests per second: 7845.60 [#/sec] (mean) /benchmark/phpmark/cakephp-1.2.1/benchmark Requests per second: 335.80 [#/sec] (mean) Document Path: /benchmark/phpmark/codeigniter-1.7.0/index.php Requests per second: 736.70 [#/sec] (mean) Document Path: /benchmark/phpmark/yii-1.0.1/index.php Requests per second: 1631.67 [#/sec] (mean) Document Path: /benchmark/phpmark/yii-1.0.1/index2.php Requests per second: 1022.23 [#/sec] (mean) Document Path: /benchmark/phpmark/zend-1.7.3/index.php Requests per second: 588.00 [#/sec] (mean)
VM
Document Path: /benchmark/phpmark/baseline/index.html Requests per second: 1438.92 [#/sec] (mean) Document Path: /benchmark/phpmark/baseline/index.php Requests per second: 1219.19 [#/sec] (mean) /benchmark/phpmark/cakephp-1.2.1/benchmark Requests per second: 39.98 [#/sec] (mean) Document Path: /benchmark/phpmark/codeigniter-1.7.0/index.php Requests per second: 140.32 [#/sec] (mean) Document Path: /benchmark/phpmark/yii-1.0.1/index.php Requests per second: 228.92 [#/sec] (mean) Document Path: /benchmark/phpmark/yii-1.0.1/index2.php Requests per second: 200.07 [#/sec] (mean) Document Path: /benchmark/phpmark/zend-1.7.3/index.php Requests per second: 62.05 [#/sec] (mean)
Quote
And it also sounds logical: yiilite contains all classes, so it takes a longer time to parse/read the bytecode from cache, as for yii.php when only the required classes are loaded. Makes sense to me, that yii.php is faster than yiilite.php with a "perfect" cache.
Yes, perhaps we should remove yiilite from the release?
The remaining benefit of yiilite seems to be that it strips off those trace statements.
Well, there might be situations where you can't set apc.stat=0 on production systems. Then yiilite still performs better. So i'd say it makes sense to have it there.
Maybe put some guideline into the docs, when to use yii and when it's better to use yiilite?
Quote
Interesting. I would still like to figure out what I can do to get Yii to go faster than CodeIgniter. In both of your tests, Yii is faster than CI. I have written to luddic, but I don't think he's around anymore. His last post was from December 4, the same day he registered.