Doing it wrong again - this time with Zend Optimizer plus

The guys at PHP have now committed to shipping Zend Optimizer Plus with future releases of PHP, I thought I'd have a play around with it.

tl;dr

While normally I run my PHP from Apache + mod_php, for the purposes of this exercise, it was easier for me to set up nginx / php-fpm using PHP 5.3.3/APC 3.1.2 and 5.5.1/Zend Opcache 7.0.2dev. All were compiled from source using default settings on a 32-bit PCLinuxOS 2012 distribution (kernel 2.6.38.8) on a dual AMD 4200 machine with 2Gb of memory.

Tests were run using ab on localhost, each test was run by first seeding the opcocde cache (ab -n 2 -c 2) then taking a measurement (ab -n 1000 -c 100).

For both APC and ZOP+ tests were run with an without checking for modified timestamps on PHP source files.

Final scores

My results showed Zend Optimizer Plus to be no faster than APC. Although the numbers are within the margin of error, if anything ZOP+ was slower.

Config	Application	Timestamps	time per request (ms)
Php 5.3.3 + APC	Dokuwiki	yes	6.274
Php 5.3.3 + APC	Dokuwiki	no	6.271
Php 5.5.1, ZOP	Dokuwiki	yes	6.643
Php 5.5.1, ZOP	Dokuwiki	no	6.293
Php 5.3.3 + APC + mysql	Wordpress	yes	36.012
Php 5.3.3 + APC + mysql	Wordpress	no	35.868
Php 5.5.1, ZOP, mysqlnd	Wordpress	yes	35.905
Php 5.5.1, ZOP, mysqlnd	Wordpress	no	35.978
static HTML from Dokuwiki	Dokuwiki	n/a	0.171
static HTML from Wordpress	Wordpress	n/a	0.182

This seems to fly in the face of what is currently being reported elsewhere.

More stuff about my methodology

I wanted to create a reasonably realistic test, hence using 2 off the shelf Content Management Systems. Wordpress uses a MySQL backend for its data: and there is a further difference beteween the APC and ZOP+ configurations: the former uses libmysqlclient while the latter was built with MySQLnd (which meant I had to rewrite the database class to use the mysqli_ functions in place of mysql). The effects on performance are complex and tied to the level of concurrency but at 100 concurrent HTTP requests I was expecting this to be minimal. Dokuwiki, on the other hand, uses file based storage.

Other reviews

The PHP wiki page about the change links to a spreadsheet showing what look like impressive stats. The stats are reported as requests per second for various configurations.

Some more reviews:

https://managewp.com/boost-wordpress-performance-zend-optimizer - doesn't show response times / requests/second but does say that load and memory usage were lower implying greater capacity using ZOP+ compared with APC

http://halfelf.org/2013/trading-apc-for-zend/ reports a similar reduction in CPU and memory, but again no response times.

http://www.ricardclau.com/2013/03/apc-vs-zend-optimizer-benchmarks-with-symfony2/ results again in req/s, and showing an improvement of around 10-15%

http://massivescale.blogspot.co.uk/2013/06/php-55-zend-optimiser-opcache-vs-xcache.html compared ZOP+ and Xcache finding approx 15-20% improvement in req/s and similar reduction in response times with Joomla.

The optimizer bit(s)

PHP opcode caches have been around for a long time. ZOP+ brings something new: a code optimizer. Since it is still generating pcode it doesn't apply the CPU specific tweaks that a native code compiler does. Despite the 32 bits integer used to set the optimizer flag, only 6 flags are recognized by the optimizer (and the last pass only cleans up the debris left by the first 5). The optimizations are mostly substitutions - replacing PHP's builtin constants with literals, post increment with pre-increment, compile time type-juggling of built-in constants and such like. There is no inlining of functions in loops. No branch order prediction. Having dug through the code, I was not expecting it the optimizer to deliver revolutionary speed improvements.

And yet, Dimitry's spreadsheet shows 'ZF Test (ZF 1.5)' going from 158 req/s to 217 req/s!

I presume this refers to the Zend Framework. While this far from speedy I find it astonishing that the performance of the code should improve so much with a few relatively simple tweaks to the opcodes - it rather suggests that there is huge scope for optimizing the code by hand. Although I also note that the performance of 'Scrom (ZF App)' only improves by around 8%.

What am I doing wrong?

The consistent difference (apart from the opcode cache) in my experiment was using different versions of PHP - to a certain extent I'm not really comparing like-for-like. I can only assume that if I ran APC against PHP 5.5.1 and/or ZOP+ with PHP 5.3.3 I would see a very different story. However if you are seeking optimal performance at low load levels (rather than optimal capacity) then there seems to be little incentive to apply this upgrade.

There are anecdotal reports of stability issues with APC on PHP 5.4+; there may be sound technical and economic reasons why APC is not being actively maintained and for ZOP+ to be a better strategic choice.

A clear choice?

I can live without APC's support for user-data caching. But the elephant in the room is the fact that ZOP+ does not reclaim memory: if your code base is larger than the cache size, or the cache fills up with old verions of code, it forces a full flush and re-initialization. This should not be a problem for sites with a dedicated devops personnel managing releases to a small number of applications using a continuous deployment strategy. However for the rest of us there needs be a significant performance advantage with ZOP+ to make this a price worth paying.