Usable Memcache

 I like simple.

Memcache is simple.

But sometimes it's maybe too simple.

I needed a substrate for holding session data which was accessible from more than one host (and potentially, more than one application). Given the lack of persistence in memcache I had a look at Couchbase....and found something even more bewildering than Oracle! Redis looked like it wouldn't take me weeks to get running, but while frequently touted as a replacement for memcache, I couldn't find any documentation stating whether it had a binary compatible memcache interface. So back to memcache.

First problem: I want high availability - that means more than one instance. Oh, C(r)AP! Sharding is easy enough, but replication needs a bit more work. A bit of digging and I found mcrouter which, along with haproxy means I can have load balancing and failover. But recovery is still missing. 

The memcache distribution comes with a Perl script which will copy memcache data from another instance. However when I tested it, I found the copy operation was very lossy - I was only getting around 70% of the data across to the new instance (HIGHLY variable) when hammering the source with a 1:1 mix of updates and gets. A bit disappointing - but understandable really. Maintaining a consistent list of all known items would add a lot of complexity and performance problems.

After a bit of reading I implemented and tested my own script using lru_crawler. Although still not perfect, in testing it was achieving >99%, again while the source was getting hosed. This is now available at https://github.com/symcbean/mcseed/blob/main/mcseed.php 

I disabled the package systemd unit file and created my own which uses a shell script to

1. Block external access to the memcache and mcrouter ports
2. start the memcache binary
3. run mcseed to populate the cache
4. allow incoming traffic to the mcrouter and memcache ports

Job done.

 

 

Unexpected PHP/MySQL failure!

Somehow my life/code is never as simple as the examples on w3schools.

Once again I should to explain a lot of the context (and apologize for some of it) before I get to the point. But the short version is both the mysql and mysqli extensions seem to cast all mysql data as strings.

Normally this goes unnoticed when you're working with PHP - its dynamic type conversion handles all the bumps. But not this time.

I wrote an application for tracking data quality some time ago. This runs various rules against several large datasets and collects the exceptions in a single polymorphic table which looks something like:

rule VARCHAR(20)
yearmonth INTEGER
PK1 INTEGER
PK2 VARCHAR(30)
PK3 VARCHAR(30)
data CLOB
age INTEGER

Each row of output from each rule is stored in a record in this table. Periodically the historic data is deleted.

Yes, I confess I'm holding a structured opaque data blob in a relational database! But by synthesizing PK1, PK2 and PK3 from the rule data it means I can compare month on month outputs to get the age of the exception. And I have a reliable unique identifier for each record in the (very large) table.

All well and good. This is currently tracking 320 rules and the application maintains summary level data. It has been running for about 8 years. But users as *so* demanding! They wanted to see one of the rules (customer whose email addresses pass a regex check but fail MX lookup) decorated with information come a separate database. Not such a big deal. The application has the capability to mailmerge the output of one ruleset to create a query for another (clever, eh?). So all that was needed was a bit of configuration.

But now the story takes a twist. The number of records involved is.....lets just say a lot. This all runs on a very old version of MySQL (because that's how RHEL rolls its distro, and everyone hates systemd) which was originally configured to give great performance on low latency storage (i.e. MyISAM) but now runs on a SAN with very high latency (but massive bandwidth).

Because of these considerations, along with the fact that I rarely rebuild the indexes, joining the output of the two rules was HORRENDOUSLY slow. Its supposed to be available at the click of a button, but I got bored waiting after one hour. This version of MySQL does not do merge joins. So I wrote my own in PHP. This took around 5 seconds to retrieve the data. But huge chunks of the data were missing! I eventually tracked this down to the fact that the mysql PHP extension was converting the PK1 integer attribute to a string. So "110" was greater than "1000" and my code was discarding lots of rows when trying to join the two datasets.

I can understand that there would be a cost if the library had to call back to the DBMS to get the type of each attribute in a resultset, but looking at the C API, it looks like the client lib is returning a correctly typed attribute, but the PHP engine is casting it as a string!

But, of course, the MySQL extension has long been deprecated. So I decided to see of MySQLi was similarly afflicted. After a brief diversion when I discovered about SO_PEERCRED I found that MySQLi does the same thing.

Local File Inclusion - why everything you ever read about file uploads is probably wrong

Allowing people to load content onto your server potentially allows them to deploy code which will be executed by your server. Your server can easily be pwned, leading to your work being destroyed/stolen and the website used for spamming, phishing, DOS attacks.....its a bad thing.

Where I've implemented file uploads, or have been asked how someone else should do it, I usually give this advice:

1. Ensure that uploaded content is held outwith the document root, failing that, in a directory configured with "php_flag engine off"
2. Only use filenames with safe characters - [a-z], [A-Z], [0-9], .-_
   (OWASP, after detailling why you should not use blacklisting as a preventative measure, offer a black list of characters :) )
3. Only allow content with a whitelisted extension to be uploaded
4. Check the mimetype of uploads - using PHP's mime_content_type(), not the value supplied by the user
5. Preferably convert the content to a different format (then back to the original format if required)

This looks like Security-in-depth, i.e. applying more protection than is actually required. I've always been somewhat skeptical of this as a justification for effort and expense (often in the context where in Security in breadth is sadly lacking). Certainly ticking each of the boxes in the list above is not always practical. But relying on ones own knowledge and understanding is a dangerous conceit.

I am indebted to wireghoul for pointing out to me that there is a rather subtle attack which can be exploited for LFI on Apache systems.

Hopefully you know that the PHP interpreter will run the PHP code embedded in any file presented to it. To give a simple example:

1. amend your webserver config to send png files to the PHP handler, e.g. by creating a .htaccess file containing
   
   

    AddHandler application/x-httpd-php .png
   



2. write your PHP code, and then append it to a valid png file....
   
       echo '<?php mail("root@localhost", "hello", "from the inside");' >>image.png
   
   
3. Point your web browser at the URL for image.png, and the picture renders in your browser, just as you would expect....but the PHP code executes too. No surprises so far. After all, if someone can reconfigure your webserver, they don't need to bother with the LFI - they've already pwned the system.

But it is also possible to get the webserver to execute PHP code embedded in an image file without changing the handler!

This comes about due to way mod_mime infers mimetypes from extensions

mod_mime will interpret anything in the name beginning with a . as an extension. From the linked documentation, note the paragraph

   Care should be taken when a file with multiple extensions gets associated with both a media-type and a handler. This will usually result in the request being handled by the module associated with the handler. For example, if the .imap extension is mapped to the handler imap-file (from mod_imagemap) and the .html extension is mapped to the media-type text/html, then the file world.imap.html will be associated with both the imap-file handler and text/html media-type. When it is processed, the imap-file handler will be used, and so it will be treated as a mod_imagemap imagemap file.

The mod_mime documentation authors go on to explain that this is the default behaviour for handlers, but that it is possible to ensure that only the last extension is used for choosing the handler with a FilesMatch block around the SetHandler directive.
 
So if we name our uploaded file image.php.png then we should be able to get our PHP code executing on a server.

Indeed, guess what method PHP recommend for configuring your server? (as implemented by Redhat, Suse, Ubuntu and others).

Let's try it out!

In the earlier example, we appended the PHP code to the end of the file. When I tried this again with the double extension, I got a lot of unusual characters in my browser. The Content-type was PHP's default test/html not image/png. So instead, the PHP code needs to prefix the image and set the content-type header:

    ( echo -n  '<?php  header("Content-Type: image/png");
      mail("root@localhost", "hello", "from the inside");
      ?>' ; cat original_image.png ) >> image.php.png
              
Point your browser at the URL, and once again, PHP code executes, the image renders but this time using the default config on most webservers.

The double extension is known but not by many.

It's not just me. I had a look around the web at advice being given about PHP file uploads - and a lot of it is wrong.

All the articles I've looked at claim that it only works where the extensions *other* than '.php' are unknown to the browser. As demonstrated here. this is NOT true.

Validating the mimetype won't protect against this kind of attack; although it would detect the file I created in the example, it won't work if the PHP code is embedded somewhere other than the start of the file. If it is contained in an unreachable block or as EXIF, then the code will execute, the file will be a valid image (based on its extension) and a mime check (which reads the first few bytes of the file) will say its an image. It will only render as an image when served up by the webserver if output buffering is enabled by default - but this will not often be a problem for an attacker.

Transactional websites and navigation

There's lot's of things that make me happy. In my professional life, it's getting stuff done, helping other people or learning something new. Recently I learnt something which was probably widely known but I'd managed to miss all these years. I'm so inspired that I'm going to share it with you all.

A lot of transactional websites crumble when you dare to do something as reckless as use the back button or open more than one window on the site. The underlying reason is that the developer is storing data relating to the transaction – i.e. specific to a navigation event in a single window - in the session – which is common to all the windows. A very poor way to mitigate the problem is to break the browser functionality by disabling the back button or interaction via a second window. I must admit to having used this in the past to mitigate the effects of problems elsewhere in the code (alright, if you must know – I bastardized Brooke Bryan's back button detector, and as for the new window....well the history length is zero)

But how should I be solving the problem?

The obvious solution is to embed all the half-baked ingredients of a transaction in each html page sent to the browser and send the updated data model back to the server on navigation. This can work surprisingly well as long as the data on the browser is cleared down between sessions. But with increasingly complex datasets, this becomes rather innefficient, particularly on slow connections. Further there are times when we want the transaction state to reflect the session state: consider a shopping basket – if a user fills a shoppng basket then ends their session we might want to retain the data about what they put in their shopping basket – but we might also want to release any stock reserved by the act of adding it to the basket. Often the situation arises where we end up with (what should be) the same data held in more than one place (browser and server). At some point the representations of the truth will diverge – and at that point it all goes rather pear shaped.

A while back I created a wee bit of code for point and click form building – PfP Studio. Key to the utility of this was the ability to treat a collection of widgets (a form) as a widget itself. And the easiest way to achieve that was to support multiple windows. When I first wrote this, I decided that the best way to handle the problem was to add partitioned areas to the session – one for each window. This depended on the goodwill of the user to open a new window via the functionality in the app rather than the browser chrome: each window had to carry a identifier (the “workspace”) across navigation events. Effectively I was rolling my own session handling with transids

This has a number of issues – the PHP sites warns about leaking authentication tokens but there's also a lot of overhead when you start having to deal with javascript triggerred navigation, and PRGs.

Then the other day I discovered that the browser window.name property was writeable in all major browsers! Hallelujah! Yes, it still means that you need to do some work to populate links and forms, but it's a lot simpler than my previous efforts – particularly with a javascript heavy site.

Any new window (unless it has been explicitly given a name in an href link or via window.open) has an empty string as the name – hence you simply set a random value if it's empty – and the value persists even if you press the back button.

While I think I've explained what I'm trying to say, a real example never goes amiss:

if (''==window.name) {
   var t=new Date();
   window.name=t.getMilliseconds()+Math.random();
}

Zend Optimizer Plus - still not following the party line

Having previously failed to get a significant difference in benchmarks between PHP 5.3 and 5.5, I was successful in establishing that the optimizer was producing code which ran slightly faster (about 2% with DokuWiki).

This time I tried ramping up the concurrency to see if PHP could deliver on its performance promises.

Running

ab -n 12000 -c $X http://localhost/src/doku.php

For $X in [10,50,100,200,300,400,500,600,700,800]  I got....

For less than 500 concurrent connections, PHP 5.3.3 is slower - but only about 5%. At more than 500 concurrent requests, 5.5.1 is slower!

I'm just no good at this benchmarking business.

Both were running from nginx+php-fpm with the same (common) config. 5.5.1 had full ZOP+ optimization enabled.

Zend Optimizer Plus - trying to do it right

Reading further (didn't bookmark the link and can't find it now) alongside benchmarks showing ZOP+ to be around 20% faster for "real world" applications there's also mention of a big reduction in memory usage.

I'm quite prepared to believe that better use of memory is possible - the runtime footprint of PHP code seems to be around 8 times the footprint of the script on disk - so plenty of scope for improvement. I was running my tests with 100 concurrent connections - not nearly saturating my machine. I expect that running with a much higher load / less memory would translate into the the performance improvements reported elsewhere - more testing required?

Meanwhile I had another look at the optimizer. I repeated the setup from last time, running with PHP 5.5.1, ZOP+ with filemtime checking off, fetching a single Dokuwiki page. The control test with full optimization - as per previous run - is giving slightly different results than last time. Since I'm running this on my home machine and it's also running lots of other things like X, KDE, my browser, mail client... it's possible that the system isn't in exactly the state it was in when I ran the previous tests.

Full Optimization: 

opcache.optimization_level=0xffffffff

6.334 ms/req

Optimization disabled : 

opcache.optimization_level=0

6.452 ms/req

Repeating the test several times gave consistent results; about 2% improvement in speed.

Not revolutionary, but every little helps - and in fairness Dokuwiki already has good performance optimization in the PHP code.

When time allows I'll go back and look at memory / CPU usae while running ZOP+ vs APC

Doing it wrong again - this time with Zend Optimizer plus

The guys at PHP have now committed to shipping Zend Optimizer Plus with future releases of PHP, I thought I'd have a play around with it.

tl;dr

While normally I run my PHP from Apache + mod_php, for the purposes of this exercise, it was easier for me to set up nginx / php-fpm using PHP 5.3.3/APC 3.1.2 and 5.5.1/Zend Opcache 7.0.2dev. All were compiled from source using default settings on a 32-bit PCLinuxOS 2012 distribution (kernel 2.6.38.8) on a dual AMD 4200 machine with 2Gb of memory.

Tests were run using ab on localhost, each test was run by first seeding the opcocde cache (ab -n 2 -c 2) then taking a measurement (ab -n 1000 -c 100).

For both APC and ZOP+ tests were run with an without checking for modified timestamps on PHP source files.

Final scores

My results showed Zend Optimizer Plus to be no faster than APC. Although the numbers are within the margin of error, if anything ZOP+ was slower.

Config	Application	Timestamps	time per request (ms)
Php 5.3.3 + APC	Dokuwiki	yes	6.274
Php 5.3.3 + APC	Dokuwiki	no	6.271
Php 5.5.1, ZOP	Dokuwiki	yes	6.643
Php 5.5.1, ZOP	Dokuwiki	no	6.293
Php 5.3.3 + APC + mysql	Wordpress	yes	36.012
Php 5.3.3 + APC + mysql	Wordpress	no	35.868
Php 5.5.1, ZOP, mysqlnd	Wordpress	yes	35.905
Php 5.5.1, ZOP, mysqlnd	Wordpress	no	35.978
static HTML from Dokuwiki	Dokuwiki	n/a	0.171
static HTML from Wordpress	Wordpress	n/a	0.182

This seems to fly in the face of what is currently being reported elsewhere.

More stuff about my methodology

I wanted to create a reasonably realistic test, hence using 2 off the shelf Content Management Systems. Wordpress uses a MySQL backend for its data: and there is a further difference beteween the APC and ZOP+ configurations: the former uses libmysqlclient while the latter was built with MySQLnd (which meant I had to rewrite the database class to use the mysqli_ functions in place of mysql). The effects on performance are complex and tied to the level of concurrency but at 100 concurrent HTTP requests I was expecting this to be minimal. Dokuwiki, on the other hand, uses file based storage.

Other reviews

The PHP wiki page about the change links to a spreadsheet showing what look like impressive stats. The stats are reported as requests per second for various configurations.

Some more reviews:

https://managewp.com/boost-wordpress-performance-zend-optimizer - doesn't show response times / requests/second but does say that load and memory usage were lower implying greater capacity using ZOP+ compared with APC

http://halfelf.org/2013/trading-apc-for-zend/ reports a similar reduction in CPU and memory, but again no response times.

http://www.ricardclau.com/2013/03/apc-vs-zend-optimizer-benchmarks-with-symfony2/ results again in req/s, and showing an improvement of around 10-15%

http://massivescale.blogspot.co.uk/2013/06/php-55-zend-optimiser-opcache-vs-xcache.html compared ZOP+ and Xcache finding approx 15-20% improvement in req/s and similar reduction in response times with Joomla.

The optimizer bit(s)

PHP opcode caches have been around for a long time. ZOP+ brings something new: a code optimizer. Since it is still generating pcode it doesn't apply the CPU specific tweaks that a native code compiler does. Despite the 32 bits integer used to set the optimizer flag, only 6 flags are recognized by the optimizer (and the last pass only cleans up the debris left by the first 5). The optimizations are mostly substitutions - replacing PHP's builtin constants with literals, post increment with pre-increment, compile time type-juggling of built-in constants and such like. There is no inlining of functions in loops. No branch order prediction. Having dug through the code, I was not expecting it the optimizer to deliver revolutionary speed improvements.

And yet, Dimitry's spreadsheet shows 'ZF Test (ZF 1.5)' going from 158 req/s to 217 req/s!

I presume this refers to the Zend Framework. While this far from speedy I find it astonishing that the performance of the code should improve so much with a few relatively simple tweaks to the opcodes - it rather suggests that there is huge scope for optimizing the code by hand. Although I also note that the performance of 'Scrom (ZF App)' only improves by around 8%.

What am I doing wrong?

The consistent difference (apart from the opcode cache) in my experiment was using different versions of PHP - to a certain extent I'm not really comparing like-for-like. I can only assume that if I ran APC against PHP 5.5.1 and/or ZOP+ with PHP 5.3.3 I would see a very different story. However if you are seeking optimal performance at low load levels (rather than optimal capacity) then there seems to be little incentive to apply this upgrade.

There are anecdotal reports of stability issues with APC on PHP 5.4+; there may be sound technical and economic reasons why APC is not being actively maintained and for ZOP+ to be a better strategic choice.

A clear choice?

I can live without APC's support for user-data caching. But the elephant in the room is the fact that ZOP+ does not reclaim memory: if your code base is larger than the cache size, or the cache fills up with old verions of code, it forces a full flush and re-initialization. This should not be a problem for sites with a dedicated devops personnel managing releases to a small number of applications using a continuous deployment strategy. However for the rest of us there needs be a significant performance advantage with ZOP+ to make this a price worth paying.

Starting a new website - part 2

The other tool I looked at was OpenOutreach

This is a bundle of Drupal and various widgets. Although the result is something which is well integrated, and with a huge amount of functionality, once again this comes at a horrendous performance cost. Even browsing through a couple of pages, cached serverside, with APC enabled (and sized to hold all the code) via http://localhost/, it was taking a visible 3 seconds to load the page. Spookily the openoutreach site (120 ms RTT) actually loaded faster on my machine than the local version - suggesting that maybe there was scope for a lot more tuning - particularly running a caching proxy in front (I was using a fairly stock Apache 2.2 pre-fork). And once again, cron jobs were a requirement.

It's not that I have anything against cron - and plenty of hosting packages come with cron support these days. It just begs the question of why did the people who wrote what is intended to be an off-the-shelf package not make so little attempt to minimise the dependencies? Why did they think that cron that was the easiest way to run a job? Do I have time to monitor the cron jobs and make sure they are working properly? But more importantly if they've chosen a quick fix solution here, what other corners have been cut?

The more I read about performance an usability the more I keep coming back to the magic 1 second. While there continues to be measurable productivity improvements down to around 0.3 seconds, and Google, Amazon and others have published studies showing that there is a measurable impact at lower time intervals, 1 second seems to standout as a massive milestone in page loading times - it is possible to get significantly faster page loads than this - but this seems to be a breakpoint of diminishing returns.

Sorry Drupal / Openoutreach, maybe not a fail, but B minus.

At this point I could have started looking round the many, many other open-source PHP CMS available. But I decided that I'd already spent too much time on this and went back to good old Dokuwiki. Dokuwiki does have issues - being file based it's difficult to scale to very high volumes of content / users and tricky to scale beyond a single server (the simplest way to this is to run additional nodes are read-only and replicate content updates from a single master node - since it's file based, this is easy to setup). However for my purposes scalability is not going to be an issue - the core system is very fast. The more recent versions have slowed down a bit at the front end due to the addition of jquery - this and all the javascript is loaded at the top of each page - leading to a rather ugly 200-350 millisecond pause in the screen rendering even loading from cache. However the template/theme for the application is contained in a plugin (there's lot's of themes available) which is a relatively simple bit of PHP code.

Unfortunately Dokuwiki bundles all the content to be written into the section together in tpl_metaheaders() - so although I was able to get the scripts at the bottom, it needed (small) changes to the core Dokuwiki source code.

With the core system (version Weatherwax) making these changes and moving the scripts to the bottom seemed to have no adverse effect on the functionality of the system. Unfortunately it does break a lot of existing plugins. This also exposed another issue with this approach. A common feature with many CMS, particularly those using javascript capable plugins is that they tend to embed javascript within the HTML of the page. In addition to the problem that this usually expects a library to already have been loaded, it also makes it impossible to implement a Content Security Policy banning inline scripts (a very simple and effective way to protect against XSS attacks).

Programming, like crack cocaine, is a rather hard habit to break - so I immediately started thinking about ways to solve this. The inline code usually just does 2 things:

• declares the data to be used by the rendering script
• invokes the rendering code

The data of course doesn't need to be inside a script tag - it just needs to be somewhere the script can read it. As for rendering the code - a simple DOM parser could be invoked after the javascripts were loaded and parsed to invoke the renderers. This could cause re-flows and re-paints - but by using block elements it need not be disruptive (certainly a lot less disruptive than stopping the initial rendering of the page to process the scripts.

So, first problem, where to put the data. 

It's trivial to add custom properties to the DOM (although there are some reports of issues in MSIE prior to version 8) I've seen successful implementations running in version 6 and 7. To be strictly correct you should declare in these in a custom DTD (as Facebook do with FBML) and not doing so can result in the browser rendering in quirks mode. But a bit more reading revealed another great feature in HTML5: you can add custom attributes to nodes without a custom DTD as long as the name is prefixed with 'data-' and the remainder is all in lower case. This is backwardly compatible with older browsers and avoids having to write a custom DTD. Some browsers will still be forced into quirks mode - but since I've spent rather a lot of time structuring the CSS and HTML to be responsive this should not be a big problem. Yeah!

I was about to start writing the code to implement the scanner, when I stopped and thought about this. I was creating a javascript framework. In order for it to have any value, I'd have to re-write the plugins I wanted to use and ideally get other plugin writers to join my bandwagon. And there was also a risk that the core developers of Dokuwiki might introduce new features that broke when the scripts were at the bottom.

A quick email to the Dokuwiki developers list showed that the issue of scripts at the bottom had already been raised, considered and binned. If this was the road to go down, then I'd have to spend some time writing nice well behaved and integrated code - and that would also mean taking time to learn a lot more about Dokuwiki internals.

A further issue with this approach is that I'd still have to refactor any plugins I wanted to use to fit into this new framework - once again I was losing sight of the main objective here - plug and play!

Was there another way to solve the problem? Yes! although it's not backwardly compatible, HTML5's pushstate allows you to load content using ajax, and set the location in the browser without transitioning the entire page. There's a really nice implementation of this in the pages on Github - which degrades nicely on older browsers. And the source code / documentation for PJAX is available (from GitHub of course). 

Now, all the custom modifications could go into the template rather than the core code (and it's only about 4 lines of code). The only caveat is that I don't have a browser which supports it on any computers at home so I'll need to do some upgrading before I can try it out for real.

I will try to go back to looking at scripts-at-the-bottom in Dokuwiki, but for now I've got a working solution that performs well and provides a base for some nice eye-candy effects.

Starting a new website

I've been volunteered as the webmaster for a club my daughter is involved with. Currently they have  a rather plain static html website and a free Invision Power Board forum. Increasingly the members are using Facebook as a means of communicating. This is far from ideal - the discussions and postings add a lot of value to the club and they are being given away gratis on Facebook. Further given the imminent adoption of the Instagram act, I felt it's important to maintain additional controls over users copyright.

I'm  big fan of free software and data - but in this case, the members are expecting a bundle of additional products, services and discounts. While it's a non-profit organization with a very small turnover there are significant costs involved. Hence I felt it was a good idea to provide a more realistic alternative to Facebook on the Club site, including a members only area.

The first problem was the hosting. The site was using the starter package from HeartInternet. Although it was PHP enabled, the amount of storage was very small - allowing users to upload their own content would have filled the quota very quickly, and there was no  database support. Upgrading the account to support the expected usage would have been exorbitantly expensive. So the first task was to find a hosting company.

While I can find lots of very cheap hosting companies claiming to operate in the UK - nearly all of them have their datacentres in the US/Canada. It's not that I've got anything against the former colonies - but I already have a site hosted in Florida, and although packets go back and forth at around half the speed of light (I measured it - how sad) the latency is a killer. I don't know if there's a paucity of UK service providers or if they're just not as good at SEO as the ones across the Atlantic, but this proved rather hard. There's also a large number of dummy review sites out there. But it did confirm that the prices on offer from HeartInternet were rather expensive. www.hostpapa.co.uk claim to offer UK web hosting with unlimited bandwidth, unlimited storage along with the usual PHP/MySQL  combo for under £2 a month! It's only when you dig a little deeper you find out they have no UK Data Centres. I emailed them to ask if they could provide hosting in the UK - their reply: "for securit [sic] purpose we do not disclose the locations of our data centers".

Sorry hostpapa: FAIL

So, for the record, I did manage to track down jomongee, x9internet, 123reg (whom I already use for DNS) and Ronald MacDonald (I'm guessing no relation to the burger chain). Ronald is offering virtual servers at very low prices - although I'm a bit of a control freak, and am confident I could setup the server at least as well as most ISPs, I just don't have the time to do this - I'll just stick with a basic shared host. Notably none of these companies plaster their front pages with bogus awards and certificates. I'll probably go with x9internet - whom offer a reasonable hosting package, but what really swung it for me was the refreshingly honest way they appear to do business.

So that's the hosting side sorted out. I just need some software to run it all. Ideally the system would provide:

• a CMS for publishing web pages - basic navigation, user authentication, public and member only pages
• a forum where users could vote up/down posts and acquire points (like stack overflow)
• WYSIWYG editor for forum
• image uploads on forum posts
• billing/payment for membership via Paypal or similar

Looking around the interent, I chose Pligg, Anahita, OpenOutReach and Oxwall to look at in more detail.

I decided to start with Oxwall because it was (IMHO) by far the prettiest. However....

1. Finding the requirements for the software (even what platform it runs on) from the website was very difficult. There's next to no technical documentation at all.

2. The requirements which are provided (in the zip file - no tarball) were a cause for concern

   - does not work with Suhosin

   - requires a cron job to be run *every* minute

3. The requirements were incomplete - the package relies heavily on mod_rewrite

4. The installation instructions don't work - assuming that the installer is suppoed to amend the URL in "Go to http://www.mysite.com/install to run the install script"  appropriately, I tried

http://localhost/mypath/install to get a 404 page

5. After running the installation script, the documentation goes on to tell the installer to run 'chmod 777' on all the directories created by the installation.a !!!!

6. Allowing the Oxwall .htaccess files to do anything they want:

    Options -All -Multiviews

    AllowOverride All

    Order deny,allow

    Deny from all

...Some progress - I got access forbidden (403) instead of 404 from the install URL.

7. The .htaccess file in the root directory contains:

Options +FollowSymLinks

RewriteEngine On

AddEncoding gzip .gz

AddEncoding gzip .gzip

  ForceType text/javascript

/FilesMatch>

  ForceType text/css

RewriteCond %{REQUEST_URI} !^/index\.php

RewriteCond %{REQUEST_URI} !/ow_updates/index\.php

RewriteCond %{REQUEST_URI} !/ow_updates/

RewriteCond %{REQUEST_URI} !/ow_cron/run\.php

RewriteCond %{REQUEST_URI} (/|\.php|\.html|\.htm|\.xml|\.feed|robots\.txt|\.raw|/[^.]*)$  [NC]

RewriteRule (.*) index.php

....i.e. it's not going to work unless it's installed in the root directory

There's also a lot of other things which are, at best strange here.

Creating a new vhost and moving the files into the DocumentRoot still resulted in a 403 error

8. After moving my current /var/www/html elsewhere and moving the Oxwall files back into the default webroot, I still got a 403 error at http://localhost/install

9. Pointing my browser at http://localhost/index.php, I finally got some output from Oxwall! It told me "Your hosting account doesn't meet the following requirements: zip PHP extension not installed"

- yes, another undocumented dependency.

10. installed php-zip, and got a configuration form (although the absence of any styling was a hint that the URL rewriting still wasn't working properly)

I know getting web paths sorted out is not easy - but I hate front controllers. But trying to create an arbitrary namespace using mod_rewrite is just asking for trouble. (BTW the Oxwall wiki runs on Dokuwiki - which I've written about before and is very cool).

While I could probably fix the problems and get a working site together (probably even fix the cron silliness) it's just not worth the effort - that the software packaging has been so sloppy, means that there are probably lots more gremlins in the code - I do not want the site pwned by the first script kiddy to come along.

It's a shame that someone has worked so hard to produce something which looks so nice and appears to have a lot of functionality in it, but makes so many basic errors.

Sorry Oxwall: FAIL 

Compiling PHP

Since I've been playing around with computers for more years than I care to remember, I used to be very familiar with the process of unpacking tarballs and compiling code from source. But to be honest it's not something I do very often these days: Most Linux distributions come with an extensive array of pre-compiled binaries: the package management software keeps me up to date with security patches: my C skills are a bit rusty: And life's just too short!

But recently I've been looking at LAMP performance in some detail. I was surprised to find the PHP on my workhorse desktop (PCLinuxOS 2012) had been compiled with no optimization and the resulting binary stripped. I should note that at the time I installed it, there was neither a 64-bit nor an AMD specific port of the installation, hence the OS build was more about compatibility than performance.

So I had a play around to see if there was any benefits at compile time.

PHP is a scripting language and a lot of it's functionality is implemented in extension libraries. Dynamically linking these at runtime does have a performance overhead (although with fastCGI and mod_php, since the code forks rather than loads, this shouldn't be too great). For most people the ability to choose which extensions are loaded at runtime (and thus trim the memory footprint) outweighs the small processing overhead of runtime linking. Unfortunately my test methodology didn't allow me to measure the actual impact of static vs dynamic. In the absence of a site to test and complex tools for HTTP load testing - ab would not cut the mustard - I was using the CLI SAPI where there would be a big performance drop which would not happen on a properly configured webserver.

To compare the different optimization levels I compiled PHP 5.3.22 using gcc 4.5.2 with O0, O2 and O3 then timed 'make test'. My test machine was a dual core AMD athlon

property	O0	O2	O3
CFLAGS	CFLAGS="-march=native -pipe" CXXFLAGS="${CFLAGS}"	CFLAGS="-march=native -O2 -pipe" CXXFLAGS="${CFLAGS}"	CFLAGS="-march=native -O3 -pipe" CXXFLAGS="${CFLAGS}"
Average (sys + usr) seconds	214.0	206.7	207.7
Std Dev (sys + usr) seconds	6.2	0.5	1.0
Max RSS (Kb)	569.8	569.8	570.0
Exe size Kb	6707.9	6880.8	7403.3
size of tripped exe	6257.5	6437.5	6973.6

I've not shown the results here, but I saw no measurable difference between the usr + sys times nor the max rss comparing the stripped and un-stripped binaries.
Interestingly the O3 optimization is very slightly, but measurably slower than O2. And O2 is around 5% faster than O0
The gain of 5% seems a little disappointing compared to metrics reported for other programs but I would expect to see to see greater gains for PHP code implementing (for example) encryption and image processing.

Crappy code

I spend way too much time dealing with other peoples' crappy code. To that end, a very long time ago, I wrote a quick script that would help me navigate around a PHP source file in order to work out what it's supposed to, and what it actually does.

It generates a call graph of the functions and methods in a PHP file, example shown below.



It had been sitting, festering on my HD for a while, and I thought "Hey, maybe someone else might find this helpful" so I uploaded it to phpclasses.org

So blow me, not only do they find it useful - but they want me to support the damn thing too! And then I get nominated for an award! I had a quick look at the source code. OMG - it's a mess!

So a couple of late nights were spent cleaning it up until the API is at least usable.

You can get the (cleaned up, slightly) source code here

Dokuwiki is smarter than I am

In a previous episode, I was moaning that I couldn't get my DokuWiki RSS feed to display using the RSS parser - it was my wetware which was not up to the job!

I'd configured this Dokuwiki to disallow any anonymous access - and the RSS generator (feed.php) was correctly showing all the items which the anonymous user was allowed to see - i.e.none!

I didn't realise this until I'd downloaded the latest stable DokuWiki and ported the hacks I've applied to it, then patched my installation - which obviously did not fix the problem.

In the meantime, I've created a new user with read access to the entire wiki - then changed my RSS wiki syntax to pass the auth details:

{{rss>http://www.mysite.com/dokuwiki/feed.php?u=anonymousRss&p=s3cr3t}}

Apologies to the DokuWiki team!

PHP and long running processes

It seems this question keeps coming up on the PHP newsgroups and, now that I've plugged into Stack Overflow - I keep seeing it their too:

How I do I start a PHP program which takes a long time to complete and how do I track its progress?

While these tend to attract lots of replies, they are usually wrong.

The first thing to consider is that you need to seperate the the thing which takes a long time from its initiation, the ongoing monitoring and whatever final reporting is required.

Since we're talking about PHP its fair to assume that in most cases, the initiation will be a PHP script running in a webserver. However this is not a good place to keep a long-running program.

1) webservers are all about turning around requests quickly - indeed most have failsafe mechanisms to prevent one request hanging about too long.

2) the webserver ties the request to both the execution of the script and to the client socket connection. Typically NOT keeping a browser window open somewhere waiting for the job to complete is an objective for the exercise. Although the dependence on the client connection can be reduced via ignore_user_abort() that was never its intended purpose.

3) long-running typically means it will have quite different resource requirements than a typical web page script - e.g. lots of file handles being opened and closed, more memory being consumed.

Most commentators come back with the suggestion of spawning a seperate thread of execution, either using fork or via the shell. The former obviously does not solve the webserver related issues if the interpreter is running as a module - you're just going to fork the webserver process. You've not solved any of the web related issues and created a whole lot of new ones.

You need to create a new process certainly.

The obvious type of process to create would be a standalone PHP interpreter to process the long running job. So is there a standalone interpreter available to the webserver? The prospective implementor would need to check (and whether the webserver runs as chroot). So lets assume there is, our coder writes:


print shell_exec('/usr/bin/php -q longThing.php &');

A brave attempt. However they will soon find that this doesn't behave as well as they expected and keeps stopping. Why? because all the process they created runs concurrently with the php which created it, it is still a child of that process. Now this is where it starts to get complicated. In our example above, the webserver process finishes with the users script immediately after it creates the new process - however it will probably hang around waiting to be assigned a new request to deal with. However at some point the controller for the webserver processes will decide to terminate it - either as a matter of policy because it has dealt with a certain number of requests (for apache: MaxRequestsPerChild) or because it has too many idle processes (apache's MinSpareServers). However the webserver process should not stop until all its child processes have terminated. How this is dealt with varies by operating system and of course, webserver. Regardless, the coder has created a situation which should not have arisen.

But on a Unix system there are lots of jobs which run independently for long periods of time. They achieve this by:

1) they are first started, say as pid 1234, and try to fork, say to pid 1235 after calling fork, pid 1234 exits
2) pid 1235 will become the daemon - it closes all its open fds including those for stdin, stdout and stderr
3) pid 1235 now calls setsid(), this dissociates this process from the tree of processes which led to its creation (and typically makes it a child of the 'init' process).

You can do all this in a PHP script, assuming you've got the posix and pcntl extensions. However in my experience its usually a lot simpler to ask an existing daemon to run the script for you:


print `echo /usr/bin/php -q longThing.php | at now`;

But how do you get progress information? Simple, just get your long running script to report its progress to a file or a database, and use another, web-based script to read the progress / show the final result.

Troubleshooting (updated Sep 2014)

Following on from the feedback I've received, there's a couple of things to check if it doesn't go according to plan.

The atd has its own permissions system implemented via /etc/at.allow and /etc/at.deny - see you man pages for more info.

On Redhat machines, the apache uid is configured with shell /bin/nologin - this will silently discard any jobs submitted to it, hence a more complete solution is:

putenv("SHELL=/bin/bash");
print `echo /usr/bin/php -q longThing.php | at now 2>&1`;

A note about systemd (updated Jun 2016)

The latest "feature" to be announced for systemd is that it will kill user processes when they logout. I don't currently have a machine running systemd to test what impact this might have, but since the apache user never logs in never mind logging out, and I recommend using atd to invoke the process (which is specifically designed to run a program regardless of whether the user is logged in) I don't expect a negative impact on my solution.