Starting a new website

I've been volunteered as the webmaster for a club my daughter is involved with. Currently they have  a rather plain static html website and a free Invision Power Board forum. Increasingly the members are using Facebook as a means of communicating. This is far from ideal - the discussions and postings add a lot of value to the club and they are being given away gratis on Facebook. Further given the imminent adoption of the Instagram act, I felt it's important to maintain additional controls over users copyright.

I'm  big fan of free software and data - but in this case, the members are expecting a bundle of additional products, services and discounts. While it's a non-profit organization with a very small turnover there are significant costs involved. Hence I felt it was a good idea to provide a more realistic alternative to Facebook on the Club site, including a members only area.

The first problem was the hosting. The site was using the starter package from HeartInternet. Although it was PHP enabled, the amount of storage was very small - allowing users to upload their own content would have filled the quota very quickly, and there was no  database support. Upgrading the account to support the expected usage would have been exorbitantly expensive. So the first task was to find a hosting company.

While I can find lots of very cheap hosting companies claiming to operate in the UK - nearly all of them have their datacentres in the US/Canada. It's not that I've got anything against the former colonies - but I already have a site hosted in Florida, and although packets go back and forth at around half the speed of light (I measured it - how sad) the latency is a killer. I don't know if there's a paucity of UK service providers or if they're just not as good at SEO as the ones across the Atlantic, but this proved rather hard. There's also a large number of dummy review sites out there. But it did confirm that the prices on offer from HeartInternet were rather expensive. www.hostpapa.co.uk claim to offer UK web hosting with unlimited bandwidth, unlimited storage along with the usual PHP/MySQL  combo for under £2 a month! It's only when you dig a little deeper you find out they have no UK Data Centres. I emailed them to ask if they could provide hosting in the UK - their reply: "for securit [sic] purpose we do not disclose the locations of our data centers".

Sorry hostpapa: FAIL

So, for the record, I did manage to track down jomongee, x9internet, 123reg (whom I already use for DNS) and Ronald MacDonald (I'm guessing no relation to the burger chain). Ronald is offering virtual servers at very low prices - although I'm a bit of a control freak, and am confident I could setup the server at least as well as most ISPs, I just don't have the time to do this - I'll just stick with a basic shared host. Notably none of these companies plaster their front pages with bogus awards and certificates. I'll probably go with x9internet - whom offer a reasonable hosting package, but what really swung it for me was the refreshingly honest way they appear to do business.

So that's the hosting side sorted out. I just need some software to run it all. Ideally the system would provide:

• a CMS for publishing web pages - basic navigation, user authentication, public and member only pages
• a forum where users could vote up/down posts and acquire points (like stack overflow)
• WYSIWYG editor for forum
• image uploads on forum posts
• billing/payment for membership via Paypal or similar

Looking around the interent, I chose Pligg, Anahita, OpenOutReach and Oxwall to look at in more detail.

I decided to start with Oxwall because it was (IMHO) by far the prettiest. However....

1. Finding the requirements for the software (even what platform it runs on) from the website was very difficult. There's next to no technical documentation at all.

2. The requirements which are provided (in the zip file - no tarball) were a cause for concern

   - does not work with Suhosin

   - requires a cron job to be run *every* minute

3. The requirements were incomplete - the package relies heavily on mod_rewrite

4. The installation instructions don't work - assuming that the installer is suppoed to amend the URL in "Go to http://www.mysite.com/install to run the install script"  appropriately, I tried

http://localhost/mypath/install to get a 404 page

5. After running the installation script, the documentation goes on to tell the installer to run 'chmod 777' on all the directories created by the installation.a !!!!

6. Allowing the Oxwall .htaccess files to do anything they want:

    Options -All -Multiviews

    AllowOverride All

    Order deny,allow

    Deny from all

...Some progress - I got access forbidden (403) instead of 404 from the install URL.

7. The .htaccess file in the root directory contains:

Options +FollowSymLinks

RewriteEngine On

AddEncoding gzip .gz

AddEncoding gzip .gzip

  ForceType text/javascript

/FilesMatch>

  ForceType text/css

RewriteCond %{REQUEST_URI} !^/index\.php

RewriteCond %{REQUEST_URI} !/ow_updates/index\.php

RewriteCond %{REQUEST_URI} !/ow_updates/

RewriteCond %{REQUEST_URI} !/ow_cron/run\.php

RewriteCond %{REQUEST_URI} (/|\.php|\.html|\.htm|\.xml|\.feed|robots\.txt|\.raw|/[^.]*)$  [NC]

RewriteRule (.*) index.php

....i.e. it's not going to work unless it's installed in the root directory

There's also a lot of other things which are, at best strange here.

Creating a new vhost and moving the files into the DocumentRoot still resulted in a 403 error

8. After moving my current /var/www/html elsewhere and moving the Oxwall files back into the default webroot, I still got a 403 error at http://localhost/install

9. Pointing my browser at http://localhost/index.php, I finally got some output from Oxwall! It told me "Your hosting account doesn't meet the following requirements: zip PHP extension not installed"

- yes, another undocumented dependency.

10. installed php-zip, and got a configuration form (although the absence of any styling was a hint that the URL rewriting still wasn't working properly)

I know getting web paths sorted out is not easy - but I hate front controllers. But trying to create an arbitrary namespace using mod_rewrite is just asking for trouble. (BTW the Oxwall wiki runs on Dokuwiki - which I've written about before and is very cool).

While I could probably fix the problems and get a working site together (probably even fix the cron silliness) it's just not worth the effort - that the software packaging has been so sloppy, means that there are probably lots more gremlins in the code - I do not want the site pwned by the first script kiddy to come along.

It's a shame that someone has worked so hard to produce something which looks so nice and appears to have a lot of functionality in it, but makes so many basic errors.

Sorry Oxwall: FAIL 

Whither MSIE

Looking at my stats here on blogger.com, MSIE traffic has dropped to just 5% - behind Chrome, Firefox and Safari. Although it's fallen a long way from the market share MSIE had even 5 years ago, this is still a lot more skewed than I see on the real sites I manage.

I guess I need to start showing those MS users a bit more love (they do need it).

Compiling PHP

Since I've been playing around with computers for more years than I care to remember, I used to be very familiar with the process of unpacking tarballs and compiling code from source. But to be honest it's not something I do very often these days: Most Linux distributions come with an extensive array of pre-compiled binaries: the package management software keeps me up to date with security patches: my C skills are a bit rusty: And life's just too short!

But recently I've been looking at LAMP performance in some detail. I was surprised to find the PHP on my workhorse desktop (PCLinuxOS 2012) had been compiled with no optimization and the resulting binary stripped. I should note that at the time I installed it, there was neither a 64-bit nor an AMD specific port of the installation, hence the OS build was more about compatibility than performance.

So I had a play around to see if there was any benefits at compile time.

PHP is a scripting language and a lot of it's functionality is implemented in extension libraries. Dynamically linking these at runtime does have a performance overhead (although with fastCGI and mod_php, since the code forks rather than loads, this shouldn't be too great). For most people the ability to choose which extensions are loaded at runtime (and thus trim the memory footprint) outweighs the small processing overhead of runtime linking. Unfortunately my test methodology didn't allow me to measure the actual impact of static vs dynamic. In the absence of a site to test and complex tools for HTTP load testing - ab would not cut the mustard - I was using the CLI SAPI where there would be a big performance drop which would not happen on a properly configured webserver.

To compare the different optimization levels I compiled PHP 5.3.22 using gcc 4.5.2 with O0, O2 and O3 then timed 'make test'. My test machine was a dual core AMD athlon

property	O0	O2	O3
CFLAGS	CFLAGS="-march=native -pipe" CXXFLAGS="${CFLAGS}"	CFLAGS="-march=native -O2 -pipe" CXXFLAGS="${CFLAGS}"	CFLAGS="-march=native -O3 -pipe" CXXFLAGS="${CFLAGS}"
Average (sys + usr) seconds	214.0	206.7	207.7
Std Dev (sys + usr) seconds	6.2	0.5	1.0
Max RSS (Kb)	569.8	569.8	570.0
Exe size Kb	6707.9	6880.8	7403.3
size of tripped exe	6257.5	6437.5	6973.6

I've not shown the results here, but I saw no measurable difference between the usr + sys times nor the max rss comparing the stripped and un-stripped binaries.
Interestingly the O3 optimization is very slightly, but measurably slower than O2. And O2 is around 5% faster than O0
The gain of 5% seems a little disappointing compared to metrics reported for other programs but I would expect to see to see greater gains for PHP code implementing (for example) encryption and image processing.

How not to manage an IT dept

I think enough water has flowed under the bridge for me to talk about my experiences as IT manager in a small start up company where I worked until 6 years ago. The characters and events described herein are entirely genuine; only the facts have been changed to protect the innocent.

Having previously worked as Computer Services Manager for a UK retailer, the job sounded like everything I could have wished for - head of IT for a small company producing and selling digital media (that's ringtones for the civilians out there). Hands-on IT work, developing software. Hence I  was happy to accept the job at a significant salary cut from my previous job where I was increasingly just managing contracts.

I joined the company near the end of it's first year of trading when it had a turnover of just £80,000 but promising things were afoot. In the following year, it turned over £2M, then £4M, and was on target for £6M when I left 30 months later. But within a year of my departure it had stopped trading and subsequently wet into liquidation. I'd like to think that I had something to do with its success while I was there. But for now I'll start at the beginning.

In terms of infrastructure they were unusual in doing their own hosting. The implementation of the infrastructure was remarkably well thought out. The code which ran their websites and PRS IVR systems however left a lot to be desired. The code had been delivered an East-European individual who operated through a chain of shell companies. He was then sub-contracting the work to the cheapest bidder. Hence the code was all in different styles and not very sophisticated. However, to be fair, since each page (PHP script) had no dependencies on any other bit of code at least any issues were well isolated, and the code was commented in English.

I was expected to take on the management of the infrastructure and some of the development work, with the plan that I would build a small team of in-house developers. Looking after the servers was easy enough - I made relatively few changes to the configuration. But based on my previous experience, I did spend a lot more time defining and documenting processes than my employers were really comfortable with, although anyone with a background in CMMI / ITIL would probably have said that I didn't go nearly far enough. Along side this I was developing new functionality for the systems and trying to recruit more IT staff.

As this was after the internet bubble had burst, people describing themselves as web developers were ten-a-penny - and that's about all I was able to offer them in a salary. But I really needed competent software engineers / analyst programmers - that the sites were mostly developed in PHP was merely an implementations detail. Hence my first big problem was how to recruit and retain people with real programming skills at half the salary which was available elsewhere.

The story will continue in a future post....

Wifi fettling - rt2870sta / rt5370sta confusion

I'm redecorating my dining room - which is also where my computer lives. Part of this involved re-siting my computer. Rather than run a cable across the room for my internet connection I thought I'd use wireless. I had an old 3com card in my parts drawer - but it appears my kids / the dog had been playing with it. I've always been very happy with 3com kit, but since money is rather tight I decided to go with a basic USB adapter. A quick Google and I found a few people saying that this one worked with Linux.

It's tiny!

It comes with a mini cd which includes source code for Linux drivers! Yeah! Reading the docs, it described ow to build and install the RT2870STA kernel module. But my PCLinuxOS 2012 installation  comes with the driver and firmware. So I spent an hour or so trying to get it to work to no avail.

Ho hum, let's try the supplied driver. No configure - just make and make install. But it didn't generate a rt28750sta.ko, instead I got a rt5370sta.ko - it seems that the documentation bundled with the code is out of date. Really I should have gotten the hint when I checked the device with lsusb:

Bus 001 Device 005: ID 148f:5370 Ralink Technology, Corp.

After doing a modprobe I was up and running.

I've disabled the ONBOOT setting for my wired ethernet and enabled it for the new device via drakconf and it all works perfectly (even little green bars in the system tray).

10 feet from the router I get a good signal. I don't know how well it would work at a greater distance with such a small antenna (but our phones get a signal from quite a distance away). It's not really a problem for this machine since it's never going to be very far from the access point.


Link Quality=100/100  Signal level:-52 dBm  Noise level:-83 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

(note to self must remember to add mac address to admin ACL and change the forwarding on access point).

I also got a new desk on Amazon. It's compact and (importantly) feels very solid. The only downsides were that it was a bit tricky to assemble and with my PC on the bottom shelf, the fan is rather noisy (shelf resonates a bit).

Paypal? Privacy? Competence? Apparently not.

Despite my (repeated) warnings, the implementation of the new cookie legislation caught our technical gurus with their figurative pants down. Some disclaimers were cobbled together quickly and posted up.

But it seems we're doing a much better job than Paypal.

Don't get me wrong - I know that (done properly) cookies greatly enhance the security of interaction - indeed I'd be very wary of a payment processing system that didn't implement cookies - but then I'm also very wary of payment processors whom don't implement the law.

Bought some stuff today from Paypal - the first time I'd used Paypal for several months - no mention of cookies. It actually dropped no less than 40 cookies on my browser! FFS! No mention of cookies let alone an opt-in. And while the majority of the cookies were session cookies, it's interesting to note that Paypal DOES NOT END the session at the completion of payment.  That's right. after paying for goods via paypal and returning to the orignal site, you are still logged into Paypal!

"We use cookies written with Flash technology to help prevent fraud "

 Are the criminal classes still that incompetent that don't know how to get around evercookie?

To top it all, it seems that Paypal now stuff more data into their cookies than they are willing to consume. after I wrote this post I went back to double check if there was any effort to comply with The Privacy and Electronic Communications (EC Directive) Regulations 2003, only to get failures accessing https://www.paypal.com/ with an error message indicating that I was returning more cookies than it could cope with:

 (the green bar was added by me to redact my data). It seems that someone thought it a good idea to transfer transactional information via cookies.

Just now the news is full of the arrogance and incompetence of the financial institutions - to a certain extent I have reserved judgement. But increasingly it appears that the IT operations are run by the CEOs nephew who once read a book about programming - OK maybe I exaggerate. There are some very competent and brilliant people out there, some of whom I'm acquainted with - it just seems that the less able IT people of my ken seem to be in the well paid jobs. Grrrrrr!

Very Non Cooperative (really)

The powers that be at my work have decided to revisit the issue of BYOD / external access. I'd previously implemented solutions at two previous employers so thought I'd be able to come up with something suitable here.

In both the previous exercises, I'd arrived at the conclusion that the easiest way to implement this was in terms of services - and relying on open protocols such as HTTP, SMTP, telnet etc (before you reach for your guns, the telnet thing was due to a requirement for dg200 terminal emulation- and I couldn't get a client which would run over ssh - so the telnet was encapsulated in SSL using stunnel). However this project is a bit different for various reasons - not least a very real concerns that the users will leave their computers on trains and park benches. While others in the office had previously come up with horribly complicated encryption schemes - this is a nightmare to support. So after a good deal of thought I realized I could solve a whole load of problems at one stroke by using a remote window / desktop protocol. VNC was the obvious choice due to the wide availability of clients. And I'd previously implemented lots of VNC servers on Linux and on MSWindows - it was always a no-brainer. I did have cunning plans for dealing with small screen real estate, keyboard-less devices etc - but best to start with little steps.

So I fired up synaptic on my PCLinuxOS desktop and installed TightVNC server and a client. Running the server from the command line, it works (but obviously no window manager, and the simple standalone VNC auth). So I set it ll up to run through xinetd (similar to this), re-configure kdm and xfs, fire up a client, enter my username and password - "the server has closed the connection". Check firewall - no problem. Check logs - nothing there. Double check my config changes - all OK. Just to make sure I do clean reboot. Still not working. Check the man page - what's happenned to all the X integration stuff? Gone! No xdm support!

Next I tried RealVNC direct from the RealVNC website - got a licence - read the docs....no more inetd support? WTF? The only logical reason I can think of for this is that they want to enforce their licence terms. Still, I could live with that for the POC - who knows, we might even end up paying for licences for the service - in return for support. But every time I connected to localhost, I got "user not recognised or password was blank". RealVNC do say on their website that this can be an issue on some versions of Linux - and the solution is to disable PAM authentication (a bit weird since they say elsewhere that it is not available in the 'free' version). So I updated the configs, restart the server, to no avail. Tried various tweaks and fettling. Checked the firewall. Nothing. Oh, and there's no 'uninstall' functionality - so had to reverse engineer the installation to clean it up.

Have I got dumber with old age or is this another case where a good product has turned into bloatware?

Aaaaarrrrghhhh!

(some updates added as comments)