Anyone who knows me will not be surprised to hear that I think measuring user-experience and how users interact with your website is a very good idea. If you're in the business of trying to collect or analyse this information, then this post is addressed to you.
As I've often said, looking at the standard server-side logs can be very informative - but its only half the story. To get a better picture you need to go client-side. And that means Javascript. For many people / organisations, there just isn't the time or money to develop your own solution - and of course there are no end of vendors trying to flog their wares to you.
This post was prompted by a wasted hour investigating unusual patterns in referer stats. Where I work, phishing poses a very serious risk. Despite this, (and a large IT staff, dedicated security team. and an annual turnover well into the billions) there are no SPF records in our published DNS records! The referer stats for out customer facing website shows our logos appearing in lots of web-based email readers (including those from service providers who are known to validate SPF) - implying that it is more than just a risk. The is a shocking and absurd set of circumstances which I am still trying to resolve after 2 years.
However, that's not what this gripe is about.
This week I noticed a few referals from a very long URL starting with xxxxx.stcllctrs.com (where xxxxxx is the name of my employers parent organisation). The URL was not obviously an email reader. Dropping the URL into a browser returned a 200 response with no content. So I had a look at the root URL, http://xxxxx.stcllctrs.com/ Where I found the documentation for 'jsunpack' (http://jsunpack.jeek.org/dec/go) a tool 'designed for security researchers and computer professionals'. This is primarily a javascript code obfuscator. Interestingly, the URL for jsunpack seems to link to a form allowing people to report possible abuses of the tool - which has a record of its use at http://xxxxx.stcllctrs.com/ flagged as suspicious.
I then Googled for xxxxx.stcllctrs.com and found that our parents organisation had several references to this site, loading javascript files and NOSCRIPT content. Looking at the Javascript it was serving up, it was rather difficult to read (since it was obfuscated) but seemed to be doing strange things with cookies. The domain also appears in several ad blocking lists. Alarm bells started ringing!
Of course my employers make up for the quality of the security policy with the quantity of it - so I couldn't do proper whois lookup - but looking at tools on the web - this turned out to be a 16 bit subnet owned by Savvis.net. The name is registered with viatel.com. So both the netblock and DNS registration are effectively anonymous.
Obfuscated code, unusual URLs, cookie manipulation, anonymous hosting, greyware listings - DING DING DING!!!
Most of the whois services available online are provided by companies trying to sell registration services- the one I used initially did not provide any information about the registrant (and reformatted the content significantly so it looked like viatel was the registrant). But I eventually found another site (in Romania of all places!) which gave the registrant contact - speed-trap.com limited. This proved to be the Rosetta stone to unravelling what was really going on.
Speed-Trap appear to be a legitimate organisation providing web-usage monitoring services to companies. Surprisingly, they have a number of very high profile customers including direct.gov.uk, RBS, Axa and others. Yet they behave online like a script-kiddy - obfuscating their identity as well as code deployed to run in my browser, leaving other peoples hacking code
on their own website.
DirectGov have a link to their privacy policy on each and every page in their site (for the benefit of those from the colonies - DirectGov is the single, open access portal spanning all central government services in the UK). They clearly state they use javascript and cookies to record and analyse your usage of the site. They do not state that this information is processed by a third party. Indeed they go to unusual lengths to suggest that this information would only be shared with other bodies in extreme circumstances. RBS and http://www.axa.co.uk/privacy take a similar tack.
http://www.direct.gov.uk/en/SiteInformation/DG_020456
http://www.rbs.co.uk/global/f/privacy.ashx
http://www.axa.co.uk/privacy
From https://www.dephormation.org.uk/
"Intercepting, monitoring, eavesdropping, tapping communications requires legal authority, or consent from both parties to the communication."
Although there are some differences to BTs Phorm rollout (in that case, it was clear that Phorm were using the information for other purposes than just usage analysis) I find it very worrying that the UK government and several large financial institutions should be misleading their customers (or citizens) like this.
