Whatever happened to scripting?

Don't you just love "enterprise" tools. Most of the ones I've had the pleasure of working with seem to have been around a very long time, belonged to companies which have progressively been bought over by bigger and bigger corporations, have been developed by different teams with different methodologies and coding styles. It's a miracle they work at all.

But one common theme, and one that they all tend to be very good at is making sure that they are at the top of the food chain in their field. Most provide good downstream connectivity, collecting data from all sorts of different sources. But it it is exceedingly difficult to integrate them to upstream components - for reporting, user management, logging etc.

The latest problem was to get data out of Microstrategy. It has a SOAP interface for invoking reports remotely. But try finding any documentation for it. It also has a "simple" HTTP based interface (where report definitions are specified in the URL). Again with no available documentation. I asked the Data warehouse team whether they knew anything about these interfaces. Answers ranged from "What's SOAP?" to "no". It has a scheduler for running reports - can't we just dump these on a filesystem somewhere?....apparently not.  So how can you get information out in a machine readable form? We can send an email.

Great. Email I can do. So I fire up putty and start hacking together a script to get the file out of an email and hand it over to my app. Fetchmail -> procmail -> metamail. Simples.

....only metamail is not available in RHEL. I've previously blogged about mail processing in RH. I really don't want to write my own MIME handler. While there's lots of PHP implementations on the internet, you need to look hard to find the ones which are robust and well written. But even then the parsing is done by loading the entire message into memory. Not very handy if the message is 100Mb+ and using PHP.

I could download metamail and compile it....but looking around the internet, it doesn't seem to have been actively maintained. Indeed there hadn't been any significant changes since I'd sent in some bug reports about 15 years ago! Investigating further I found ripmime which does what I need. So a quick security scan and it seems ideal.

This might be a good point to describe what I looked for in checking its provenance.

• It seems to be bundled in several Linux distros (i.e. other people like it and are using it).
• Older versions have some CVEs logged against it - now fixed. This is good on several counts - again it shows that people are using it and finding security problems and the security problems are getting fixed.
• the other products flagged in the same CVEs put it in respectable company
• I went to the origin website for the tarball - found other interesting security stuff related to email handling.
• scanned the source for anything that might indicate an alternate function (fork(), exec*(), system(), socket stuff)
• looked to see if the code was aware of obvious attacks (such as Content-Disposition: attachment; filename="/etc/passwd";).

All good. It would have taken me a very long time to implement all this myself.

Really RedHat, ripMIME should be part of RHEL!

I know Linux is now mainstream - but that doesn't mean I want a complex black box which I can't diagnose or re-purpose. If I wanted that I would have bought MS Windows! (are you reading this KDE PIM developers, systemd developers). 

Fortunately it's trivial to build ripmime (no dependencies other than glibc and iconv).

Project back on track. Thank you Paul.