One phrase which I’ve uttered on numerous occasions in the last two years is,
Recently, I’ve taken to putting more emphasis on the “integration is hard” part of this phrase. I keep seeing evidence, everywhere I look, of the importance of skills and systems that allow different variations and innovations to be easily incorporated back into wider systems. A new technology, by itself, is generally insufficient to make an impact in society. Lots of time-consuming system changes need to happen as well, before that technology can deliver “end-to-end” results.
In a small way, I’ve been seeing more evidence of this same point in the last few days, as I’ve been struggling to assemble a system of software to support a new application.
The application I’d like to support is the “IdeaTorrent” that powers, among other things, the Ubuntu Brainstorm site.
This software is an open source (free of charge) implementation of a voting and review system for ideas. I’d like to provide a version for my own purposes. The Ubuntu site helpfully has a link to the providers of the software, IdeaTorrent.
In principle, the installation instructions look simple enough:
- Use a webserver with PHP installed
- Install the database software PostgreSQL (v8.3)
- Install Drupal (content management software)
- Install IdeaTorrent
- Do some configuration.
In practice, things are proving tougher. The word “however” appears quite a few times in what follows…
The first webserver I obtained had a different database installed, MySQL. Although I heard from many people that, by various criteria, MySQL is preferable to PostgreSQL, the IdeaTorrent installation notes were clear that the system would work only if PostgreSQL were present.
However, with this webserver, I failed at the first hurdle. I found I couldn’t install any new software, since it was a “shared” server that other people were using too. That environment (not unreasonably) prohibited any user from gaining “root access” security permissions. If I wanted to change the components, I would need a different working environment.
I therefore purchased a more expensive hosting system. (I am hopeful of a refund for the first one – from the same supplier, GoDaddy). This system does allow installation of new software. And, to make things easier, it comes with PostgreSQL already installed. That took me to the task of installing Drupal. That task ended up taking me more than 24 hours elapsed time, as I went along a crash learning course about Linux file permissions, the BASH command shell, PHP, and Drupal itself. So far, so good.
Finally I installed IdeaTorrent, and reconfigured Drupal so that it recognised the IdeaTorrent modules.
However, lengthy error messages started getting displayed. Various functions were missing, so the system would not work. The functions in question were meant to be supplied by PostgreSQL. At this stage, I paid more attention to the part in the IdeaTorrent installation instructions that said “PostgreSQL 8.3 (Important! Won’t work below 8.3)”. On checking, I found out that the system provided by GoDaddy has PostgreSQL 8.2.9.
8.2.9 looks numerically close to 8.3, but clearly it’s not close enough.
Further investigation showed me that Linux has an impressive looking system for assisting with upgrades of components. I followed instructions that I found spread out over the Internet – including the quaintly titled article “An Almost Idiot’s Guide to PostgreSQL YUM“.
These instructions let me to downloading a small “.RPM” file onto the server, and then invoking a command called “YUM“. (Apparently YUM stands for “Yellowdog Updater, Modified”!) It looked like it was working.
However, when I started up the system again, I saw that the version of PostgreSQL that was running was still 8.2.9.
Another 24 hours elapsed – with lots of red herrings along the way. I eventually realised that some of the online links had misled me to pick the wrong .RPM file. (The one I was trying to install was actually, despite appearances, an even older version.) Finally I found the right file. This time, when I ran the YUM magic, lots more things happened.
However, these changes broke the admin “control panel” interface when GoDaddy provided, in order to administer databases. Without this interface, the remainder of the installation (as required to set up Drupal) would probably be impossible.
Not for the first time, I went back to square one (GoDaddy have a handy service whereby they will return the whole system to its inital “provisioned” state). This time, I was more careful in how I did the upgrades. This time, the control panel interface kept working fine.
However, this time, although the functions were shown whereby I could administer the databases, the functions all told me:
Admin password is out of sync. Database management will be unavailable until the admin password is changed.
However, everything I try to do to enter (or redefine) the “admin password” fails. My current guess is that the handling of admin passwords somehow changed between PostgreSQL v8.2 and PostgreSQL v8.3.
At the moment, I’m stuck again. Perhaps I’ll figure out what password I ought to be typing in. Or perhaps I’ll have the whole system reprovisioned yet again (for about the fifth time) and do the upgrades even more carefully this time. (Any advice from readers of this blog will be gratefully received!)
I see my trials and tribulations of the last few days as an example of how it’s hard to alter individual components of a complex software system. These components can have all kinds of dependencies on each other – explicit dependencies, implicit dependencies, data format dependencies, obscure dependencies, bug dependencies, and so on.
My takeaway isn’t to assign any blame to any particular system. My takeaway is that:
- Anyone planning to alter components in a complex software system should set aside plenty time to debug integration issues that arise along the way;
- Anyone embarking on such an endeavour should be ready to call in expert advice from people who have significant prior experience in that same task.
Indeed, integration is hard. However, that’s a reason for all of us to design systems that are “integration friendly”.
Added later
Since writing this blogpost, I’ve now understood why the “control panel” phpPgAdmin application was failing to accept passwords, and the problem is solved.
It turns out that a clean installation of PostgreSQL specifies (via the file pg_hba.conf) that user access is checked via an “ident” (identification) system that does not use passwords. General advice is that this should be changed to use passwords instead. Changing the word “ident” to “md5” in the above-mentioned configuration file does the trick – and phpPgAdmin is now working for me again.
dw2 
Hi David,
Integration is even harder on Red Hat-derivative boxes, in my experience (and that of many sysadmins that I trust). You could do a lot worse than getting yourself a virtual machine from someone like Bytemark (http://www.bytemark.co.uk/hosting/virtual_machines), with Ubuntu on it. The Debian packaging system (that Ubuntu also uses) is just about the best, and happily deals with dependencies and upgrades in a very elegant fashion. I’ve been running Debian-based systems for about a decade, and never looked back.
The problem with web-based admin solutions is that, whilst they hide the complexity, they also provide a layer of abstraction that can make integration work such as your project more difficult than it needs to be. With that in mind, I recommend you fire up a command-line and dig into the documentation at http://www.postgresql.org/docs/ … and ask questions on twitter, or by all means drop me an email and I’d be happy to help where possible.
Comment by Andrew Savory — 20 December 2009 @ 5:55 pm
Thanks Andrew,
The Bytemark service looks attractive. I especially like the fact that it offers Ubuntu – since I can see that Ubuntu puts a particular emphasis on ease of installation and configuration.
At the moment, I’m stuck in the situation of having spent some money with one supplier (GoDaddy – which doesn’t offer Ubuntu) and it’s still possible that everything will start working fine again. I’ve seen periods of slow progress interspersed with periods of very fast progress, as I’ve learned more about how the system works.
I expect I’ll be asking more questions on Twitter – thanks for the offer of assistance!
Comment by David Wood — 20 December 2009 @ 6:08 pm
I’ll assume that your second takeaway ‘ready to call in expert advice’, refers to a general internet trawl. “Google is your friend”.
In many cases, the answer may already be out there. So, specifically, “the person who experienced the issue previously, and posted up a solution is, at that moment, your very best friend”.
In some cases, you’ll be a pioneer. So, after battling a problem for 24 hours the temptation to press on without further delay once a solution is found is great. However, as good citizens of tech communities, sharing problems hit and how they were resolved will give something back.
So here’s a message with a suitable Christmas message: Integration is hard: let’s help each other by sharing what we learn.
Comment by Nigel Street — 20 December 2009 @ 6:17 pm
Hi Nigel,
I agree – Google web searches have thrown up lots of useful info and ideas. I’ve been surprised, when typing in a long error message, how many hits that precise message gets. Of course, the hits vary in quality and in relevance. Several of my “red herring” detours were sparked by me trying to apply advice, found on the web for one circumstance, for my slightly similar circumstance.
However the phrase I mentioned at the end of my tale – “Admin password is out of sync. Database management will be unavailable until the admin password is changed” – gave me precisely zero hits on Google. That was the reason I aired it here, hoping for advice.
There are occasions when Google web searches are probably NOT the most cost effective approach. These are circumstances when it might make better financial sense to pay for advice from an expert consultant. That may or may not apply for my present project, but it does apply in lots of smartphone development projects that I’ve seen over the years. It’s been a frequent major mistake for companies to under-estimate (1) the pain of integration, (2) the potential rapid benefits of engaging suitably experienced consultants.
Comment by David Wood — 20 December 2009 @ 6:46 pm
I went through a similar experience recently when I tried to setup an Apache/PHP/Python/PostgreSQL/Django/MediaWiki stack. It was incredibly frustrating, but also instructive, and did at least provide several web-developer friends with much amusement. Whilst I can’t say that I didn’t assign any blame, I did also have a few more useful takeaways:
(1) Software-as-a-service makes a lot of sense; especially in the web world where there are so many different components, combinations, and updates.
(2) I’ve finally understood why no-one could ever explain to me why this stuff always takes so long. There’s no one big thing, or abstractable step that’s hard, it is simply a littany of things not quite working properly; i.e. it’s debugging.
Comment by Antony — 20 December 2009 @ 6:24 pm
Hi Antony,
MediaWiki is one of the things on my list to setup in due course – after the IdeaTorrent project is off the ground.
I may get back to you for some practical tips on MediaWiki setup…
Comment by David Wood — 20 December 2009 @ 7:04 pm
I’m interested to hear how IdeaTorrent goes as I quite like the idea of it being nice and simple. It seems like people at work like Spigit, but others also like IdeaTorrent.
As far as hosting goes we are using Amazon Web Services more and more and this is working out well for hosting and you have almost full control. I guess you can see why sometimes the IT teams can’t deliver things as fast as you’d expect sometimes as we get “dependency hell” all the time. I’ve had it sometimes even worse – one package wants version x
Comment by Ian McDonald — 20 December 2009 @ 6:27 pm
Hi Ian,
Like you, I’ll be interested to see how IdeaTorrent and Spigit compare. I think Spigit has a lot of potential too – but not being open source, it’s probably considerably more expensive…
I’ll take a look at Amazon Web Services
Comment by David Wood — 20 December 2009 @ 7:03 pm
Shaun Phillips has set up Mediawiki recently on Amazon Web Services…
Comment by Ian McDonald — 21 December 2009 @ 9:25 am
A few years back I helped out on documenting install experiences for OpenXchange, which was a nightmare. Each different flavour of linux had different install instructions and required specific (outdated) versions of things like BerkeleyDB and Postgresql, which took days to track down. If you’re going into the unknown, then you’re going to have to do the install at least twice. I would not have even tried on a hosted server! These days at home I fire up an empty VM and build on that to test, or use something like susestudio.com. Then you can host the vmware image on somewhere like here http://www.vmware-hosting.com/
Comment by James Melbow — 24 December 2009 @ 9:50 am
Hi James,
I ended up asking the server hosts to reprovision the system at least five times before it all worked.
It’s looking good now (and functioning well).
However, now that there’s lots of data in the system, I realise that reprovisioning it again will be much harder. That makes me pause before attempting any new large installation process (eg for MediaWiki).
On the plus side, I think I can now describe myself as a Linux server nano-expert: I’ve got a good understanding of at least 10^(-9) of the issues that can arise on a Linux server…
Comment by David Wood — 24 December 2009 @ 12:30 pm
Hi David,
I wonder if you managed to catch Electric Dreams while it was on TV? During the show they took a family back to the technology and culture of the 1970s and then advanced them a year each day up to the present day.
One of the interesting observations was how technology that arrived in the mid to late 80s (such as video recorders or early PCs) were incredibly complex to use often arriving with thick manuals. By the mid 90s the same technology had greatly simplified user interfaces with sometimes two or three buttons where there had been a whole raft before.
I’m hoping that as we progress into the 2010s we’re going to start seeing the same happening in technology, specifically software, integration. It’s well recognised to be a very importantly (and, yes, highly complex) problem but a lot of people are looking into it and I have good hopes for it.
Of course the natural counter to this is that increasingly people are becoming more and more dependent on open source projects and the people who work on those are less invested in integration issues. We can only hope that service industries pop up in the market niches that leaves to plug those integration difficulties holes.
Comment by David Durant — 25 December 2009 @ 2:35 am
Hi David,
Sadly, I missed the Electric Dreams series, but I’ve heard people talking about it.
I’m sure you’re right that increased ease of use will be a strong characteristic of successful products of the near future. This covers, not only ease of use for consumers, but also ease of development (ease of use for developers) and ease of customisation-and-integration (ease of use for systems creators).
It’s my view that, in all types of software – including open source – the platforms that are successful will (increasingly likely) be the ones that provide first-class ease of use. This puts a strong requirement on the platform designers to ensure that customisation-and-integration can happen smoothly.
Comment by David Wood — 25 December 2009 @ 4:15 pm