One phrase which I’ve uttered on numerous occasions in the last two years is,
Recently, I’ve taken to putting more emphasis on the “integration is hard” part of this phrase. I keep seeing evidence, everywhere I look, of the importance of skills and systems that allow different variations and innovations to be easily incorporated back into wider systems. A new technology, by itself, is generally insufficient to make an impact in society. Lots of time-consuming system changes need to happen as well, before that technology can deliver “end-to-end” results.
In a small way, I’ve been seeing more evidence of this same point in the last few days, as I’ve been struggling to assemble a system of software to support a new application.
This software is an open source (free of charge) implementation of a voting and review system for ideas. I’d like to provide a version for my own purposes. The Ubuntu site helpfully has a link to the providers of the software, IdeaTorrent.
In principle, the installation instructions look simple enough:
- Use a webserver with PHP installed
- Install the database software PostgreSQL (v8.3)
- Install Drupal (content management software)
- Install IdeaTorrent
- Do some configuration.
In practice, things are proving tougher. The word “however” appears quite a few times in what follows…
The first webserver I obtained had a different database installed, MySQL. Although I heard from many people that, by various criteria, MySQL is preferable to PostgreSQL, the IdeaTorrent installation notes were clear that the system would work only if PostgreSQL were present.
However, with this webserver, I failed at the first hurdle. I found I couldn’t install any new software, since it was a “shared” server that other people were using too. That environment (not unreasonably) prohibited any user from gaining “root access” security permissions. If I wanted to change the components, I would need a different working environment.
I therefore purchased a more expensive hosting system. (I am hopeful of a refund for the first one – from the same supplier, GoDaddy). This system does allow installation of new software. And, to make things easier, it comes with PostgreSQL already installed. That took me to the task of installing Drupal. That task ended up taking me more than 24 hours elapsed time, as I went along a crash learning course about Linux file permissions, the BASH command shell, PHP, and Drupal itself. So far, so good.
Finally I installed IdeaTorrent, and reconfigured Drupal so that it recognised the IdeaTorrent modules.
However, lengthy error messages started getting displayed. Various functions were missing, so the system would not work. The functions in question were meant to be supplied by PostgreSQL. At this stage, I paid more attention to the part in the IdeaTorrent installation instructions that said “PostgreSQL 8.3 (Important! Won’t work below 8.3)”. On checking, I found out that the system provided by GoDaddy has PostgreSQL 8.2.9.
8.2.9 looks numerically close to 8.3, but clearly it’s not close enough.
Further investigation showed me that Linux has an impressive looking system for assisting with upgrades of components. I followed instructions that I found spread out over the Internet – including the quaintly titled article “An Almost Idiot’s Guide to PostgreSQL YUM“.
These instructions let me to downloading a small “.RPM” file onto the server, and then invoking a command called “YUM“. (Apparently YUM stands for “Yellowdog Updater, Modified”!) It looked like it was working.
However, when I started up the system again, I saw that the version of PostgreSQL that was running was still 8.2.9.
Another 24 hours elapsed – with lots of red herrings along the way. I eventually realised that some of the online links had misled me to pick the wrong .RPM file. (The one I was trying to install was actually, despite appearances, an even older version.) Finally I found the right file. This time, when I ran the YUM magic, lots more things happened.
However, these changes broke the admin “control panel” interface when GoDaddy provided, in order to administer databases. Without this interface, the remainder of the installation (as required to set up Drupal) would probably be impossible.
Not for the first time, I went back to square one (GoDaddy have a handy service whereby they will return the whole system to its inital “provisioned” state). This time, I was more careful in how I did the upgrades. This time, the control panel interface kept working fine.
However, this time, although the functions were shown whereby I could administer the databases, the functions all told me:
Admin password is out of sync. Database management will be unavailable until the admin password is changed.
However, everything I try to do to enter (or redefine) the “admin password” fails. My current guess is that the handling of admin passwords somehow changed between PostgreSQL v8.2 and PostgreSQL v8.3.
At the moment, I’m stuck again. Perhaps I’ll figure out what password I ought to be typing in. Or perhaps I’ll have the whole system reprovisioned yet again (for about the fifth time) and do the upgrades even more carefully this time. (Any advice from readers of this blog will be gratefully received!)
I see my trials and tribulations of the last few days as an example of how it’s hard to alter individual components of a complex software system. These components can have all kinds of dependencies on each other – explicit dependencies, implicit dependencies, data format dependencies, obscure dependencies, bug dependencies, and so on.
My takeaway isn’t to assign any blame to any particular system. My takeaway is that:
- Anyone planning to alter components in a complex software system should set aside plenty time to debug integration issues that arise along the way;
- Anyone embarking on such an endeavour should be ready to call in expert advice from people who have significant prior experience in that same task.
Indeed, integration is hard. However, that’s a reason for all of us to design systems that are “integration friendly”.
Since writing this blogpost, I’ve now understood why the “control panel” phpPgAdmin application was failing to accept passwords, and the problem is solved.
It turns out that a clean installation of PostgreSQL specifies (via the file pg_hba.conf) that user access is checked via an “ident” (identification) system that does not use passwords. General advice is that this should be changed to use passwords instead. Changing the word “ident” to “md5” in the above-mentioned configuration file does the trick – and phpPgAdmin is now working for me again.