20 December 2009

Integration is hard

Filed under: integration, Linux — David Wood @ 5:29 pm

One phrase which I’ve uttered on numerous occasions in the last two years is,

Fragmentation is easy, integration is hard

Recently, I’ve taken to putting more emphasis on the “integration is hard” part of this phrase.  I keep seeing evidence, everywhere I look, of the importance of skills and systems that allow different variations and innovations to be easily incorporated back into wider systems.  A new technology, by itself, is generally insufficient to make an impact in society.  Lots of time-consuming system changes need to happen as well, before that technology can deliver “end-to-end” results.

In a small way, I’ve been seeing more evidence of this same point in the last few days, as I’ve been struggling to assemble a system of software to support a new application.

The application I’d like to support is the “IdeaTorrent” that powers, among other things, the Ubuntu Brainstorm site.

This software is an open source (free of charge) implementation of a voting and review system for ideas.  I’d like to provide a version for my own purposes.  The Ubuntu site helpfully has a link to the providers of the software, IdeaTorrent.

In principle, the installation instructions look simple enough:

  1. Use a webserver with PHP installed
  2. Install the database software PostgreSQL (v8.3)
  3. Install Drupal (content management software)
  4. Install IdeaTorrent
  5. Do some configuration.

In practice, things are proving tougher.  The word “however” appears quite a few times in what follows…

The first webserver I obtained had a different database installed, MySQL.  Although I heard from many people that, by various criteria, MySQL is preferable to PostgreSQL, the IdeaTorrent installation notes were clear that the system would work only if PostgreSQL were present.

However, with this webserver, I failed at the first hurdle.  I found I couldn’t install any new software, since it was a “shared” server that other people were using too.  That environment (not unreasonably) prohibited any user from gaining “root access” security permissions.  If I wanted to change the components, I would need a different working environment.

I therefore purchased a more expensive hosting system.  (I am hopeful of a refund for the first one – from the same supplier, GoDaddy).  This system does allow installation of new software.  And, to make things easier, it comes with PostgreSQL already installed.  That took me to the task of installing Drupal.  That task ended up taking me more than 24 hours elapsed time, as I went along a crash learning course about Linux file permissions, the BASH command shell, PHP, and Drupal itself.  So far, so good.

Finally I installed IdeaTorrent, and reconfigured Drupal so that it recognised the IdeaTorrent modules.

However, lengthy error messages started getting displayed.  Various functions were missing, so the system would not work.  The functions in question were meant to be supplied by PostgreSQL.  At this stage, I paid more attention to the part in the IdeaTorrent installation instructions that said “PostgreSQL 8.3 (Important! Won’t work below 8.3)”.  On checking, I found out that the system provided by GoDaddy has PostgreSQL 8.2.9.

8.2.9 looks numerically close to 8.3, but clearly it’s not close enough.

Further investigation showed me that Linux has an impressive looking system for assisting with upgrades of components.  I followed instructions that I found spread out over the Internet – including the quaintly titled article “An Almost Idiot’s Guide to PostgreSQL YUM“.

These instructions let me to downloading a small “.RPM” file onto the server, and then invoking a command called “YUM“.  (Apparently YUM stands for “Yellowdog Updater, Modified”!)  It looked like it was working.

However, when I started up the system again, I saw that the version of PostgreSQL that was running was still 8.2.9.

Another 24 hours elapsed – with lots of red herrings along the way.  I eventually realised that some of the online links had misled me to pick the wrong .RPM file.  (The one I was trying to install was actually, despite appearances, an even older version.)  Finally I found the right file.  This time, when I ran the YUM magic, lots more things happened.

However, these changes broke the admin “control panel” interface when GoDaddy provided, in order to administer databases.  Without this interface, the remainder of the installation (as required to set up Drupal) would probably be impossible.

Not for the first time, I went back to square one (GoDaddy have a handy service whereby they will return the whole system to its inital “provisioned” state).  This time, I was more careful in how I did the upgrades.  This time, the control panel interface kept working fine.

However, this time, although the functions were shown whereby I could administer the databases, the functions all told me:

Admin password is out of sync. Database management will be unavailable until the admin password is changed.

However, everything I try to do to enter (or redefine) the “admin password” fails.  My current guess is that the handling of admin passwords somehow changed between PostgreSQL v8.2 and PostgreSQL v8.3.

At the moment, I’m stuck again.  Perhaps I’ll figure out what password I ought to be typing in.  Or perhaps I’ll have the whole system reprovisioned yet again (for about the fifth time) and do the upgrades even more carefully this time.  (Any advice from readers of this blog will be gratefully received!)

I see my trials and tribulations of the last few days as an example of how it’s hard to alter individual components of a complex software system.  These components can have all kinds of dependencies on each other – explicit dependencies, implicit dependencies, data format dependencies, obscure dependencies, bug dependencies, and so on.

My takeaway isn’t to assign any blame to any particular system.  My takeaway is that:

  • Anyone planning to alter components in a complex software system should set aside plenty time to debug integration issues that arise along the way;
  • Anyone embarking on such an endeavour should be ready to call in expert advice from people who have significant prior experience in that same task.

Indeed, integration is hard.  However, that’s a reason for all of us to design systems that are “integration friendly”.

Added later

Since writing this blogpost, I’ve now understood why the “control panel” phpPgAdmin application was failing to accept passwords, and the problem is solved.

It turns out that a clean installation of PostgreSQL specifies (via the file pg_hba.conf) that user access is checked via an “ident” (identification) system that does not use passwords.  General advice is that this should be changed to use passwords instead.  Changing the word “ident” to “md5” in the above-mentioned configuration file does the trick – and phpPgAdmin is now working for me again.

18 June 2008

The dangers of fragmentation

Filed under: fragmentation, Linux, Olswang, Open Source, Symbian — David Wood @ 8:46 am

My comments on mobile Linux fragmentation at the Handsets World event in Berlin were picked up by David Meyer (“Doubts raised over Android fragmentation“) and prompted a response by Andy Rubin, co-founder of Google’s Android team. According to the reports,

On a recent comment by Symbian’s research chief, David Wood, that Android would eventually end up fragmented, Rubin said it’s all part of the open source game.

Raising the example of a carrier traditionally having to wait for a closed platform developer to release the next version of its software to “enable” the carrier to offer new services, Rubin said carriers could just hire a developer internally to speed up that process without waiting any longer.

“If that fragmentation is what [Wood] is talking about, that’s great–let’s do it,” said Rubin.

Assuming these reports are accurate, they fall into the pattern of emphasising the short-term benefits of fragmentation, but de-emphasising the growing downstream compatibility problems of a platform being split into different variants. They make fragmentation sound like fun. But believe me, it’s not!

I noticed the same pattern while watching a panel on Open Source in Mobile at one of the Smartphone Summit events that take place the day before CTIA. The panel moderator posed the question, “Is fragmentation a good or bad thing?” The first few panellists were from consultancies and service providers. “Yes”, they said, smiling, “Fragmentation gives more opportunity for doing things differently – and gives us more work to do.” (I paraphrase, but only slightly.) Then came the turn of a VP from one of the phone manufacturers who have struggled perhaps more than most with the variations and incompatibilities between different mobile Linux software stacks. “Let’s be clear”, came the steely response, “fragmentation is a BAD thing, and we have to solve that problem”.

Luigi Licciardi, EVP of Telecom Italia, made similar remarks at the ‘Open source in Mobile’ conference in Madrid in September 2007. He said that one thing his network operator needs in terms of software platforms which they would consider using in the mobile phones, is ‘backwards compatibility’ – in other words, a certain level of stability. (This sounds simple, but I know from my own experience that backwards compatibility requires deep skill and expertise in the midst of a rapidly changing marketplace.) Moreover, the software platform has to be responsive to the needs of the individual operators: the operator needs to be able to go and talk to a company and say “give us these changes and modifications”. He also said that the platform needs to be open to applications for network connections and end users, but has to be closed to malware. In other words it has got to have a very good security story. (Incidentally, I believe Symbian has uniquely got a very strong security story, with platform security built deep into the operating system.) Finally, he emphasised that “a fragmented Linux is of no interest to operators”.

This topic deserves more attention. Let me share some analysis from a transcript of a talk I gave at the Olswang “Open Source Summit” in London last November:

The point is that there is a great tendency in the mobile phone space for mobile Linux variants to fragment and split. This was first drawn to my attention more than two years ago by Avi Greengart who is a US-based analyst. He said that mobile Linux is the new Unix, meaning that despite the best intentions of all involved, it keeps on going its own separate ways.

So why is that happening? It is happening first of all because fragmentation is easy. This means that you can take the code and do whatever you like with it. But will these changes be brought back inside the main platform? Well I claim that, especially in a fast moving market such as smartphones, integration is hard. The changes tend to be incompatible with each other. Therefore it is my prediction that, on average, mobile Linux will fragment faster than it unifies.

It is true that there are many people who say it is very bad that there are all these different mobile Linux implementations. It is very bad because it has caused great problems for developers: they have to test against so many stacks. These people ask themselves, “Can’t we unify things?” And every few months there is a new group that is formed and says, in effect, “Right, we are going to make a better job of unifying mobile Linux than the last lot, they weren’t doing it fast enough, they weren’t doing it seriously enough, so we are going to change that.” But I see the contrary, that there is a greater tendency to fragment in this space than to unify, and here’s why.

It is always easier and quicker to release a device-specific solution than to invest the extra effort to put that functionality into a reusable platform, and then on into a device. In other words, when you are racing to market, when the market leaders are rushing away from you and leaving more and more clear blue water between you and them, it is much more tempting to say, “well I know we are supposed to be thinking in terms of platform, but just for now I am going to serve the needs of my individual product.”

Interestingly we had the same problem in the early days of Symbian. One of the co-founders of Symbian, Bill Batchelor, coined the phrase “the Symbian Paradox”, which is that we found it hard to put functionality into the platform, rather than just serve it out to eager individual customers via consultancy projects. But we gradually learned how to do that, and we gradually put more and more functionality into the platform, suitable for all customers, and therefore more and more customer projects benefited more widely.

So why is mobile Linux fragmenting in a way that perhaps open source in other areas isn’t fragmenting? First, it is an intense, fast moving industry. Symbian as the market leader, together with our customers, is bringing out significant new functionality every three or four months. So there is no time for other people to take things easy and properly invest in their platforms. They are tempted to cut corners – to the detriment of the platform.

Second, if you look at how some of these consortia are meant to work, they are meant to involve various people contributing source code. If you look at some of their architecture diagrams, you might get one company in say Asia, which is contributing one chunk of software, which is meant to be used by other companies the world over. Well guess what happens in a real life project? Another company, let’s say a company trying to ship a Linux based phone in America, and surprise, surprise, the software doesn’t work, it fails to get FCC approval, it doesn’t meet the network operators’ needs or there are bugs that only show up on the network operators in America. So what do they say? They say to the first group (the people out in Asia) “would you mind stopping what you are doing and come and fix this, we are desperate for this fix for our software”. The group in Asia say, “well we are very sorry, we are struggling hard, and we are behind as well, we would rather prioritise our own projects, if you don’t mind, shipping our own software, debugging it on different networks”.

At this point you may raise the question: isn’t open source meant to be somewhat magical in that you can all just look at it and fix it anyway; you don’t need to get the original person to come and fix it? But here we reach the crux of the matter. The problem is there is just too much code, there are such vast systems, it is not just a few hundred lines, or even a few thousand lines of code, there are hundreds of thousands or even millions of lines of code in these components and they are all interfacing together. So somebody looks at the code and they think, “Oh gosh, it is very complicated”, and they look and they look and they look and eventually they think, “Well if I change this line of code, it will probably work”, but then without realising it they have broken something else. And the project takes ages and ages to progress..

Compare this to the following scenario: some swimmers are so good they can actually swim across the English Channel, they go all the way from England to France. Suppose they now say, “Yes I have cracked swimming, what will I do next? Oh I will swim all the way across the Atlantic, after all, it is just an ocean and I have already swum one ocean, so what is different about another ocean?” Well it is the kind of difference between the places where open source has been doing well already and the broader oceans with all the complications of full telephony in smartphones.

So what happens next in this situation? Eventually one company or another may come up with a fix to the defects they faced. But then they try and put it back in the platform, and the first company very typically disagrees, saying “I don’t like what you have done with our code, you have changed it in a very funny way, it isn’t the way we would have done it”. And so the code fragments – one version with the original company, and the other in the new company. That is how it ends up that mobile Linux fragments more than it unifies.

I say this firstly, because I have contacts in the industry who lead me to believe that this is what happens. Secondly, we have the same pressures inside Symbian, but we have learned how to cope with it. We often get enhancements coming back from one customer engagement project and at first it doesn’t quite fit into the main OS platform, but we have built up the highly specialist skills how to do this integration.

As I mentioned, integration is hard. You need a company that is clearly responsible and capable for that. This company needs to be independent and trustworthy, being motivated – not by any kind of ulterior motive, but by having only one place in the value chain, doing one job only, which is that it is creating large scale customer satisfaction by volume sales of the platform.

Blog at WordPress.com.