13 September 2008

Addressing the fragmentation of mobile operating systems

Filed under: fragmentation, runtimes — David Wood @ 12:46 am

Back in February this year, Vodafone CEO Arun Sarin called on the mobile industry to reduce the number of operating systems in use. As reported in Computer World at the time, “Vodafone CEO calls for mobile OS consolidation“:

Mobile phone operating systems are key to [user] experience, he said, but there are too many of them; as many as 30 or 40, Sarin estimated.

We have to reduce that number. There’s no way that developers of cool applications can develop for that many operating systems. If we had three, four, five, that would be better,” he said…

One reason for the proliferation of mobile phone operating systems is that, historically, handset suppliers offered their own proprietary code to make the best use of their phones’ limited processing and memory resources. Developing applications for such closed systems is difficult.

Today’s high-end phones, though, have as much computing power as low-end PCs, spawning a new market for operating systems such as Symbian OS or Windows Mobile that run on phones from multiple manufacturers.

That ought to reduce the number of software platforms in the market, but more are still arriving, including several based on the open-source operating system Linux.

This same topic of there being too many mobile operating systems came up this week at the Symbian Foundation roundtable discussion held alongside the CTIA show in San Francisco. During the Q&A, several of the press and analysts in attendance pressed panellists on this point:

As the number of mobile operating systems increase, will fragmentation become a major issue for customers and developers?

As reported by Brad Smith in Wireless Week,

Christy Wyatt, vice president of Software Platforms and Ecosystem for Motorola, said there is always the danger there could be a complete fragmentation of operating systems, especially with the growing varieties of “open source” operating systems. But she said the reality is that companies involved in software services will work to focus on a few operating systems.

Marin Perez in InformationWeek takes up the story:

Additionally, Wyatt said she expects to see a “sharp contraction” in the number of mobile platforms in the future, including one consistent version of mobile Linux.

At the same time, the roundtable panellists suggested that resourceful and determined developers would find ways to ensure that their applications can run on the platforms with significant market share and revenue potential:

Because of the diversity in hardware, there will never be one unified language, said Oren Levine, product market manager in Nokia’s S60 group. But Levine thinks developers will have significant incentive to make their applications work on multiple platforms due to the size of the overall mobile market.

So how much trouble is being posed to developers, in practice, by this diversity of mobile operating systems? Will a multitude of different operating systems continue in existence (contrary to the request of Arun Sarin), each driven and supported by its own factors? Will the mobile ecosystem as a whole find ways to smooth across the differences and fragmentation caused by this variety?

Just one day after the Symbian Foundation roundtable took place, one of the CTIA keynote speakers initially sounded bleak while addressing this same set of questions:

“Developing applications for the fragmented mobile ecosystem is a Herculean effort that often results in developers creating an application that serves the least common denominator of mobile devices with a poor user experience, and an inability to effectively scale,” said Marco Boerries, executive vice president, Yahoo! Connected Life.

But then he unveiled a potential solution:

“Yahoo! Blueprint solves the most challenging problem plaguing today’s mobile landscape — now with one click, you can write once and have mobile services run across a critical mass of devices and operating systems, potentially reaching millions of users. We believe Yahoo! Blueprint is simply the best way to create mobile Internet services — and we expect will change the world of mobile development moving forward.”

Cade Metz of The Register recorded the message from Marco Boerries as follows:

Previously, the Blueprint platform was merely a means of developing widgets that run atop Yahoo! Go, an application suite available for various and sundry mobile phones.

“Blueprint is our way to help us and the rest of the industry to develop mobile services,” Yahoo! mobile president Marco Boerries said this morning, during his keynote at the CTIA Wireless IT and Entertainment trade show in downtown San Francisco.

“You can write a Blueprint service, and with one click, you can generate a standalone application that runs on hundreds of Java devices, on Windows Mobile devices, on Symbian devices and that you can freely distribute. Never has it been so easy to build mobile applications that are this powerful.”

Of course, this sounds very similar to the oft-repeated promise of “Write Once, Run Anywhere (WORA)” (sometimes also expressed as “Write Once, Run Everywhere”). Back in January 1996, Sun’s press release covering Java 1.0 stated,

“Java’s write-once-run-everywhere capability along with its easy accessibility have propelled the software and Internet communities to embrace it as the de facto standard for writing applications for complex networks,” said Alan Baratz, JavaSoft’s newly appointed president. “We’re delighted to invite developers to download Java 1.0 immediately and start building the next killer application.”

Twelve years later, after a history of (at best) mixed success with runtimes that aim to provide WORA, the prospect of a real implementation of this promise remains alluring. And it’s not just Yahoo! that are proclaiming new implementations of this ideal. Another product announcement at CTIA, this time by the Frisco, Texas-based software company Recursion Software, used similar language:

Recursion Unveils Universal Android, .NET CF and Symbian Platform

Voyager pervasive platform achieves “Write Once, Run Everywhere” for application messaging and communications

Recursion Software, Inc., an innovative provider of next-generation platforms and intelligent middleware, today at CTIA Wireless and IT 2008 announced a new release of Voyager, a pervasive software platform that blurs the lines between Android, Symbian, Microsoft .NET and Compact Framework (CF), and JEE.

Voyager turns traditional software development on its head by allowing developers to natively write and maintain one code-set in either Java or .NET and publish the code to the device(s) of their choice regardless of the virtual machine they employ, instantly expanding an application’s device and market reach. Combined with Voyager’s existing support for more than 15 embedded operating systems, this release signifies Voyager’s evolution into a pervasive platform for all “Screens of Life” (TV, PC, phone, car, ultra mobile PC).

There are many other products that sound, on first hearing, to be making claims similar to those advanced on behalf of Yahoo! Blueprint or Recursion Voyager. I’ve not seen any detailed analysis as to the comparative strengths of these different products. (I’ll be grateful to any reader of this blog who provides pointers to such analysis.) However, I have no doubt that these products can significantly reduce the effort that developers need to invest, to create certain kinds of mobile software. At least some of the pain and frustration of multiple mobile operating systems can be reduced.

But whilst these intermediate platforms address issues in the market for add-on applications, they provide less help in the arguably even more important market of “device creation software”. That’s the market of software components that are built into mobile phones:

  • Filling gaps in the operating systems themselves
  • Providing special optimisations for particular hardware or particular networks
  • Enabling low-level differentiation between different devices.

Software at this level tends to need to be written in the native language (usually C or C++) and to use the native APIs of the underlying software platform. Intermediate platforms can’t be used for these purposes, as they don’t provide sufficiently intimate connections to the device firmware.

My own experience – confirmed by many discussions over the years with people deeply involved in specific phone creation projects – is that device creation projects often suffer back-breaking delays because third party software has difficulty in slotting into the overall device software. These delays can occur, even though the third party providers in question seem to have a good track record with previous versions of their software. The problem is that incompatibilities between different versions of the operating system software result in unexpectedly different behaviour from the third party plug-ins. The changes at the platform level invalidate some of the design, the optimisation parameters, or the testing of the plug-ins.

If this kind of delay results from fairly modest incompatibilities between two different versions of the mobile operating system, how much greater is the delay caused by the incompatibilities between two totally different operating systems?

For me, this is the real reason why the mobile industry needs to evolve towards fewer mobile operating system. The companies that keep working with too many different systems are likely to find their development resources stretched too thinly – and (despite possible short-term gains) they’ll lose their effectiveness as a result.

Footnote: The topic of multiple intermediate programming environments will be covered in the keynote panel session “Who will win the runtime race?” on day two (22nd Oct) of the 2008 Symbian Smartphone Show. The panellists will include:

29 June 2008

The five laws of fragmentation

Filed under: fragmentation, leadership, Open Source, Symbian Foundation — David Wood @ 9:42 am

As discussion of the potential for the Symbian Foundation gradually heats up, the topic of potential fragmentation of codelines keeps being raised. To try to advance that discussion, I offer five laws of fragmentation:

1. Fragmentation can have very bad consequences

Fragmentation means there’s more than one active version of a software system, and that add-on or plug-in software which works fine on one of these versions fails to work well on other versions. The bad consequences are the extra delays this causes to development projects.

Symbian saw this with the divergence between our v7.0 and v7.0s releases. (The little ‘s’ was sometimes said to stand for “special”, sometimes for “strategic”, and sometimes for “Series 60”.) UIQ phones at the time were based on our v7.0 release. However, the earliest Series 60 devices (such as the Nokia 7650 “Calypso”) had involved considerable custom modifications to the lower levels of the previous Symbian OS release, v6.1, and these turned out to be incompatible with our v7.0. As a pragmatic measure, v7.0s was created, that had all of the new technology features introduced for v7.0, but which kept application-level compatibility with v6.1.

On the one hand, v7.0s was a stunning success: it powered the Nokia 6600 “Calimero” which was by far the largest selling Symbian OS phone to that time. On the other hand, the incompatibilities between v7.0 and v7.0s caused no end of difficulties to developers of add-on or plug-in software for the phones based on these two versions:

  • The incompatibilities weren’t just at the level of UI – UIQ vs. Series 60
  • There were also incompatibilities at many lower levels of the software plumbing – including substantial differences in implementation of the “TSY” system for telephony plug-ins
  • There were even differences in the development tools that had to be used.

As a result, integration projects for new phones based on each of these releases ran into many delays and difficulties.

Symbian OS v8 was therefore designed as the “unification release”, seeking as much compatibility as possible with both of the previous branches of codeline. It made things considerably better – but some incompatibilities still remained.

As another example, I could write about the distress caused to the Symbian partner ecosystem by the big change in APIs moving from v8 to v9 (changes due mainly to the new PlatSec system for platform security). More than one very senior manager inside our customer companies subsequently urged us in very blunt language, “Don’t f****** break compatibility like that ever again!”

Looking outside the Symbian world, I note the following similar (but more polite) observation in the recent Wall Street Journal article, “Google’s Mobile-Handset Plans Are Slowed“:

Others developers cite hassles of creating programs while Android is still being completed [that is, while it is undergoing change]. One is Louis Gump, vice president of mobile for Weather Channel Interactive, which has built an Android-based mobile weather application. Overall, he says, he has been impressed by the Google software, which has enabled his company to build features such as the ability to look up the weather in a particular neighborhood.

But he says Weather Channel has had to “rewrite a few things” so far, and Google’s most recent revision of Android “is going to require some significant work,” he says.

2. Open Source makes fragmentation easier

If rule 1 was obvious (even though some open source over-enthusiasts seem to be a bit blind to it), rule 2 should be even clearer. Access to the source code for a system (along with the ability to rebuild the system) makes it easier for people to change that software system, in order to smooth their own development purposes. If the platform doesn’t meet a particular requirement of a product that is being built from that platform, hey, you can roll up your sleeves and change the platform. So the trunk platform stays on v2.0 (say) while your branch effectively defines a new version v2.0s (say). That’s one of the beauties of open source. But it can also be the prelude to fragmentation and all the pain which will ensue.

The interesting question about open source is to figure out the circumstances in which fragmentation (also known as “forking”) occurs, and when it doesn’t.

3. Fragmentation can’t be avoided simply by picking the right contract

Various license contracts for open source software specify circumstances in which changes made by users of an open source platform need to be supplied back into the platform. Different contracts specify different conditions, and this can provoke lengthy discussions. However, for the moment, I want to sidestep these discussions and point out that contractual obligations, by themselves, cannot cure all fragmentation tendencies:

  • Even when users of a platform are obligated to return their changes to the platform, and do so, it’s no guarantee that the platform maintainers will adopt these changes
  • The platform maintainers may dislike the changes made by a particular user, and reject them
  • Although a set of changes may make good sense for one set of users, they may involve compromises or optimisations that would be unacceptable to other users of the platform
  • Reasons for divergence might include use of different hardware, running on different networks, the need to support specific add-on software, and so on.

4. The best guarantee against platform fragmentation is powerful platform leadership

Platform fragmentation has some similarities with broader examples of fragmentation. What makes some groups of people pull together for productive collaboration, whereas in other groups, people diverge following their own individual agendas? All societies need both cooperation and competition, but when does the balance tilt too far towards competition?

A portion of the answer is the culture of the society – as reflected in part in its legal framework. But another big portion of the answer is in the quality of the leadership shown in a society. Do people in the group believe that the leaders of the group can be relied on, to keep on “doing the right thing”? Or are the leaders seen as potentially misguided or incompetent?

Turning back to software, users of a platform will be likely to stick with the platform (rather than forking it in any significant way) if they have confidence that the people maintaining the trunk of the platform are:

  1. well-motivated, for the sake of the ecosystem as a whole
  2. competent at quickly and regularly making valuable new high quality releases that (again) meet the needs of the ecosystem as a whole.

Both the “character” (point 1) and the “competence” (point 2) are important here. As Stephen Covey (both father and son) have repeatedly emphasised, you can’t get good trust without having both good character and good competence.

5. The less mature the platform the more likely it will be to fragment, especially if there’s a diverse customer base

If a platform is undergoing significant change, users can reason that it’s unlikely to coalese any time soon into a viable new release, and they’ll be more inclined to carry on working with their own side version of the platform, rather than waiting for what could be a long time for the evolving trunk of the platform to meet their own particular needs.

This tendency is increased if there are diverse customers, who each have their own differing expectations and demands for the still-immature software platform.

In contrast, if the core of the platform is rock-solid, and changes are being carefully controlled to well-defined areas within the platform, customers will be more likely to want to align their changes with the platform, rather than working independently. Customers will reason that:

  • The platform is likely to issue a series of valuable updates, over the months and years ahead
  • If I diverge from the platform, it will probably be hard, later on, to merge the new platform release material into my own fork
  • That is, if I diverge from the platform, I may gain short-term benefit, but then I’ll likely miss out on all the good innovation that subsequent platform releases will contain
  • So I’d better work closely with the developers of the trunk of the platform, rather than allowing my team to diverge from it.

Footnote: Personally I see the Symbian Foundation codeline to be considerably more mature (tried and tested in numerous successful smartphones) than the codeline in any roughly similar mobile phone oriented Linux-based foundation. That’s why I expect that the Symbian Foundation codeline will fall under less fragmentation pressure. I also believe that Symbian’s well-established software development processes (such as careful roadmap management, compatibility management, system architecture review, modular design, overnight builds, peer reviews, and systematic and extensive regression testing) are set to transfer smoothly into this new and exciting world, maintaining our track record of predictable high-quality releases – further lessening the risks of fragmentation.

23 June 2008

Fragmentation is easy, integration is hard

Filed under: Android, fragmentation, integration — David Wood @ 2:13 pm

The Wall Street Journal reports today that “Google’s Mobile-Handset Plans Are Slowed“. The Inquirer picks up the story and adds a few choice morsels of its own: “Depressing news as Google’s Android delayed“:

However, life’s little crises just kept getting the Android down and now apparently some mobile network operators like Sprint Nextel, have abandoned any attempt to get an Android on the market until 2009. This is purportedly because the majority of Google’s attention and resources have been going to Sprint’s competitor T-Mobile USA, who still hope to have an Android mobile out by the end of Q4. We have it on good authority (from un-named sources of course) that Sprint actually asked Google “Do you want me to sit in the corner and rust, or just fall apart where I’m standing?”…

Director of mobile platforms at Google, Andy Rubin, gloomily noted that trying to develop software while the company’s irritating partners kept pushing for new features, was a time-consuming task. “This is where the pain happens”, he sighed.

I recognise this pain. It’s something that has occurred many times during Symbian’s history. That’s why I’ve emphasised a dilemma facing Android: Fragmentation is easy, but integration is hard. Coping with multiple forceful customers at the same time, while your codebase is still immature, is a really tough challenge. Glitzy demos of v2 features don’t help matters: they drive up interest that needs to be deflated, as you have to explain to customers that, no, these features aren’t ready to include in the software package for their next phones, despite looking brilliant on YouTube. Instead, the focus needs to go on the hard, hard task of integration.

18 June 2008

The dangers of fragmentation

Filed under: fragmentation, Linux, Olswang, Open Source, Symbian — David Wood @ 8:46 am

My comments on mobile Linux fragmentation at the Handsets World event in Berlin were picked up by David Meyer (“Doubts raised over Android fragmentation“) and prompted a response by Andy Rubin, co-founder of Google’s Android team. According to the reports,

On a recent comment by Symbian’s research chief, David Wood, that Android would eventually end up fragmented, Rubin said it’s all part of the open source game.

Raising the example of a carrier traditionally having to wait for a closed platform developer to release the next version of its software to “enable” the carrier to offer new services, Rubin said carriers could just hire a developer internally to speed up that process without waiting any longer.

“If that fragmentation is what [Wood] is talking about, that’s great–let’s do it,” said Rubin.

Assuming these reports are accurate, they fall into the pattern of emphasising the short-term benefits of fragmentation, but de-emphasising the growing downstream compatibility problems of a platform being split into different variants. They make fragmentation sound like fun. But believe me, it’s not!

I noticed the same pattern while watching a panel on Open Source in Mobile at one of the Smartphone Summit events that take place the day before CTIA. The panel moderator posed the question, “Is fragmentation a good or bad thing?” The first few panellists were from consultancies and service providers. “Yes”, they said, smiling, “Fragmentation gives more opportunity for doing things differently – and gives us more work to do.” (I paraphrase, but only slightly.) Then came the turn of a VP from one of the phone manufacturers who have struggled perhaps more than most with the variations and incompatibilities between different mobile Linux software stacks. “Let’s be clear”, came the steely response, “fragmentation is a BAD thing, and we have to solve that problem”.

Luigi Licciardi, EVP of Telecom Italia, made similar remarks at the ‘Open source in Mobile’ conference in Madrid in September 2007. He said that one thing his network operator needs in terms of software platforms which they would consider using in the mobile phones, is ‘backwards compatibility’ – in other words, a certain level of stability. (This sounds simple, but I know from my own experience that backwards compatibility requires deep skill and expertise in the midst of a rapidly changing marketplace.) Moreover, the software platform has to be responsive to the needs of the individual operators: the operator needs to be able to go and talk to a company and say “give us these changes and modifications”. He also said that the platform needs to be open to applications for network connections and end users, but has to be closed to malware. In other words it has got to have a very good security story. (Incidentally, I believe Symbian has uniquely got a very strong security story, with platform security built deep into the operating system.) Finally, he emphasised that “a fragmented Linux is of no interest to operators”.

This topic deserves more attention. Let me share some analysis from a transcript of a talk I gave at the Olswang “Open Source Summit” in London last November:

The point is that there is a great tendency in the mobile phone space for mobile Linux variants to fragment and split. This was first drawn to my attention more than two years ago by Avi Greengart who is a US-based analyst. He said that mobile Linux is the new Unix, meaning that despite the best intentions of all involved, it keeps on going its own separate ways.

So why is that happening? It is happening first of all because fragmentation is easy. This means that you can take the code and do whatever you like with it. But will these changes be brought back inside the main platform? Well I claim that, especially in a fast moving market such as smartphones, integration is hard. The changes tend to be incompatible with each other. Therefore it is my prediction that, on average, mobile Linux will fragment faster than it unifies.

It is true that there are many people who say it is very bad that there are all these different mobile Linux implementations. It is very bad because it has caused great problems for developers: they have to test against so many stacks. These people ask themselves, “Can’t we unify things?” And every few months there is a new group that is formed and says, in effect, “Right, we are going to make a better job of unifying mobile Linux than the last lot, they weren’t doing it fast enough, they weren’t doing it seriously enough, so we are going to change that.” But I see the contrary, that there is a greater tendency to fragment in this space than to unify, and here’s why.

It is always easier and quicker to release a device-specific solution than to invest the extra effort to put that functionality into a reusable platform, and then on into a device. In other words, when you are racing to market, when the market leaders are rushing away from you and leaving more and more clear blue water between you and them, it is much more tempting to say, “well I know we are supposed to be thinking in terms of platform, but just for now I am going to serve the needs of my individual product.”

Interestingly we had the same problem in the early days of Symbian. One of the co-founders of Symbian, Bill Batchelor, coined the phrase “the Symbian Paradox”, which is that we found it hard to put functionality into the platform, rather than just serve it out to eager individual customers via consultancy projects. But we gradually learned how to do that, and we gradually put more and more functionality into the platform, suitable for all customers, and therefore more and more customer projects benefited more widely.

So why is mobile Linux fragmenting in a way that perhaps open source in other areas isn’t fragmenting? First, it is an intense, fast moving industry. Symbian as the market leader, together with our customers, is bringing out significant new functionality every three or four months. So there is no time for other people to take things easy and properly invest in their platforms. They are tempted to cut corners – to the detriment of the platform.

Second, if you look at how some of these consortia are meant to work, they are meant to involve various people contributing source code. If you look at some of their architecture diagrams, you might get one company in say Asia, which is contributing one chunk of software, which is meant to be used by other companies the world over. Well guess what happens in a real life project? Another company, let’s say a company trying to ship a Linux based phone in America, and surprise, surprise, the software doesn’t work, it fails to get FCC approval, it doesn’t meet the network operators’ needs or there are bugs that only show up on the network operators in America. So what do they say? They say to the first group (the people out in Asia) “would you mind stopping what you are doing and come and fix this, we are desperate for this fix for our software”. The group in Asia say, “well we are very sorry, we are struggling hard, and we are behind as well, we would rather prioritise our own projects, if you don’t mind, shipping our own software, debugging it on different networks”.

At this point you may raise the question: isn’t open source meant to be somewhat magical in that you can all just look at it and fix it anyway; you don’t need to get the original person to come and fix it? But here we reach the crux of the matter. The problem is there is just too much code, there are such vast systems, it is not just a few hundred lines, or even a few thousand lines of code, there are hundreds of thousands or even millions of lines of code in these components and they are all interfacing together. So somebody looks at the code and they think, “Oh gosh, it is very complicated”, and they look and they look and they look and eventually they think, “Well if I change this line of code, it will probably work”, but then without realising it they have broken something else. And the project takes ages and ages to progress..

Compare this to the following scenario: some swimmers are so good they can actually swim across the English Channel, they go all the way from England to France. Suppose they now say, “Yes I have cracked swimming, what will I do next? Oh I will swim all the way across the Atlantic, after all, it is just an ocean and I have already swum one ocean, so what is different about another ocean?” Well it is the kind of difference between the places where open source has been doing well already and the broader oceans with all the complications of full telephony in smartphones.

So what happens next in this situation? Eventually one company or another may come up with a fix to the defects they faced. But then they try and put it back in the platform, and the first company very typically disagrees, saying “I don’t like what you have done with our code, you have changed it in a very funny way, it isn’t the way we would have done it”. And so the code fragments – one version with the original company, and the other in the new company. That is how it ends up that mobile Linux fragments more than it unifies.

I say this firstly, because I have contacts in the industry who lead me to believe that this is what happens. Secondly, we have the same pressures inside Symbian, but we have learned how to cope with it. We often get enhancements coming back from one customer engagement project and at first it doesn’t quite fit into the main OS platform, but we have built up the highly specialist skills how to do this integration.

As I mentioned, integration is hard. You need a company that is clearly responsible and capable for that. This company needs to be independent and trustworthy, being motivated – not by any kind of ulterior motive, but by having only one place in the value chain, doing one job only, which is that it is creating large scale customer satisfaction by volume sales of the platform.

« Newer Posts

Blog at WordPress.com.