Latest Android Installed Base

opensignalOpenSignal have a new version of their study into Android Fragmentation. It’s the multi-coloured graphic, that I won’t be reproducing here, you often see when some blogger starts talking about fragmentation. Instead, I find the variation of OS version market share over time to be more interesting as it helps define what minimum version to target for new projects…

androidfragmentation

I believe this data to be more useful than declared device shipments as it a) represents the installed base and b) tells us the Android version. It’s also useful to compare with Google’s own dashboard.

It has been the case for a while now that there’s no point supporting pre Android 4.0.3 (Ice Cream Sandwich). Having worked on many past projects that had to go down to 2.3, this is great relief as it simplifies development and testing. Despite the hype over fragmentation, carefully selecting test devices that cover all the Android versions (v4.03 and later), top brands, dpi bands and main screen sizes is usually sufficient on most projects.

There are still a few outlying ‘strange’ current devices to consider. For example, there’s the Samsung Galaxy Pocket 2 that runs Android 4.4.2 yet only has a ldpi 240×320 screen. I find it’s a useful device to test apps at the lowest end of the device spectrum.

Mobile, Agile, Lean and Startups

jeffpattonFor the last few years I have mainly worked with startups. Of the many companies I have worked with, very few, only about 10%, have used Agile. Does it matter?

agileAgile

On the one hand, we have a few Agile tech speakers and consultants who will tell you if you don’t do it their way you’re stupid or incompetent. On the other we have people such as Jeff Patton, who previously won Agile Alliance’s Gordon Pask Award for contributions to Agile Development, saying common Agile practice isn’t for startups. Jeff’s article is thought-provoking because he says, for many startups, learning velocity is more important than delivery velocity.

leanstartupLean startup

Martin Fowler has an related article where he says “You don’t do agile or lean you do agile and lean”. I am not sure this is always true as, in practice, Agile invariably results in a significant upfront investment in additional unit testing – but I suppose it strictly need not when mixing Agile and lean.

You really need to consider your project. Is learning velocity important? Agile and better testing came out of a time when there were huge projects that were failing. Is your mobile project that huge/complex (it can be) or are you over-engineering? What’s your initial budget? What’s the anticipated lifespan of your software? Will the code ever have to be handed over to others to continue to develop? Is the project mission critical? How many people will be developing the project and how experienced are they? How much do you trust your current development process if you have one? How quickly do you expect things to change and by how much? Answering some of these questions might help you decide how much Agile matters for your project.

Android Fragmentation

android.gifThe Register reports from Google I/O that Android will offer an i-Phone-like App Store.

The more interesting part of the report is at the end – after all, how will such an app store determine what applications are compatible with what phones?…

"They can add to it. They can remove from it. They make it their own," Rubin said. "They can rip out all the Google stuff and put in all Yahoo! stuff."

This goes for entire APIs (application programming interfaces) as well – an obvious a point of concern for some developers mulling apps for the platform. "I see Android and I see all its APIs. What’s to stop someone from turning off all those APIs?" one developer asked Rubin.

There’s nothing to stop them, Rubin said. But he indicated that Google will provide tools that enable developers to easily verify the make-up of each Android handset. "We’re providing a piece of technology – I can’t go into a great amount of detail – that tests the APIs," he explained. "This will be a script that you’ll be able to run…and determine whether all the APIs are there."

As a developer, I don’t see how this will work. First of all, developers need APIs to determine individual capabilities, not a script that does this all at once. I need to do this at runtime because capabilities may change over time as the phone software is upgraded and updated.

Even if I know a capability doesn’t exist, what do I do? I can’t code around potentially thousands of missing capabilities.

Finally, even when capability tests exist and they indicate a capability is present, this won’t guarantee compatibility. There’s already a similar situation in the Java ME world where Sun Technology Compatibility Kits (TCKs) are used to test JSRs. There are thousands of very detailed tests and despite the fact that Java runtimes (Java Virtual Machine,JVM) on phones are run through them, it’s not possible to guarantee that a Java ME application that runs on one JVM will also run on another JVM.

Why is this? Despite the large number of tests, they cannot test everything. It’s not just the presence of an API that’s important, it’s also the behaviour. An API might pass a test but have an obscure bug that only becomes evident under particular circumstances. An API might also only implement a sub-set of a capability that’s either enough to pass the test or due to lack of some phone hardware capability. Also, no matter how hard humans try, with thousands of APIs, the expected behaviour of some of them will be to some extent ambiguous and open to interpretation. There are also types of test, particularly UI related, that are very difficult if not impossible to automate (The JAVA TCK’s rely on manual tests for these).

Hence, I currently believe there will always be Android fragmentation – and certainly more so than with more controlled platforms such as Symbian, Windows Mobile and the iPhone.

Mob4Hire

mob4hire.gifI came across Mob4Hire this week. A new site that provides "Crowd Sources Mobile Application Testing". The developer submits projects involving specific handsets and network operators. End users bid on projects and get paid via Paypal.

It’s often the case that an application is tested but fails in real-life use due to limitation of some random network operator. Mob4Hire can help reduce this problem by allowing developers to reach a greater breadth of network operators and phones. I also see it as a much more targeted solution than beta testing where typically only fraction of percent of users submit useful observations.

On the other hand, I can see some companies having problems with confidentiality and the inevitable variable quality of testers. Also, this site doesn’t solve the problem of ongoing support and testing – where the majority of problems surface. If you/your company didn’t test the particular variant then it may be difficult to fix.

The Mob4Hire site/process is still beta and a bit clunky and ambiguous in places. However, I think Mob4Hire may have something useful here if they evolve the service based on developer and tester feedback.

Perpetual Beta

On the Nokia Beta Labs blog there’s an interesting post and discussion on the meaning of ‘beta’. Many people accept that in the web world ‘beta’ now means the product or service will work as expected and continually evolve. However, it’s my experience that many ‘mass market’ users will actively avoid trying an installable mobile application if it is beta. The corollary of this that if you wish a large number of people to try your working downloadable application, then don’t mark it as ‘beta’.

CppUTest Unit Test Framework

symbian.gifArtem at Agile Software Development has extended the CppUTest unit test framework to support Symbian descriptors and leaves. There’s also a heavily commented example how to use the framework.

If you are looking into unit test tools then you might also like to consider Penrillian’s Symbian OS Unit and Symbian Test Unit.

paca Mobile Center

paca.gifAt one of the London Mobile Mondays I came across the paca mobile center. It’s a facility in Marseille and Sophia Antipolis, France that fills the gap between buying all test devices yourself or outsourcing to a 3rd party test house.

For 300 Euro per day or 6000 Euro/year (both +VAT) startups can gain access to between 400 and 500 devices. There’s also a facility in Glasgow, UK that also provides similar facilities free of charge to paca members.

Facilities such as this and 3rd party test houses are great for low budget startups. However, they don’t help when it comes to resolving support issues quickly when an application is in the field… as you might not have the particular phone to reproduce the problem. Also, while the application might have worked on the particular phone during testing, issues due to different firmware or (more usually) due to different network operators or (even more usually) user error can often stop your application from working. Hence, if you can afford it, then I believe it’s best to keep your testing in-house so that you can reproduce any problems.

Update 21 Sept: I received an email from paca Mobile Center (PMC) with the following clarifications…

  • PMC is a not-for-profit organisation and the fees just cover handset purchasing.
  • Wireless innovation in Glasgow is an incubator. Their service is restricted to their companies and they have 50 handsets.
  • Handsets aren’t donated by manufacturers or operators. Instead 30 handsets are purchased each month.
  • There are also services to allow PMC to carry out tests so that you don’t have to actually travel there.
  • PMC founders are mobile application specialists and its possible to link companies to solve some issues.
  • Customers tend to buy the top 20 handsets but haven’t time to source the whole market.
  • PMC spends a lot of time identifying and sourcing new handsets – much more time than you would purchasing a phone already in the market.

Motorola Z8 Testing

uiqtesting.gifOne of the problems of application development is that it’s often not possible to test on phones that are about to be released. This is especially the case when you are in a country where phones arrive late to the market.

Sometimes it’s possible to ask phone OEM’s for pre-production phones but often there aren’t enough units to go round. Unless you have an application that is compelling for the OEM then you won’t get a device.

UIQ have solved this problem for the Motorola Z8 by offering to test applications. What’s more it’s free of charge. This offer is open until 30 September 2007.

This has made me wonder why phone OEMs don’t offer this informal service as a matter of course. OK, it would cost the OEM something – but a few people informally testing applications would be negligible in the overall scope of things. It would help improve the quality of released applications (see Ewan’s recent comments for some surprising observations), encourage developers and also allow the OEM to identify potential partners. Microsoft do a similar thing via ‘Evangelists’ who seek out and help developers in this and many other interesting ways. Note however, the kind of thing I am talking about shouldn’t be confused with Symbian Signed testing – it’s more about informal collaboration.