…what’s needed is a testing system that combines a pre-configured set of tests with the involvement of a large set of representative “real users” (often called “friendly users”) who try out whatever comes to mind.
Friendly User Testing
Given the notion that smartphone projects face untold combinations and a limitless number of applications, operating across a variety of networks, it’s impractical to expect manufacturers and operators to undertake a complete and exhaustive series of tests. Firstly, there isn’t the time; secondly, it’s too costly, and thirdly, more testing doesn’t always deliver ‘better quality’.
Commentators suggest different practical approaches to this burgeoning problem but according to experienced smartphone software authority, David Wood…
Defining FUT
the wisdom of crowds
Friendly User Test (FUT) programs employ a large group (100+) of real users inside the product development company to embrace pre-launch devices and feed back errors, issues and insights to the product development team in a structured manner, under real world conditions.
Figure 1: The increasing cost of quality
Quality improvement
validation, not just verification
As consumers become increasingly demanding and experienced, their user experience and their reaction to general performance characteristics are the most important elements of a smartphone’s perceived quality. Friendly users are uniquely placed to provide that validation feedback to the product team (are we building the right thing?) because friendly users are our closest approximation to experienced consumers.
And, whilst in the past verification tests were primarily performed in the lab, as test cases become unlimited, FUTs are seen as a practical solution to bounding an unlimited problem. FUTs – given a statistically significant sample size – force the engineering team to focus on what’s important – their customers.
So where there has always been a strong argument to perform a balanced set of tests – verification and validation, lab and field – smartphone developments increasingly shift test strategy out of the lab and into the field, where FUTs are seen as the most practical solution.
Cost reduction
find, fix & prevent
The smartphone is the formula one car of the mobile industry. It incorporates all the new sensors, radio hardware and software applications. As well as all the new designs and processes that help the engineering and commercialisation teams blend these new technologies into something that’s usable and appealing to consumers.
Testing these technologies, designs and processes means that the product development teams will inevitably discover errors. And traditionally, (see Figure 1), errors discovered at the later stages of the process are significantly more expensive to fix than errors found at earlier stages. Testing is expensive, but it costs far more to fix issues once the product is launched (see ‘Antennagate’ story below).
A well-managed FUT program promises to reduce these expenses:
Earlier discovery
It allows friendly users to participate in the earlier (cheaper) stages of the development process and unearth important defects that may not be found by formal verification tests.
Lower TTF
It can lower the error fix times (and associated costs) at the latter stages of the development process.
Insurance policy
It prevents user experience problems reaching the market, insuring the manufacturer, operator and retail channels against expensive recalls, customer support calls and customer returns.
The cost of Antennagate
‘Antennagate’ – the public discovery and dissection across social media of the iPhone 4’s signal performance problems – could have forced Apple to recall their smartphone within a month of its launch. And if recalled, according to analysts, it would have cost Apple stakeholders an estimated 1.5 billion dollars. In time, Apple opted for an orchestrated PR campaign and a free bumper surround, to help mitigate the inconvenience of a poorly performing field unit. The good news? That remedy only cost $175 million; a cheap option when compared to the product recall, although that’s still six times greater than the estimated total engineering commitment to develop a smartphone platform!