Performance Testing Mobile Applications – Conclusion
In conclusion, I wanted to answer a few questions and comments that I have got back on the series, and summarize.
When at junior school, I was asked what I wanted to be one day. I responded a scientist and inventor. The response I got stuck in my mind as it seemed totally at odds with how I perceived this career path. While I thought each day would bring new things, the “Scientific Method” is around understanding the impact of dependant and independent variables. Running the exact same test over and over again, with the small change in a single variable to understand the effects that variable has.
User Experience = Enablers + API + Network + Client.
Mobile Testing both overlaps and diverges from traditional performance testing. From the Enablers or API to the Networks nearby cell tower it is much the same, yet differs in maturity (standards and skill levels). In contrast, the last km from the cell tower to the device and the device itself are radically different. The constant changes in type and number of devices and network locations in this portion, make it impossible to test every combination and eliminate all variables. In fact the number of variables in mobile applications is so extensive, economically it is just not viable to try test every variable. On ABC wireless network, XYZ device, on release 123, in downtown corner of MNO at 12 noon, in this date, may offer a response rate of for your example user Anne of 1 second. But Anne takes 5 steps left, or tries again in 10 minutes, or upgrades her device etc and the response rate could dramatically change.
The post on Tuning in for excellence, was around the need for QA to look at the role differently, dialing-in People, Process and Tools for greater excellence. In the past, QA searched for defects. Developing and running a set of automated test cases that covered as large a percentage of possibilities, speaking in terms of 20/50/80% or more test coverage. Then handing it to performance team to develop and run their test cases, who were comfortable saying 80% of transactions will be in under xxx milliseconds. These tests were viewed in terms of success or fail, with narrow ranges of variance. If a test failed, it was handed to management, architects, development, operations etc, to try pinpoint why a test case failed or how to resolve it.
But with Mobile applications, we start with the sure knowledge that our test case coverage will only be some fraction of a percentage (less than 1%) due to the number of variables. Instead of testing for pass or fail, a strategy is needed to test to understand expected behaviour and identify any unexpected behaviour. By running the same tests to understand the impact a group of variables have, we can determine the expected impact of just that variable. Testing for Baseline Understanding, not pass or fail. Using this expected behaviour to quickly identify and isolate unexpected behaviour.
- Yes, we still don’t know the exact user experience for every device, device configuration, location and network etc. But we know what the expected experience is, the application is tuned and does work, leaving the device or the last km as the only possible issues. Aspects not really under your control.
- QA analysts need to explain to the business the challenges behind User Experience testing of mobile applications. By definition, the business sponsors are not QA Analysts, and may not understand the challenges, variables or issues. There is a need to establishing which devices, which networks, which locations and which points are to be tested for base-lining etc jointly.
- No, I am not suggesting more test cases, in fact I am suggesting that the test cases be developed once and used for functional and performance testing. I am however suggesting that the test cases be run from different points and locations and shared between teams. How else would you understand dependent and independent variables and baseline? If each point and team run a different test case, we only understand that that test case passed or failed.
- Yes of course, my best practice involves using both Virtualization and Automation tools, process and people. This blog is not consulting, its for a broad general market. Your organization processes, your tools, your people, could be very different. I suggest you build your own Best Practices for your organization, this is just an attempt to get you thinking on the approach.
- This approach takes significantly less effort than current approaches I have seen and heard of. But testing for Baseline Understanding vs pass or fail, will enable far quicker identification of issues and help locate them. Understanding which issues are under your control and worth pursuing. This is an approach, and each application and environment is different.
I encourage anyone that has something to share, to comment, ask questions, share success or speak to concerns below.