Padraic Woods
Written by Padraic Woods
Published 2014-05-28

Setting up a mobile device lab

/

It is becoming more and more challenging to test apps and websites on different devices, with different screen sizes, running different operating systems and with multiple browsers installed. How can you guarantee that your website or app works on every device and on every browser?

Nothing beats testing on the same devices as your users, not emulated or simulated versions of those devices. Physical interactions like pinching, zooming and scrolling, hardware features like the camera, GPS locations, the accelerometer, testing real occurring events like battery consumption or site performance are all best tested using physical devices.

Entry image

What is a mobile device lab?
A mobile device lab is a collection of mobile devices available for testing new apps and services.

What were the mobile device lab requirements?

  • The device lab needs to represent the mobiles devices, the operating systems and the browsers of our users. Older devices are retired from the device lab in correspondence with the OS/browser/devices we support at that time. We also ask the testers not to update the operating systems on the devices. The devices chosen for the device lab are based on
    1) the most popular devices visiting our free website vg.no
    2) the most popular devices running our Android, iOS and Windows apps
    3) the devices of users who have reported bugs
    4) a range of devices close to retirement. A combination of devices running the minimum versions of OS and browsers we support.
    5) new devices we believe will be popular.
  • The device lab needs to contain devices of different screen sizes.
  • The device where possible should support both wifi and cellular connections: We want to test the devices both on wifi and and the cellular network. Where possible we bought devices that support both. The tester needs to use their own sim card on the test phone if they wish to test on a cellular network.
  • It should be possible to lock the device lab. Devices are expensive and we don’t want to loose any of them.
  • Devices need to be constantly charged and immediately available for testing. There is nothing worse than having to wait for a device to charge each time you want to test something.
  • The device lab needs to be accessible for multiple departments. The device lab needs to be placed in a location where all departments can access it.
  • It must be possible for a tester to loan the device from the device lab and test at their desk.

Who uses the device lab?
Web developers. app developers, designers, testers and the the support department. The device lab is shared among all departments and internal companies at VG. For example VGTV (web TV company), Tactus (internal ad agency), E24/Dinepenger (Global and personal financial publications) and VG (news site).

What are the alternatives to owning a physical device lab?
There are a number of commercial and open source alternatives to owning a device lab and physically testing on devices. They range from changing the user-agent in your browser to appear as if its a mobile device, testing on remotely accessible device labs, crowd sourced testing or using emulators and simulators to test.

The most frequent and easiest way to test is by switching your user agent in your browser to simulate a certain device. (e.g User Agent Switcher)

There are numerous commercial and non commercial emulators and simulators available to help test devices. They include websites that offer browser emulation, physical device labs in the cloud and Google and Apples simulators.(List of emulators available, 19 Tools to Test your Site for Mobile Devices)

Crowd sourced testing
A very effective way to test multiple devices with different network connections and located at different locations is through crowd sourced testing. For example releasing alpha and beta versions of your Android app to a limited number of your android app users.

There are numerous device labs available where external developers and designers can go and test their products. Check out Open devicelab

Some device and OS statistics for vg.no
TabletPageViews

TabletVisits

MobileVisits

MobilePageViews

DevicesVGAndroidApp

OSVGAndroidApp

DeviceVisitsVG

How are the devices setup?
All devices are installed with the default factory settings. The devices are constantly connected to a power source. We have created an Apple-ID, a Google account and a windows live account to be used on the different platforms. Each device is registered in a document displaying device name, Operating System version, screen size and each device has the same information displayed on a sticker on the back of the device.

Constantly powering up the batteries may expedite the degradation of the batteries. It may be worth using a wall socket electricity timer to limit the frequency of top ups. For example turn off the charging of devices at night.

How do people test devices?
People are asked to sign-in and sign-out devices. They register with their name, the date they took the device and the name of the device they are testing. When they return the device they register that they have returned the device.

Where do we get the devices?
The majority of our devices have been bought or donated to the device lab by employees.
We try to buy in the newest most popular brands when they are launched.

What does the device lab look like?
There are several possibilities ranging from making a device lab to buying a commercial solution. We chose the latter and ended up buying a “TabCabby Charge & Sync”
A movable and lockable unit connected to a power supply that can store 25 devices. The unit also has the ability to sync devices from a master device or compile apps on multiple devices at once from one central connection (for example from xcode to iOS devices).
VG_DeviceLab3

VG_DeviceLab2

VG_DeviceLab1

Tab Cabby

Other physical solutions include vanamco device lab and Griffin multi-doc

How do you test in different browsers?
For a comprehensive overview of how to synchronize testing of a website on multiple devices check out synchronized cross device testing using Ghostlab, remote preview, Adobe Edge inspect and grunt.

Possibilites:
Adobe Edge inspect installed on all devices so that you can synchronize webpage loading on all devices.
SHIM: Shim is a node.js app that enables simultaneous, synced web surfing across a variety of devices and browsers.
Grunt: The Javascript task runner.
Ghostlab: Sychronised browser testing for web and mobile.
Remote preview: Preview a url on a large number of devices simultaneously.

Overview of Device Versions
Windows: http://en.wikipedia.org/wiki/Windows_Phone_version_history
iOS: http://en.wikipedia.org/wiki/IOS_version_history
Blackberry: http://en.wikipedia.org/wiki/BlackBerry_OS#Version_history
Android: http://en.wikipedia.org/wiki/Android_version_history
Android version comparission: http://socialcompare.com/en/comparison/android-versions-comparison
Android % distribution: http://developer.android.com/about/dashboards/index.html
iOS version statistics: http://david-smith.org/iosversionstats/

Written by Padraic Woods
Published 2014-05-28