How to understand mobile statistics and metrics

In the tech industry, we are bombarded with numbers. Every other day there is a new study showing percentages and huge numbers that are supposed to prove or make us understanding something.
Unfortunately, I feel like we are becoming desensitised to numbers and most people have problems taking away the right conclusions from such reports.
First of all, every study needs to be taken with a grain of salt. It is very hard for surveys and such to have the same outcome twice in a row (just search for a market share study on Google and check out the first 4 links).
However, this doesn't mean there isn't good insight to be internalized. Let's try to better understand statistics and metrics in the mobile industry so we can benefit from them.

Types of Statistics Available

Sales market share

Sales market share is the most common statistic used to track the current trends in the mobile market. This information usually comes out a month or so after the end of a quarter and tells you how many products a company or platform sold in the previous quarter.

What this stat is good for

• Following the trends in success of the various platforms and manufacturers to see where the market is headed. This can help you anticipate what platforms or handsets you need to watch next.
• When the entire market is growing as fast as mobile is, a company can have record sales and still lose ground to its competition. You'll see it here first.

What to watch for when you use this stat

• Sales numbers are not the same as installed base numbers. A new platform may sell well in the previous quarter, but it will take some time before the sales of new products exceed the existing devices that people already own.
• Know whether or not the statistic is measuring platform sales or manufacturer sales. Some analysts only look at manufacturer sales which will tell you how HTC is doing in comparison to Apple, but won't tell you how Android is doing compared to iPhone.
• It helps to understand the seasonal nature of sales. Apple historically releases a new iPhone model in late Q2 or early Q3. Ergo, Apple sales historically jump in Q3.
• Understand what market is being defined. Most of what I look at is smartphone market share. Some people believe that you should look at the entire sales of a platform. So market share for the iOS platform should include iPhone, iPad and iPod Touch. Even if this argument was correct, you'll find that it is next to impossible to find data that looks at the world that way.

Installed base market share

Whereas sales market share looks at what was sold recently, the installed base market share—sometimes called subscriber market share—attempts to figure out what percentage of each type of phone is currently in use in the real world. These numbers don't change as rapidly as the sales numbers.
Because phones will eventually be lost, stolen, broken or replaced, you can't simply add up all of the phones sold in the past to find out how many phones are currently being used. For that reason, information on the installed base comes from surveying end users.

What this stat is good for

• If you can find a match between installed base and a market that you want to target, it can help you determine what devices you need to worry about.
• It helps you understand how quickly changes in the sales numbers are resulting in changes in the number of people using new devices.

What to watch for when you use this stat

• Almost all of the numbers are based on a particular geographic area and not worldwide data.
• Just because someone has a phone, doesn't mean they are likely to use the capabilities of that phone in a way that makes sense for your business. For example, if someone has a feature phone, they are less likely to have a data plan and thus less likely to browse the web.

Financial information

I'm the first to admit that I'm not someone you want to take financial advice from. But as I mentioned above, at some point in a debate about mobile phones, someone will issue the trump card that while Apple may not be dominating in market share, that they dominate in profits. Where does that information come from?
The simple answer is that it comes from each companies quarterly financial reports. It's not hard to find the financial reports of each company so I'm going to leave that as a exercise for the reader. Instead, I'll want to highlight a few sources that talk a bit about the profit breakdown of the market.

What this stat is good for

• If you're investing, this is obviously critical to make sure you make smart stock decisions.
• Most companies provide some information about the units sold in their financial statements or their earnings phone calls.

What to watch for when you use this stat

• Unless you own stock in these companies, I'm not sure what good this information does for you. Obviously if a company is in dire financial trouble, that will impact the longevity of a platform. There are theories that we misunderstand the lesson of the pc market and that market share doesn't matter as much as we think. That we should be watching profits instead. Others say the only thing matters is market share because it has it own momentum. That companies should stomach lower profit margins for a few years while the gold rush is on to secure a large market share that will drive long term value. Who's right? I don't know.
• When you compare RIM financials to other companies, note that their financial quarter is one month off from everyone else's. It makes direct comparisons more difficult.

Platform breakdowns

Sometimes you need to find information within a given platform. Maybe you want to know how many handsets are still using a particular version of an operating system to determine if you need to support it.

What this stat is good for

• Determining what versions you need to support.
• Understanding how OS version plays into any platform fragmentation

What to watch for when you use this stat

• Data is usually extremely high level. When you encounter a bug in a particular point release of an OS, it is hard to figure how may people are currently using that point release.

Carrier data

There is a lot of information available about carrier market share, average revenue per user (ARPU), and what services are being used on each carrier. I generally don't find this information as useful for our business or customers. That said, some of the information on how phones are being used can be fascinating.

What these stats are good for

• Looking for other ways to augment your plans with services like SMS and MMS.
• Understanding the total market for something like location based services.
• Random information to wow your friends at dinner parties (Did you know the average teenager in the U.S. sends and receives 3,338 text messages per month?)

What to watch for when you use these stats

• General data may not be indicative of what your customers are going to do or the services they use.

Demographic surveys

Demographic information is one of my favorite areas of mobile statistics. I believe there are trends in mobile that the tech press misses because they aren't able to step outside the tech bubble and see how the demographics of smartphone users is shifting from the early adopter tech enthusiast to the general population and what that change means.

What these stats are good for

• If you know the demographics of your customer base, it can help ensure your mobile strategy lines up with how your customers are likely to be using mobile devices.

What to watch for when you use these stats

• The survey methodology is always important, but it is particularly important for demographic information. It is easy to find surveys that claim to have some demographic insight that have sample bias (e.g., only asking people who respond to mobile advertising).

Mobile web metrics

Back in early 2008 when Google created an iPhone optimized version of their search engine, one of the Google representatives justified the decision to develop a special version by saying, “It's about usage. Not unit.”
When you're looking at mobile web statistics, you're looking at usage patterns to try to make decisions about what you need to support.
Unfortunately, nearly every public source of data about mobile web usage is problematic for reasons I'll list below. The only data that is really useful is your own data about what your customers are using and what traffic your web site gets. And even the latter can be problematic.
If your site has a crummy mobile experience because it is slow or has requires flash, then current traffic to your site is likely not indicative of what will happen when you have a mobile optimized site.

What these stats are good for

• Trying to understand what phones and browsers are being used on the mobile web.
• Understanding how people are using the mobile web.

What to watch for when you use these stats

• Technical problems may cause mobile usage to be under reported. Many web analytics packages require javascript to function. Many mobile phones don't support javascript or even if they do, the javascript can be take too long to process on a device and fail before it can report back to the analytics server.Nearly every analytics engine provides a server side alternative that should be used to get accurate mobile information, but many sites owners don't implement them or don't implement them correctly. When looking at aggregate data about mobile usage, you have to wonder how much the data is skewed by this fundamental problem.
• To expand on that previous point for a moment, PPK has been providing reports from StatCounter which is mentioned below. On a mailing list for mobile web, Luca Passani, the lead developer of WURFL, said that he had “4 million lines of logs from a major social network” and that his analysis of those logs showed Openwave at 8% of traffic and NetFront at over 12%. Unfortunately, he can't share the source of the logs. What do you do with this information? I think you have to look at multiple sources for common elements and then get your own data from your own site and customers.
• Mobile web usage of non-mobile optimized sites is unlikely to tell you much about what the usage will look like when a site is mobile optimized. Most of the aggregate analytics information lumps desktop and mobile web sites together in the reports.

Advertising networks reports

A few of the mobile advertising companies publish reports on what devices they see on their networks. There are some interesting things that can be found in these reports. At the same time, I feel like the data from these reports is commonly either misunderstood or misused.
The reports are most useful when people take them for what they are. They have inherent sampling bias because every advertising network has some sort areas where they do better and where they don't. When they offer ads both on the mobile web and inside native apps, the reports are going to skew heavily towards platforms with robust app ecosystems.
The worst data comes from when people try to infer more from the reports than is possible or when the advertising networks use their own networks to do a survey of users.
BTW, two of the ad networks that used to provide information on a regular basis—AdMob and Quatrro Wireless—no longer provide reports as they've been bought by Google and Apple respectively.

What these stats are good for

• Some sense of the relative numbers of devices being used in different geographies.
• The reach of advertising networks if you're trying to decide which to use.

What to watch for when you use these stats

• As mentioned above, the networks all have some sort of bias. Figuring out what it is can be difficult.
• Beware networks that offer advertising both inside apps and on the mobile web. Usually these networks will only offer advertising inside apps on a couple of platforms (e.g., iPhone and Android) which means those platforms will account for more impressions.
• Surveys that solicit participants via the ad network that don't have someone working to balance the data with known demographics are likely to be inaccurate. This is why when companies do any sort of surveying, they have to know the demographics and weight the results to match the complexion of the audience. I've seen studies that ask people who responded to ads inside apps how many apps they had installed. By definition, this excludes people who either don't use apps or don't use apps that feature advertising on that network.

Apps metrics

Because of the popularity of app stores and the success developers have had selling apps, there is a LOT of information about app usage. Much of it comes from services providing analytics to app developers.

What these stats are good for

• Understanding how people are using apps..
• Helping to determine your app strategy.

What to watch for when you use these stats

• Like all of the other statistics, understanding the source of the data. Is it a survey? What was the methodology? What platforms were included in the source data?

Which stats should you care about?

Depending on what type of business you have and your role, the mobile statistics that you care about are different. Figuring out what questions matter to you is the first step in finding the statistics that will help you make decisions.

App developers

App developers need to figure out where they have the best chance to sell their apps or make money from advertising. In particular, developers are often trying to determine which platform to develop for first.
Because the factors that make an app successful are not necessarily the same ones that make a platform dominate, market share and sales numbers don't matter as much. Developers will probably weigh the benefits of building something for an established market like iOS where development is easier but the competition is more intense with the opportunities of new markets like Windows Phone 7.
App developers need to look at the app store metrics, advertising data anddemographic information (particularly for things like adoption of location-based services).

Web developers

Web developers are accustomed to being able to look at high-level statistics on browser market share and screen resolution and make educated decisions about what platforms they need to support. If they are working on an existing site, there is likely existing analytics that can be used.
When it comes to mobile web, there aren't the same easy answers. Existing analytics on a desktop site are not going to provide much insight into how a site will be used once it is mobile optimized. Some analytics systems don't even handle mobile web very well.
Aggregate information on mobile web usage is also problematic because it requires site owners to add server side alternatives to the normal javascript-based analytics code. Finally, global numbers on mobile browser usage may not match the target customer base.
In addition to reviewing mobile web metrics, web developers should also look atdemographics to try to match their target audience to the devices they are likely to use. The installed base of phones can also be useful.
The ultimate data comes from the customers of a web site or service. The more you can learn about what devices they are using and how they are using them, the better you can target your efforts.

Marketing

From a pure marketing perspective, you're going combine your objectives withdemographics, installed base and advertising information to determine where to spend your money.

Investors

They should obviously look at financial information, but looking at trends in sales market share also makes sense as an indicator of how companies are competing against others in the market.

Your role here

The point of this list wasn't to be definitive about what anyone in particular should be concerned with. The data that matters to you should be determined by your interest and objectives when it comes to mobile.
I track a lot of mobile statistics, but I know the ones I care about and why I care about them. They matter because of the type of work I do and my theories on what will happen in the mobile market.
But the data that matters to you may not be the ones that matter to someone else. Assuming they should care about the same things you do is the most common mistake I see people make when trying to understand what is happening in the mobile market.
 

Back to Blog