Analytics and automatic app feedback: A slippery but interesting slope for Windows Phone apps

In my Windows Phone 7 app I’ve enabled anonymous feedback via an opt-in mechanism: when setting up their account, they can opt-in to the use of the location service, plus opt-in to sharing anonymous use and crash reports.

I’m trying to be a good citizen by asking users to opt-in, having a clear privacy policy, and keeping things simple. Not all developers do this, but it’s good practice I believe.

In return I’m using the data to better understand:

• who my user base is (is it growing, constant, etc.)
• what parts of the app I should be improving (I found lots of searching being done, etc.)
• if I should consider localizing the UI for other countries of the world

A side benefit is knowing that thousands of times the app has been used, it’s in real hands. My app has used for over 37,360 check-ins since being released (as of a few nights ago), and I can keep track of things ahead of waiting for the marketplace’s data that comes out many days behind.

As interesting as data is, it takes work to do this all right. App developers should consider using analytics at some point if it makes sense – at a minimum, during your beta development period, to learn what parts of the app are used the most.

By understanding better what key parts of an app my users are spending time on, I can work to figure out where to invest time in making improvements in future updates, and if you’re smart with the data (don’t be greedy, don’t be inappropriate with it, etc.), everyone should be OK with that value.

With such great power comes responsibility, too, so let’s tread carefully out of respect of our users… there was a short exchange of emails about this between some passionate Windows Phone developers today and I just wanted to at least share my experience with this post this evening.

Debating analytics

Analytics can range from the basic (what size screen is being used? what “page” in an app has been navigated to?) to crazy (where was the last touch on-screen? a unique representation of the phone? the phone number of the user?). Thankfully the Windows Phone is very secure thanks to its nice sandbox for apps to play in, and so app developers cannot get ahold of data like other platforms (phone numbers, etc., are off-limits).

That’s a good thing and a bad thing: as a developer, I’d love to be able to give my user’s a big “find friends I may already know by examining my address book” option… but the idea that “Joe’s random duck noise app” might be able to get my contact list without me knowing is scary. So I’m glad that restriction is in place today.

The Wall Street Journal has a nice interactive analysis of many of the top iPhone and Android phone apps online here – they did some interceptive research into the data stream that apps had, what was being sent over the wire by apps, etc. Very good investigative journalism! It is clear that there’s a lot of interesting things happening, and the data can be useful for many purposes.

I’m all for analytics when it’s acknowledged, useful, and clearly covered in a privacy policy. Detractors may dislike the concept, the additional bandwidth, etc., but this sort of technology has been around for so many years that we’re all used to it on the web. Most analytics implementations use 1x1 clear GIFs, sending the data along in the query string of the image’s URI. We’re not talking about gigs of data here.

At this time there is not a marketplace ingestion requirement that your users must opt-in to all the data types sent from your app, so as developers we have a responsibility to tread with care – it’d be hard to enforce, but letting users know is important in the trust relationship between developer and customer.

I personally don’t like the idea of apps that use analytics without asking for opt-in first, but they’re out there, and clearly controls like the ad controls get tons of data for those advertisers. These things fund apps, so who knows...

Windows Phone is very secure

To re-emphasize, Windows Phone is secure if you compare it with many other platforms: today a Windows Phone app cannot troll the list of contacts on the device; if it asks for the unique phone identifier, or other information like the make/model of the phone, that capability is detected at ingestion time, so a user about to download your app will receive a clear indication that you app makes use of that data, leaving the consumer to make a trust decision before even agreeing to install & download the application. Slick!

One interesting thing is that the concept of the “unique phone identifier” is combined with other key properties, all exposed through an interface called the ‘extended phone properties’ – info like the amount of memory available on the device, the peak working set of the process, etc. It would be nice if this was more clear: advertising controls probably want the unique ID, but many developers may just want the simple memory counter, etc., for debugging and error reporting purposes.

I have a few mitigations to worries that people have about analytics and data, and I hope as app developers you will consider both…

Mitigation ask #1: opt-in for analytics

Treat your users with respect, offer an opt-in to the analytics. Please!

This is easier for apps that already must prompt users, such as the location sharing/location service apps, so understandably apps that are “ready to go” from the start might need to come up with a better process. Considering many settings screens are hidden away, I would rather prefer an Opt-in prompt than one day finding an Opt-out toggle switch hidden away…

In the code path for enabling and updating analytics, check for the user’s explicit opt-in permission (in isolated storage, etc.). If they haven’t given it, don’t even invoke that part of your app to startup analytics.

Mitigation ask #2: have the right privacy policy

A marketplace ingestion requirement is the link to your privacy policy, so you should already have a privacy policy (no matter how basic). But also let’s get specific in its text and use:

• Clearly call out in the privacy policy that anonymous use statistics are gathered (analytics)
• In your app,

• Include a hyperlink button to the privacy policy on the page where you ask a user to opt-in
• Include a hyperlink button to the privacy policy on the ‘About’ screen of your app

If you include these things, users that are very concerned about their privacy and how you might use the data will have something to fall back on. Not storing personally identifiable data is a good thing all-around, but gathering statistical data (like pages hit) seems acceptable for most folks.

Looking at my data

Now that my app has been in the Windows Phone Marketplace for more than a week, I have some data, though it’s difficult to interpret at this point.

App Hub / Marketplace stats

These stats are available to all app developers, right in the App Hub, but they lag by many days behind (credit cards need to clear, etc.), so you’ll need to be patient. Clearly I’m getting a few downloads, but who knows how the last 7 days have gone, given this report ends in last week.

So without analytics, at least I know “a few thousand people have downloaded my app – sweet”.

Analytics drill-down

Now, if I open up my analytics’ provider’s site, I can drill into the data that the analytics feed has been providing. I don’t have any way of knowing what percent of my users have left the “anonymous use and feedback” checkbox checked, but I hope a lot of them have. I don’t have a way of knowing if someone decides not to opt-in.

Just a quick look, right around 3/21, I can see the same jump as the marketplace’s cumulative total: obviously the app went out and people downloaded it.

From then on, although the marketplace still hasn’t given me information, I can see that my users must be happy: they’re continuing to use the app, day in and out. You can also roughly see a pattern emerging where around the weekends the app seems to be used just a little bit more – fluctuating maybe 20% by day.

Let’s dig a little deeper into what “pages” were viewed. These pages are effectively the XAML pages. I’ve also highlighted in Photoshop two of the “pages” for specific reason. In my analytics implementation, I decided to only have an analytics event when a full page is navigated to – so if you navigate from the “friends” to the “places” pivot item, I don’t record that as an event – it seems simpler, and is less chatty from a data standpoint.

Most users of the app will probably have a total of 5-7 analytics events during that use of the app, and the amount of data sent is super small.

So looking at this set of data, here’s what pops out:

• Highlighted in yellow, #13, is the number 1,600. This is my rough estimation of the number of new downloads of the app, with some error margin. I’m assuming that almost everyone who uses my app will setup an account, or sign in with theirs, taking them through this codepath. This number seems to match at least part of the download numbers from the App Hub – though the dates for the reports and ranges of days are different.
• The green highlight above shows that there were 37,630 check-ins in the few days range of the report, which is a good metric of how many people are able to use the key functionality of the app.

And now, on to the map…

Here’s the map generated on the analytics side showing the countries only that people were using the app. A note: although my app has location services, the GPS coordinates and information are not sent through analytics, and not recorded. This is clearly called out in my privacy policy. Analytics providers are able to estimate the country of use by the IP address used to make the request, so this gives a rough estimation of the countries in the world where the app is in use, weighted by visitor use.

This data tells me that I have a lot of US users, and users across the world, but right now nowhere is highlighting as a place that I might want to consider localization a must-have.

Good and fun to know!

Reacting to the data

My initial, unexperienced take on this data…

• I have a clear userbase that has begun using my application regularly
• I’m not growing my userbase much, so I need to address the key complaints, suggestions, and bugs reported by my existing users
• I wish that I could address the OAuth2 sign-in experience for first-time users, it looks like it’s taking over a minute for this step, and about 1/5 of users may be having trouble signing on properly and giving up.
• Search is being used a lot more than I anticipated, I need to polish and fix those pages!

Implementation details

At this time I’m not sharing my analytics implementation (I’m using Google Analytics b.t.w., free, easy, and I already use it with my web sites and blog, so I’m familiar with it), but there are a few things – from the Silverlight analytics framework to other providers people are using. I’ve also heard that the Preemptive product is good. I need to cleanup my code in the meantime!

In my app, I simply hook analytics up to the navigation service events, so every time a new primary page (such as “About” or “Explore a place”) are used, I fire off an analytics event that is very small (a few hundred bytes sent).

Hope this is interesting.