A Sunday afternoon project: Providing App Store stats via JSON

Last month Apple announced their auto-ingest tool for fetching the download and update statistics for apps.  The tool downloads the stats for a particular day or week as a gzipped CSV file that you can then process yourself.  This is far nicer than having to try and scrape the numbers from the iTunes Connect page although the daily numbers are still restricted by a two week limit and the weekly by 13 weeks.  Hopefully in the future you’ll be able to retrieve further into the past (assuming the data still exists).

I’d a bit of free time this weekend and decided to have a poke at this and see if I could  grab the data and present it in a easily consumed format (i.e. JSON).  Another reason for looking at this is because I have two iPhone Developer accounts and it’s be nice to see the data from both as one graph rather than having to log in and out.

To that end I wrote a couple of lumps of PHP to ingest and process the stats which is available on GitHub (https://github.com/kmonaghan/itunes-connect-auto-ingest).  There’s a couple of small steps to get it all to work.

First, create a database (or just use an existing one) and create the table in schema.sql.

Next, edit boot.php.  Here, you need to put in your database details and the details for each iTunes Connect account you have.  To do this you need to put in an array with the following key pairs:

array('username' => 'iTunes Connect username',
'password' => 'iTunes Connect password',
'vndnumber' => 'VND number',

The username and password are the details you use to log into iTunes Connect.  You’ll find the VND number if you log into iTunes Connect and go into ‘Sales and Trends’.  It’s the number beside your name at the top left of the screen.

Update: From the comments, this is how you find the Vendor ID (VND):

Choose Sales and Reports.

Choose “Reports” from the top left menu (Where default is “Top Content”).
From the form that will be shown, choose monthly report and download.
The downloaded file will be of format S_M_(VendorID)_TIMEPERIOD.TXT

You can have as many of these accounts as you want and the ingest script will check each one.

Now all the details have been set up, you can ingest the data from Apple.  This is done by simply running ingest.php.   If it’s the first time the script has been run, it will look back 14 days.  Otherwise, it will look in the database for the last successful day there was an import and attempt to import all days since then.  After your first import, you should create a cronjob to run this script once a day to capture the latest stats.  If the downloaded tar files are 0 bytes, check your login details and VND number.

Once all the data is in the db, we can view it via daily.php which outputs the results as JSON.  This will by default output the total number of units downloaded for all apps in the last month.  It can take four parameters:

  • apple_identifier: A particular app identifier.
  • product_type_identifier: This can be 1 (default) which is downloads or 7 which is updates.
  • from: The date to get the stats from.  Expected as dd/mm/yyyy.  Defaults to 32 days ago.
  • to: The date to get the stats to.  Expected as dd/mm/yyyy.  Defaults to yesterday.

And that’s that!  There’s some Javascript in stats.html which consumes the JSON and uses the Google Chart Tools to display the results.  An example of some live data using my own apps is available here.

This is very much a works-for-me lump of code, so it may not do exactly what you want but it should be easily extendable to provide exactly what you want.  It will certainly not cover every error condition.  I’ll probably add bits and pieces to this as I think of a stat I want to visualise or I come across a bug unexpected feature.

Something I might consider looking at in the future is writing a clone of the iTC Mobile app that utilises the saved data.  It’d be a nice little project to try out Core Plot.

A year in the Broadsheet

It’s been a year since Broadsheet.ie and launched and it’s been a rather interesting year to see it grow with such limited resources.  I’m going talk here about some of the bits and pieces that have interested me.

Obama > Queen > Stephen Ireland

For the last three months, Broadsheet has done over 1 million pageviews a month, which accounts for about half the total since it started.  Considering the site started off on shared hosting and is now on a vps, it’s pretty good going.  For caching, the site had been using WP Super Cache but it seemed to struggle somewhat (this, in fairness, could be down to being misconfigured).  I switched W3C Total Cache and that seems to have improved things.

The first standout post the site was one about Stephen Ireland’s house.  It was one of those posts that caught people’s imaginations and spread like wildfire across the net.  It reappears every so often on football sites, usually to howls of incredulity.  It remains the second most popular post on the site.

There’s been a couple of outstanding days where there’s been a general surge in traffic, most notably during Obama’s (78,041) and Queen Elizabeth II’s (69,154 64,735 55,199) visits.  The nice thing about days like those is that there’s generally a little extra boost to traffic afterwards.

However, despite all that, the most consistently viewed post is a silly screencap of a Yahoo question.  It ended up on Stumble Upon and people are still coming to the site nine months later for a cheap giggle.

Something that surprised me was the number of people that viewed the previous posts pages.  There’s a huge drop off from those that view the second  compared to the first page and continues to drop off rapidly.  The fifth page has less than 1% of the views of the first.  So the moral of this story is content dies very quickly.

The top five posts of the last year have been:

Facebook > Twitter > Google+

Traffic from Facebook has been pretty consistent, being about 1/3 to 1/2 more referrals than from Twitter.  That said, the site has been taken down via links from Graham Linehan and Charlie Brooker’s twitter streams.  Google+ is yet to have an impact, with barely a trickle of incoming links.  Both Boards.ie and Politics.ie both provide a steady supply of views, usually for the politically charged stuff (and pictures of Irish models, of course).

The top five referrers were:

  • Google
  • Facebook
  • Twitter
  • Stumbleupon
  • Boards.ie

Dirty Old Goats

The growth in traffic from Google has grown steadily since launch with some notable spikes (Bing and Yahoo provide so little traffic they’re not worth talking about).  Brian Cowen hungover on the radio and then the sketch on Jay Leno was the first example of search traffic spike.  Neil Prendeville and his misadventure though is the king of spikes, adding 4.7K visits to a baseline of under 1K at the time.

But the queen of keywords is Madeline Mulqueen after her appearance in that Rubberbandits video.  Plenty of dirty beggars out there looking for more photos of her and who are we to disappoint?

Six of the top ten searches bringing people to the site involve ‘broadsheet.ie’ (mostly because people are too lazy to switch to the URL bar), so excluding them the top ten is:

  • madeline mulqueen
  • gary busey
  • neil prendeville
  • alison o’riordan
  • jean byrne
  • dominic hyde
  • sarah carey
  • jay leno brian cowen
  • damo and ivor
  • brian cowen drunk

I wonder if Alison is happy with her internet fame?

IE6, why won’t you die?

Firefox rules the roost, with 29.98% of the browser share.  IE follows with 24.70% but Chrome is snapping at its heels with 23%.  Safari lags behind with 15.74% (which would include both desktop and mobile versions).

8.72% of all visitors still came via IE6.  The only glimmer of hope is that this month it’s fallen to 7.88%.  One day, it will be ground into the dust and be forgotten about like a bad dream.  But that day is not today.

Android accounts for about the same amount of traffic as iPads and iPods combined but is still 1/3rd of the traffic from iPhones (and if there wasn’t an app it would only be higher). After that there’s there usual suspects of Blackberry/Nokia/Samsung (in that order).

Half a year of App Store stats

As we enter the second half of the year, it’s as good a time as any to look back at how my apps have been doing in the Apple App Store.

These numbers cover from the 3rd of January to the 3rd of July, 2011.  As I’m a fool, I hadn’t downloaded the stats for the months prior to this, so this post is a record for myself as much as anything else.

Pint of Plain

Downloads: 2,589 (down 1,709 on the previous 26 weeks)
Updates: 4,979
Pint of Plain was my first app and after 18 months on the App Store.  At the start of this year, I released a major update which was essentially a complete rewrite.  One of the changes I made was to include iAds.  It’s not been a huge success for me, with revenue of $4.92 for the last six months.  I’ve not seen a live iAd myself yet so I’m assuming part of the problem is a low fill rate.

The app has a pretty poor rating but I’ve never been sure how much of an impact this has had.  If I do another version, I may abandon the current version in the App Store and essentially relaunch with a rebranded app.


Downloads: 694 (up 417 on the previous 26 weeks)
Updates: 28

TaxCalc.ie is my only paid app.  I was originally selling it at the €1.59 price point, but over Christmas I decided to experiment with dropping the price to 79 cents.  Sales increased by a little more than double which meant I was making a tiny bit more than I was at the old price.

The app is regularly #1 in the Irish Finance apps but this doesn’t translate into huge numbers of downloads.  It has rarely had more than 10 downloads in any one day and there have been times it has taken the top spot on the back of three or four downloads.  Still, I don’t look a gift horse in the mouth and I get a little thrill when I see the app icon used for Finance in the category listings.

The low number of updates isn’t surprising since there hasn’t been an update since early December.


Downloads: 4,330
Updates: 3,045

Broadsheet.ie was the first of two apps I’ve released this year.  It’s based on the first version of TTWordPress (which I originally wrote for the Visionary.ie app).  Of all the apps it certainly gets the most usage and there is a constant stream of reviews and suggestions.  The app certainly wouldn’t be such a success without the content coming from the website since it is rather basic.


Downloads: 1,290
Updates: 320

Visionary.ie is the other app I’ve released this year.  Considering it’s a very niche market (i.e. mostly clients of Darren) I’m very pleased how well the app has done.  There was a huge surge of downloads initially as the app was mentioned in a photography newsletter.  Since then it’s slowed to a trickle but that was fully expected.

Of the four apps, I track usage in three of them via Flurry (TaxCalc.ie being the odd one out as I don’t want people to think I’m sending any of their tax information outside of the app).  The most interesting piece  of information for me is the percentage of users with iOS 4 and the percentage using an iPhone 3G.  Between 66% (for Broadsheet.ie) and 78% (for the other two) of users have a device running iOS 4 and less than 3% still have a 3G.  Considering that there’s been just under 7000 users of the three apps (with very little cross over between them) that gives about 210 3G to 5460 on iPhone 4, iPhone 3Gs and iPad users.  This makes me think it’s getting to the stage where dropping iOS 3 support is not going to hugely impact on downloads.

Overall I’m happy enough with how my apps are doing, especially considering how little I directly promote them.

The executable was signed with invalid entitlements. (0xE8008016)

Update 2014/03/12 This QuickLook plugin is a great way of seeing what’s in your provisioning profiles: https://github.com/chockenberry/Provisioning

The code signing for iPhone apps can drive me batty sometimes.  It rarely works first time for me and the errors you get are completely unhelpful.  It doesn’t help that for a variety of reasons I’ve 3 developer profiles on my work machine.

Today, I’ve been getting the following:

The executable was signed with invalid entitlements.

The entitlements specified in your application’s Code Signing Entitlements file do not match those specified in your provisioning profile.


The very least they could do here is list which entitlements are not matching.  To figure this out, you need to view the provisioning profile and find the Entitlements key.  It’ll look something like this:


You need to make sure the contents of your Entitlements.plist match these exactly.  After you’ve made your change, clean your project and rebuild.  Your app should now install and run on your device.

TTWordPress – a Three20 library to display a WordPress blog in an iOS app

DEPRECIATED use KMWordPress instead.
DEPRECIATED use CBPWordPress instead.

TTWordPress is a library I wrote to easily display a WordPress blog in an iOS app.  I originally wrote it for the Visionary.ie app and was then able to expand on that to create the Broadsheet.ie app.  Since I found it useful, I thought I’d release it into the wild and see what happens.

To provide the data from a WordPress blog is a nice machine readable format, the excellent JSON API plugin is used.  Obviously, it needs to be installed for the iOS to be able to retrieve anything but that’s a trivial matter.  If you want users to be able to comment on posts, you need to activate the ‘Respond’ controller in the API settings.  On the iOS side of things, the library uses Three20 for the consuming the JSON and displaying the information in tables.

To use the library in your own app, you first need to add Three20 to your project (if you’re not already using it).  Open the example TTWordPress project and copy the WordPress folder and all its contents to your project.

To point the library at your own blog, you need to edit the following line in TTWordPress.h:

#define WP_BASE_URL @"http://ttwordpress.karlmonaghan.com/"

In this file, you can also change the default title for the latest posts view and the way dates are displayed for posts and comments.

Then in your AppDelegate, add the following includes:

#import "WordPressBlogViewController.h"
#import "WordPressAddCommentViewController.h"

Then add the following lines:

[map from:@"tt://examplepostlist" toSharedViewController:[WordPressBlogViewController class]];
[map from:@"tt://blog/author/(initWithAuthorId:)" toViewController:[WordPressBlogViewController class]];
[map from:@"tt://blog/category/(initWithCategoryId:)" toViewController:[WordPressBlogViewController class]];
[map from:@"tt://blog/post/comment/(initWithPostId:)" toModalViewController:[WordPressAddCommentViewController class]];

The first line sets the path to a view controller that will list the latest posts from your blog, the second shows the posts belong to a particular author, the third from a particular category and the fourth is the modal pop up for making a comment.  The assumption here is that the list of posts from your blog will be a tab in your app hence being a shared view controller as opposed to the other two which are created and destroyed on demand.

The final step is to call the latest post view controller from somewhere.  In the sample project, it’s added to the UITabViewController. Once you’ve done that, compile and run your app and you should see your blog posts.

The code is available via a Git repo at https://github.com/kmonaghan/TTWordPress.

If you do use it, I’d love to hear from you.

The icon used is from the excellent Glyphish set.

DEPRECIATED use KMWordPress instead.

Jobordole.com: a rebuttal

On Sunday night, I launched jobordole.com, a simple calculator that tries to work out if you’d be better off working or on the dole.  This was all inspired by an article about two people being offered a job at €28K and refusing it because they were better off on the dole.  I don’t know much about the ins and outs of claiming anything from welfare, so I decided to educate myself and see if this was true (as I suspected it wasn’t).

I’ve been getting traffic today from a blog called ‘Progressive-Econonomy@TASC‘ where there was a post by Michael Taft called ‘A life of luxury on the dole‘.  The post takes issue with how I’ve made my calculations and the language I use.  Since I’ve been responding to people on forum threads, I’m going to do the (unusual for me) step of responding here to his criticisms.

First off, he’s reading far, far too much into how I phrased things.  I used ‘you could get‘ for the dole part as your means tested and the value displayed only the possible maximum amount.  To make this clearer, I’ve changed it to ‘The maximum you could be entitled to‘.

I used ‘ You get’ for a person’s pay as if you are paid, the figure is far firmer.  I can determine’s a persons take home pay with far greater accuracy as a large chunk of the variables are already known.  It also could be because I work so automatically think of myself in these situations.  But that would be reading far, far too far into a throw away sentence written around a far more important value.

On to his first proper complaint – Family Income Support.  This is a fair complaint and I’ve included it in the calculation.  However, for the example at hand, the mythical job refuser would still get more on the dole (€15.81) so my calculations are still valid.  I’ve added a few links to the FIS information on welfare and fingers crossed, one of the many people who don’t claim it will see it and start benefiting.

The second part of the argument if about the rent supplement.  I found the numbers on how many people are actually receiving the supplement to be extremely interesting (and far lower than I would have guessed).  However, just because only 13.5% of those on Jobseekers Allowance get it doesn’t invalidate my calculator.  As he points out, one of my assumptions is that you do get it.  If you know you don’t or wouldn’t get the rent supplement you can easily see from my breakdown that it’s not worth refusing a job.  It is a key factor and I make no apologies for using it in the calculator.  I have added the stat to my assumptions so people are clear that there’s only a slim chance of receiving it.

On to the number I picked for the rent.  It is the maximum and it was picked so the example would get the maximum possible rent supplement.  On the contention that the Daft report says the average rent in Dublin is €993, I can’t see that figure directly (and I’m too lazy to add the number for the various Dublin areas myself).  What I do see is that the rents in the Dublin area can range from €850 to €1,284 so while €930 is at the bottom end of the range, it is still within the average rent.  Also, the prices used in the Daft report are the displayed price and not the price a property was rented at.  My own (and many friends and acquaintances) experience is that at the moment the asking price is only a start and places are rented for much lower.  However, anecdotes are not evidence so we’ll go with the Daft figures. 10 of the 24 areas have average rents under €930 so I don’t think it unreasonable that a family could get a two bed at that price.  A quick search on Daft for 2 beds under €1000 (since you can’t pick €930 exactly) throws up a number of properties for or under our magic €930 (which will accept the all important rent allowance).

The next part of the complaint is that €930 is higher than you’re allowed elsewhere.  This is true, and I don’t hide that fact.  The table listing all the maximum rents allowed in each area is made available via the link entitled ‘Show/hide a list of the limits.’.  Our example family though lived in Dublin so it’s not going outside of the parameters.  Going on then about the different limits without changing the example is moving the goalposts somewhat.  I’m making a claim about a very particular set of circumstances.  It’s easy to plug in numbers for the other areas and see if the same results come out in favour of taking the dole (not at all).

I’m not claiming that everyone gets €191 a week in rent allowance.  It is worked out individually for each set of numbers put in.  The figure of €191 only applies if you are paying exactly €930 a month in rent.  It’s disingenuous to imply (as is done in the paragraph about the average received) that the calculator always assigns this amount.  If you do take the average of €106 for rent allowance, then you’d need to earn over €20K before a job become more financially rewarding.

So at the end of this, my contention and calculation still stands – if you live in Dublin, have a spouse and dependant child and if you are entitled to rent allowance (and get the maximum allowed of €930), you are better off on the dole.  All that Michael Taft’s post has done is put meat on the bones and tell you how likely it is for the conditions to be right (which, in fairness, is very much worth doing).

What bugs me a bit is that Mr. Taft didn’t bother to scroll a little further beyond the assumptions and click on the contact me link and get in touch.  I’m always open to suggestions on improvements and I’ve been keeping an eye on a couple of threads where the site has been mentioned to see if there’s anything I can easily add.  I found out about the post from my access logs, a mention in a thread and an e-mail from a randomer who likes the idea but thinks I should improve it.

As a final parting shot, I take issue with the post title.  I’m not saying a person should go on the dole because it’s all sweetness and light and fun for all the family.  I am merely pointing out in some very specific situations a person couldn’t be blamed for turning down a job as they would end up financially worse off.  In fact, most of the time the calculator shows that people should get (or stay in) a job.  There’s plenty of other reasons to take a job but if a family is struggling in the current harsh climate, can we really blame them for picking the financially sounder option?

I will certainly not be taking down the site but I am open to comments on how to improve it. Thanks for the traffic, in this economy every little helps.

Pint of Plain V2.0

Last week, the V2.0 of my Pint of Plain app was approved for sale on the Apple App Store.  It’s a major update of the app, with most of the code base completely rewritten.  I think this version is a vast improvement on V1.2 and I’m planning a few more updates once I get some free time again.

What’s been interesting (for me anyway) to see over the last week is the number of people that download the update.  From what I’ve seen from previous updates, the bulk of users will update in the first week and then a trickle after that (in a classic long tail style graph).  Since it was approved, there’s been 1,144 downloads.  This compares to about 10,000 total downloads since the app was released in December ’09.

It’s a useful app to test out various libraries I come across so I decided I’d mention the ones I used this time around.


The core functionality of the app uses the fabulous (if a little delicate) Three20 library.  Using the table classes and JSON library meant I could quickly rebuild the primary screens of the app.  The crowning glory though is the photo gallery.  It’s pretty straight forward to add a slick gallery once the images have been pulled from the API.

I have been frustrated though that when a new version of Xcode is released, the library needs tweaking before it works again.  There’s been promises to improve this in future releases, so fingers crossed there.

The lack of documentation is a hinderance as well.  This is mitigated by the fact that most of the methods are well named.  Still, when you’re not sure where you should be even looking, it can be annoying.


Appirater is a great little library for quickly adding an alert reminding people to rate the app.   I’m not hugely convinced that a good rating boosts downloads all that much, but since this is so simple to use it’d be a shame not to add it.


In this version of Pint of Plain, there’s a few more settings the user can tweak.  I doubt many people ever look at the various settings you can change in the Settings App so I wanted to offer an in app alternate.  InAppSettingsKit does all the heavy lifting for you and like Appirator, so easy to include it was a no brainer.


In the old version of the app, a bar’s location was simply determined by using the GPS and only used if the accuracy was under 50m.  This was never a great solution and I wanted to give the user the option of dropping a pin the right location.  While search for how to do the drag and drop, I came across MapKitDragAndDrop which did everything I needed.


Not a code library but rather a set of icons, Glyphish is a great resource.  There’s an icon for nearly every tab you need and they’re all beautifully crafted.

I think the fact that there’s so many libraries and resources out there now shows how iOS has matured as a platform.  Every time I go to find out how to do something I’ve seen in another app, there’s a library for it.  It certainly makes development much easier than when I first looked at it all over 2 years ago.

How much poorer am I now?

So on the day of the government press conference, one of the lads in work jokingly asked if I would be making an update to TaxCalc.ie to show the changes announced.  He even suggested a domain for a single serving site – howmuchpooreraminow.com. Since I’d nothing better to do, I decided I throw something together.

Going about creating it

To get the 2010 breakdown, I converted the class written for the TaxCalc.ie app from Objective-C to PHP.  This was pretty trivial and helped me refamiliarise myself  with the current tax system.  It also gave me a chance to look at updating the calculator for the 2011 budget.

Rather than flood people with options all the options of the app, I decided just to give two choices for personal circumstances – single and married with 1 income.    The only tax relief I was interested in were rent, union subs (because these two were going completely) and pension (because of the large changes).

A trivial form, some if loops and and ignoring user input validation I had a crude calculator knocked together that made some sweeping assumptions about the changes in the upcoming budget.

Initially, I was only concentrating on how much worse off you’d be in 2011.  Based on some feedback from twitter, I changed things so it included some of the extra charges (increased registration fees and the site tax) as well as predicting the pain over the next 4 years.

To make things work, I had to do a bit of hand waving and educated guesses.  The press conference was light on actual details and none of the article in the paper were much help.  The Times supplement gave me the most information but I still had to make plenty of assumptions.

My assumptions

  • PRSI, income levy and tax rate (i.e. the 20% and 41% rates) stay the same for the 4 years
  • Rent and union sub relief is completely removed in the 2011 budget
  • Tax credits (i.e. how much you earn tax free) drops by 66% in 2011, 14% in 2012, 10% in 2013, 10% in 2014
  • The cutoff point (i.e. above which you pay 41%) drops a total of €6000 in increments 66% in 2011, 14% in 2012, 10% in 2013, 10% in 2014 (this was the only way I could figure out how someone on €55K would drop by €1860 by 2014)
  • The drops in credits and cutoff are based on how much the government plans to get from the changes in income tax over the next few years
  • If you’re married, you’ve only 1 income
  • College ‘registration’ fees will not rise again
  • Site tax does not rise in 2013 or 2014

Lessons Learned

1. A surprising number of people will enter a number with a comma (e.g. 25,000 as opposed to 25000).  Since I was expecting an integer and the string was URLencoded this resulted in no net change for the user as the gross was interpreted as 0.  A lot of people were confused by that so it was the first fix.  I’m guessing it’s because the number involved pay and people are used to seeing the number with a comma.

2. Just because it’s a throwaway site doesn’t mean people won’t check if the HTML validates.  When people said this to me at first I thought they meant validating inputs.  It didn’t take much to sort out (the damned tweet and share buttons were the most awkward fixes).  Next time I’ll be validating before showing it to anyone.

3. People on the internets are lovely.  I don’t do prettty, I make things work.  Hence the site was black and white with a plain old H3 tag and some centering.  A couple of hours after it started spreading around, I got an offer for some CSS from Stephanie Francis, which I jumped on.  Now the site looks like some care and attention was poured on it!

4. People need examples.  There was much confusion over why someone on €500K was no worse off than someone on €50K.  People seems to miss out on the fact that higher earners are being hit by reducing the tax efficiencies they can use for tax avoidance.  So the change in the pension relief will hit them hard.

5. People don’t read credits but those credited with the work are happy to point out the right person.  A couple of people who saw the link on twitter asked the person sharing it if the site was accurate or why they had a problem.  The asker was quickly put right and directed to me.  Which was nice.

6. Simple pages that are just a web form are now web apps.  It goes to show how much the concept of apps has taken hold that people don’t see a difference between a single serving site and an app for a phone.

What next?

Well once the budget is out on the 7th of December I’ll be able to update the details for at least 2011 and possible the following 3 years.  This will obviously make the figures more accurate (and probably more depressing).  They’ll still be a little off for some people as I won’t be including every relief and benefit, but they should be a good indication.

I’ll also be able to include things like changes to child benefit and other social welfare payments.  This should bring home how much the average family will be hit over the next few years.

Brew night

Tonight I’ve been mixing up my second proper batch of homebrew (the first doesn’t count as I never bottled it). This time I’m brewing Coopers Brewmaster India Pale Ale.

The kits are pretty easy to get going.   Adding spraymalt instead of sugar and a better strain of yeast produces a very drinkable pale ale last time round so I’m trying that again.  I did get some hops to throw in but I decided to leave them for the next attempt.

I’ll be leaving the beer in the barrel for about a week before transferring to another barrel for another week.  After that it’s the bottling process which can be a bit of a pain in the nads.

This brew should be ready for New Years so here’s hoping for another success!