A Chip of Culture – a Comic Panel Posting Twitter Bot

One of my favourite sorts of twitter accounts like Random Bobbinsverse and Random Hellboy that tweet out single panels for a comic. They add a little bit of joy to my timeline between the tweets about our current Grimdark reality. 

Recently I’ve had a need to look into creating Twitter Chatbots and creating one of these panel posting bots seemed like a good example project.

Why call the bot A Chip of Culture? Alan Moore is said to refer to floppy comic book issues “a real slab of culture” (although I can now only find references to Warren Ellis saying Moore said it). If a full comic is a “slab” then a single panel would be a “chip”.

To get the panel posting bot working you’ll need the following:

Basic Bot

The core of the bot built off the back of the tutorial “How to Make a Twitter Bot in Python With Tweepy“. The tutorial results in a bot that can reply, favourite and follow. Once you’ve had a look at that, come back here and we’ll expand it to have a MySQL backend to hold some data and integrating with Dialog Flow to provide some smarts.

If you want to skip it, you can start with the code tagged “basicbot” in the repo for this bot.

Adding MySQL

Before we can interact with the database, the MySQL connector package needs to be installed:

pip install mysql-connector-python

To keep our boilerplate code for interacting with the database in the one place, we’ll create some helper functions around our calls – connect, fetchall, insert and update. You can see the code behind these in db/db_helper.py.

The database connection details should be set as environment variables for the connect function to use:

export DB_HOST="your_host"
export DB_NAME="your_database"
export DB_USER="someuser"
export DB_PASS="apassword"

The Panels Table

The first thing we want to store in the database is a list of comic panels that we will be posting. This table will contain all the information needed to post about a panel – who the creators are, what series, when it was published etc.. The only required field will be the id. Create the table using the following SQL:

CREATE TABLE `panels` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `image_uri` varchar(1000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `twitter_media_id` varchar(20) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `writer` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `artist` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `letterer` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `colourist` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `inks` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `series` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `issue` int(11) DEFAULT NULL,
  `published` varchar(45) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `last_used` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `number_of_uses` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=22 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
  • id (int): Unique identifier for the row
  • image_uri (varchar): Location in filesystem of the panel image
  • twitter_media_id (varchar): Media id from Twitter after image is uploaded
  • writer (varchar): Who was the writer on the comic
  • artist (varchar): Who was the artist on the comic
  • letterer (varchar): Who was the letterer on the comic
  • colourist (varchar): Who was the colourist on the comic
  • inks (varchar): Who inked the comic
  • series (varchar): What series the panel appeared
  • issue (int): What issue the panel was in
  • published (varchar): When was the issue published
  • last_used (datetime): When the panel was last posted
  • number_of_uses (int): How many times the panel has been posted

We’ll create a class to map a row from the panels table into an object. This class will have a property which will generate the appropriate text describing the panel. As most of the columns are optional, this function will only include the relevant information.

class Panel:
    def __init__(self, details):
        self.id = details['id']
        self.image_uri = details['image_uri']
        self.twitter_media_id = details['twitter_media_id']
        self.writer = details['writer']
        self.artist = details['artist']
        self.letterer = details['letterer']
        self.colourist = details['colourist']
        self.inks = details['inks']
        self.series = details['series']
        self.issue = details['issue']
        self.published = details['published']
        self.last_used = details['last_used']
        self.number_of_uses = details['number_of_uses']

    @property
    def tweet_text(self):
        tweet_text = ""

        if self.letterer:
            tweet_text = f"L: {self.letterer} " + tweet_text

        if self.colourist:
            tweet_text = f"C: {self.colourist} " + tweet_text

        if self.artist:
            tweet_text = f"A: {self.artist} " + tweet_text

        if self.writer:
            tweet_text = f"W: {self.writer} " + tweet_text

        if len(tweet_text) > 0:
            tweet_text = "\n" + tweet_text

        if self.published:
            tweet_text = f"{self.published} " + tweet_text

        if self.issue:
            tweet_text = f"No. {self.issue} " + tweet_text

        if self.series:
            tweet_text = f"{self.series} " + tweet_text

        return tweet_text

When posting, we want to select a panel at random from the least used panels. This means that a panel will not be reused until all other panels have been shown. We’ll add a helper function called select_panel to perform the query, shuffle the results and return a Panel object.

def select_panel():
    logger.info("Selecting a panel")

    panel = None

    sql_select_Query = "SELECT * FROM panels WHERE number_of_uses = (SELECT MIN(number_of_uses) FROM panels)"

    records = fetchall(sql_select_Query)

    logger.info(f"Number of records available: {len(records)}")

    if len(records):
        shuffle(records)

        panel = Panel(records[0])

    return panel

When a panel is posted, we want to store the media id of the panel image (so we don’t have to keep uploading the same image multiple times) and increment the use count. To do this, we’ll add a function called update to the panel class.

    def update(self):
        logger.info("Updating a panel to mark used")

        self.number_of_uses = self.number_of_uses + 1

        sql_update_query = """Update panels set number_of_uses = %s, twitter_media_id = %s where id = %s"""
        inputData = (self.number_of_uses, self.twitter_media_id, self.id)
        update(sql_update_query, inputData)

        logger.info(f"Updated panel {self.id}")

Now that we can fetch a random panel and update that it has been used, let’s put it to use with a new script, autopostpanel.py.

#!/usr/bin/env python
# tweepy-bots/bots/autopostpanel.py

import tweepy
import logging
import time
import sys
import os
#This is so we can find out DB files
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/../')

from config import create_api
from db.panel import select_panel

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

def post_panel(api):
    logger.info("Attempt to post a panel")

    panel = select_panel()
    media_ids = None

    if panel:
        tweet_text = panel.tweet_text

        if not panel.twitter_media_id:
            media = api.media_upload(panel.image_uri)
            panel.twitter_media_id = media.media_id

        media_ids=[panel.twitter_media_id]
    else:
        tweet_text = "No panel available!"

    logger.info(f"Tweeting: {tweet_text}")

    tweet = api.update_status(
        status=tweet_text,
        media_ids=media_ids
    )

    panel.update()

    return tweet

def main():
    try:
        api = create_api()
        post_panel(api)
    except Exception as e:
        logger.error("Error on post", exc_info=True)

if __name__ == "__main__":
    main()

When this script is run, it fetches a random panel from the least used panels, uploads the panel (if it hasn’t been uploaded before), tweets it out and records the usage.

The updated code for this section is under the “postpanel” tag in the repo.

Recording Replies

In the current auto-reply script, it will forget what tweets have been replied to when it stops executing and will attempt to reply again on subsequent runs. This is less than ideal as it will only annoy people seeing the same thing over and over. To avoid this, we will record when a tweet has been replied to in the database.

Create the table to hold the tweet id of replied tweets with the following SQL:

CREATE TABLE `replies` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `tweet_id` varchar(20) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

To interact with this table, we will create two helper functions, last_reply to get the id of the last tweet that was replied to and did_reply to record that a tweet was replied to.

def have_replied(tweet_id):
    logger.info(f"Checking if tweet {tweet_id} has already been replied to")

    sql_select_Query = "SELECT * FROM replies WHERE tweet_id = %s LIMIT 1"

    records = fetchall(sql_select_Query, (tweet_id,))

    if records and len(records) > 0:
        return True
    return False

def did_reply(tweet_id):
    logger.info("Adding a tweet id to mark it replied")

    sql_insert_Query = "INSERT IGNORE INTO replies (tweet_id) VALUES (%s)"
    insert(sql_insert_Query, (tweet_id,))

def last_reply():
    logger.info("Get the id of the last tweet replied to")

    sql_select_Query = "SELECT * FROM ellis_panel.replies ORDER BY id DESC LIMIT 1"
    records = fetchall(sql_select_Query)

    logger.info(f"Number of records available: {len(records)}")

    if len(records) < 1:
        return 1
    return int(records[0]['tweet_id'])

The autoreply.py script needs to be updated to use these functions (see lines 37 and 43 below). For did_reply we pass through the id_str parameter of the tweet object rather than the 64-bit integer id.

#!/usr/bin/env python
# tweepy-bots/bots/autoreply.py

import tweepy
import logging
from config import create_api
import time
import sys
import os
#This is so we can find out DB files
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/../')

from db.replies import last_reply, did_reply

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

def check_mentions(api, keywords, since_id):
    logger.info("Retrieving mentions")
    new_since_id = since_id
    for tweet in tweepy.Cursor(api.mentions_timeline,
        since_id=since_id).items():
        new_since_id = max(tweet.id, new_since_id)
        if tweet.in_reply_to_status_id is not None:
            continue
        if any(keyword in tweet.text.lower() for keyword in keywords):
            logger.info(f"Answering to {tweet.user.name}")

            if not tweet.user.following:
                tweet.user.follow()

            api.update_status(
                status="Please reach us via DM",
                in_reply_to_status_id=tweet.id,
            )

            did_reply(tweet.id_str)

    return new_since_id

def main():
    api = create_api()
    since_id = last_reply()
    while True:
        since_id = check_mentions(api, ["help", "support"], since_id)
        logger.info("Waiting...")
        time.sleep(60)

if __name__ == "__main__":
    main()

When the script is run, it will ignore tweets that have already been replied to and avoid duplicates statuses (that would be rejected by the Twitter API anyway) and won’t annoy other users by bombarding them with the same response.

The updated code for this section is under the “recordreply” tag in the repo.

Integrating with Dialogflow

Now that we know when we’ve responded to a mention, let’s make our responses a little more intelligent by passing them through Dialogflow.

If you haven’t already, sign up to Dialogflow and login to the console.

An agent needs to be created to mange the interactions. Hit ‘Create Agent’. Call the agent ‘ChipofCulture’ (no white space is allowed in the name) and hit ‘Create’. All the other options are configurable later in the settings page.

We now have an agent but it will only respond to the default questions. To correctly respond to mentions, we need to set up intents that will interpret what has been said and respond appropriately. The intent we will set up is to figure out when the bot has been asked to send a panel.

Click ‘Create Intent’. Call the intent ‘Random Panel’. Before the intent can be triggered, some example phrases need to be added. Based on these training phrases the agent will try and match was is sent to it and return the appropriate intent. Click ‘Add Training Phrase’. Enter ‘Show me a comic’ and ‘Send me a panel’. Hit ‘Save’.

You can test the intent by typing into the input box on the top right. If you enter ‘show me a panel’ (a combination of our two phrases) you will see that the correct intent is picked up.

The last piece is to add an action to the intent. this action will be used later in our scripts. Click ‘Add Parameters and Action’. Put ‘send_panel’ in the ‘Enter action name’ input box. Hit ‘Save’. Test the intent again and you’ll see the action is filled out.

If you hit the ‘Diagnostic Info’ button after testing your intent, you will see the raw JSON that will be retuned by the API.

We’re now ready to access the agent from our scripts. To open up API access, it must be turned on via the Google Cloud console. Once you are logged into the console, select APIs & Services. Click on ‘Enable APIs and Services’ and search for Dialog Flow and activate the API.

To access the API, you’ll need to create a service account to authenticate against. Download the json file and open it up. Find the key ‘client_email’ and note the generated address. Go to the Google Cloud console and click on ‘IAM’. If the e-mail address from your json file isn’t listed, click ‘add’. Enter the e-mail address in ‘New Member’ and for ‘select a role’, search for ‘Dialogflow API Client’ and click ‘Save’.

Once all this is setup, we need to install the Dialogflow package and other supporting Google packages for Python:

 pip install dialogflow
pip install google-auth 
pip install google-cloud-storage 

We now need to add a function that will take the text from a mention, pass it to Dialogflow and figure out what to do with the response.

#!/usr/bin/env python
# tweepy-bots/df/df_helper.py

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

def detect_intent_texts(project_id, session_id, text, language_code):
    """Returns the result of detect intent with texts as inputs.

    Using the same `session_id` between requests allows continuation
    of the conversation."""
    import dialogflow_v2 as dialogflow
    from google.protobuf.json_format import MessageToDict

    session_client = dialogflow.SessionsClient()

    session = session_client.session_path(project_id, session_id)

    text_input = dialogflow.types.TextInput(text=text,
        language_code=language_code)

    query_input = dialogflow.types.QueryInput(text=text_input)

    response = session_client.detect_intent(session=session,
        query_input=query_input)

    return MessageToDict(response.query_result)

The response from the API is turned from a protocol buffer object into JSON before being returned.

To use the function, we call it with our agent’s project id, a session key (we’re using a freshly generated UUID), the text of the tweet and a language encoding which we’ll take from the language set in the tweet.

You find the project id on the settings page accessed by clicking on the cog icon and will be stored as an environment variable:

export DIALOGFLOW_PROJECT_ID="your-project-id"

The UUID is if we want to expand out capabilities and be able to keep a session going with the agent. Using this is left as an exercise for the reader.

The autoreply.py script is updated with this function call. The result returned is checked for the send_panel action

def check_mentions(api, since_id):
    logger.info("Retrieving mentions")
    new_since_id = since_id
    for tweet in tweepy.Cursor(api.mentions_timeline,
        since_id=since_id).items():
        new_since_id = max(tweet.id, new_since_id)
        if tweet.in_reply_to_status_id is not None:
            continue

        result = detect_intent_texts(dialogflow_project_key, str(uuid.uuid4()), tweet.text, tweet.lang)

        if result['action'] == 'send_panel':
            logger.info(f"Answering to {tweet.user.name}")

            if not tweet.user.following:
                tweet.user.follow()

            post_panel(api, username=tweet.user.screen_name, reply_tweet_id=tweet.id)

            did_reply(tweet.id_str)

    return new_since_id

Based then on the action set in the response, we can do the appropriate thing, which here is to send a random comic panel.

The post_panel function in autopostpanel.py has changed slightly, to accept an optional Twitter handle and tweet id to use when posting a tweet. This allows for a threaded reply rather than just a random dangling tweet.

def post_panel(api, username=None, reply_tweet_id=None):
    logger.info("Attempt to post a panel")

    panel = select_panel()
    media_ids = None

    if panel:
        tweet_text = panel.tweet_text

        if not panel.twitter_media_id:
            media = api.media_upload(panel.image_uri)
            panel.twitter_media_id = media.media_id

        media_ids=[panel.twitter_media_id]
    else:
        tweet_text = "No panel available!"

    if username:
        tweet_text = f"@{username} " + tweet_text

    logger.info(f"Tweeting: {tweet_text}")

    tweet = api.update_status(
        status=tweet_text,
        media_ids=media_ids,
        in_reply_to_status_id=reply_tweet_id,
    )

    panel.update()

    return tweet

Now when there is a tweet in the bot’s mentions that asks for a panel, the bot will reply directly to that tweet.

The updated code for this section is under the “adddialogflow” tag in the repo.

Conclusion

In this tutorial, we’ve expanded on a basic Twitter bot to add some state so it remembers what has been replied and given it some smarts to reply sensibly.

More intents can be easily added to the Dialogflow agent so you can respond to other questions such as asking the bot what artists does it know about or to send a panel from a particular series.

The code is available on Github in the repo achipofculture and active on the EllisPanelBot Twitter account.

An example of the bot tweeting out a panel.

Your Deposit Calculator Is Ready

This morning, it was announced that buy to lets and owner-occupiers would have to have 30% and 20% deposit respectively and first time buyers would only need 10% for the first €220,000.

Given my penchant for calculators and the fact that I’ve a Property Price Register site, the above tweet came as no surprise and I really should have knocked something together myself as soon as I saw the news.

So without further ado, here’s a quick and dirty calculator to work out how much of a deposit you need under the new rules:

House price: €

Are you a first time buyer
Are you a buy-to-let buyer
Are you a owner-occupier

Luckily, this won’t effect me for another few years as I managed to finally close on my apartment last week but there are plenty of folks for which the possibly of a 20% deposit was a nightmare scenario.

CBPWordPress – Display content from your WordPress blog in an iOS app

CBPWordPress is an iOS library that will allow you to easily include content from a WordPress blog. The library can fetch lists of posts, individual posts and submit comments.

It is, of course, available on GitHub under the MIT License as well as via CocoaPods.

Background

Over the last few years, I’ve built an maintained the accompanying iOS app for Broadsheet.ie. It’s nearly tradition now for me to release a new version every year

The Example App

Included in the repo is an example app that is the basis for the new Broadsheet.ie app. This app will allow you to browse the site content, search for posts and submit comments.

The app also refreshes itself in the background, reminds the user of its existence in the morning and evenings and updates the logo on the home screen every 6 hours.

The full app includes Google Analytics and Mobile Ad SDK, Crashlytics and Conveser.io and has its own fork.

Installing the Plugin

Before you can use CBPWordPress, you must install the WP-JSON-API plugin to provide the data to the app. This is a slightly modified fork that provides a few extra fields to make things a bit smoother for the app.

If you want to be able to submit comments via the app, remember to turn on the Respond controller in the JSON API settings section.

Getting Started With The Example

Check out the repo from GitHub and initialise the Example pods:

git clone https://github.com/kmonaghan/CBPWordPress.git
cd CBPWordPress/Example
pod install

If you open the example project workspace and build and run the app, it should pull the latest 10 posts from Broadsheet.ie.

Using CBPWordPress In Your Own App

CocoaPods

The easiest way to use CBPWordPress is to install it via CocoaPods. To do that, just add the following line to your Podfile and then run ‘pod update’.

pod 'CBPWordPress'

Add the CBPWordPress project to your project

If you don’t use CocoaPods, add all the files in the CBPWordPress folder to your project. Then simply include the CBPWordPress.h header file where you want to use the library.

Note that you will also have to include AFNetworking in your project.

Usage

Pointing the library at your own WordPress installation is trivial. The first call to the library should be to set the root URL for the API.

[CBPWordPressAPIClient rootURI:@"http://YOUR-API-URL];

Once that is set, the calls to the API will use that URL.

Fetching A List Of Posts

To get a list of posts, you use the fetchPostsWithParams:withBlock: method from the NSURLSessionDataTask category.

In the example below, the first page of the recent posts is retrieved and the posts assigned to an array.

__weak typeof(self) weakSelf = self;

[NSURLSessionDataTask fetchPostsWithParams:@{@"page": @(1)}
withBlock:^(CBPWordPressPostsContainer *data, NSError *error) {
if (error) {
//Handle Error
return;
}

__strong typeof(weakSelf) strongSelf = weakSelf;

strongSelf.posts = data.posts;
}];

The allowed parameters are:

  • page: The page you want to fetch. Starts at 1.
  • count: The number of posts to retrieve. Defaults to 10.

Fetching A Post

If you know the post id, you can fetch a post using the fetchPostWithId:withBlock: method from the NSURLSessionDataTask category. The example below fetches post 1234 and assigns it to a local post variable.


__weak typeof(self) weakSelf = self;

[NSURLSessionDataTask fetchPostWithId:1234
withBlock:^(CBPWordPressPost *post, NSError *error){
if (error) {
//Handle Error
return;
}

__strong typeof(weakSelf) strongSelf = weakSelf;

strongSelf.post = post;
}];

If you have the URL of the post, you can use the fetchPostWithURL:withBlock: method instead. You pass the full URL of the post as the parameter.

Comment On A Post

To comment on a post, the postComment:withBlock: method from the NSURLSessionDataTask category is used. The method takes a CBPWordPressComment object as the first parameter. Below is an example comment being initialised.

CBPWordPressComment *newComment = [CBPWordPressComment new];
newComment.postId = 1234;
newComment.email = @"example@example.com";
newComment.name = @"Jonny Appleseed";
newComment.content = @"This is a comment!";
//Optional
newComment.url = @"http://somewebsite.com";
//If the comment is replying to another comment
newComment.parent = 1234;

Note that the URL and parent properties are optional but everything else is required. The parent property should be only be set if the user is replying to a comment and should be that comment’s id.

Once the comment is initialised, pass it to the postComment:withBlock: method. In the following example, the new comment is submitted and on success is set to the returned comment object.

__weak typeof(self) weakSelf = self;

[NSURLSessionDataTask postComment:newComment
withBlock:^(CBPWordPressComment *comment, NSError *error){
__strong typeof(weakSelf) strongSelf = weakSelf;

if (error) {
//Handle error
return;
}

strongSelf.comment = newComment;

Known issue: if WordPress detects a duplicate comment, the resulting return is HTML rather than JSON.

To Do

The library is very much a work in progress. Some of the planned functionality to add is:

  • Add option to fetch data from WP-API plugin
  • Implement helper methods for each WP-JSON-API endpoint (get_category_posts, get_tag_posts etc.)
  • Add a Today Extension to the example app

Contributing

Contributions via pull requests and suggestions are also welcome (although no promises that anything will be added).

If you do use this for your own app, I’d love to hear from you.

Broadsheet: Year 4

Monday was the 4th birthday of Broadsheet and as is tradition, I bring you some stats from the last year. As usual, these all come from Google Analytics and cover from 29th of July 2013 to 28th of July 2014.

The Headline Figures

  • New users up 3% to 24% of visitors
  • Users up 6.5% to 3.2m from 3m
  • Pagviews down 14.5% from 32.5m to 27.8m
  • Screenviews stand at 20.5m

So why the drop in pageviews?

It looks like the mobile apps are cannibalising the desktop site.

This is in part due to the an Android app finally getting released, but also because of the ongoing trend across the web of the growth in traffic from mobile apps.

I don’t have a full year of stats for the 2012/2013 to compare mobile usage with but with a bit of hand wavying estimations based on the 5 months of data I do have, there’s about a 25% increase in screenviews. If I combined the 20.5m screenviews (which are setup to be the equivalent of a pageview on the desktop site) between the iOS and Android apps to the desktop, you get about 12% growth year on year.

Outside of the apps while the traffic from mobile has increased by 5% to 16% of the total pageviews to 3.3m. Tablet usage just increased 1% to 793K.

iOS powered devices still deliver 2 pageviews to every 1 from an Android device.

Samsung dominates the top 50 devices used with 23 models (although some are variants on the same handset). Sony has 6, HTC weighs in with 5, Apple and Nokia have 4 each and Google just 2.

What Are You Reading?

As has always been the case, about half of all pageviews are the front page of the site. The top posts more represent what people have commented on or shared, since the stories are published in full when you’re looking at the home page.

11,464 post were published over the course of the last year. The top five by pageviews were:

Interestingly, three of the stories (‘The Mask Is Off And People Know’, Who Is He? and Dear RTÉ) were only published in the last week, while What Your Electric Guitar Says About You is nearly two years old. These are both perfect examples of the two ways posts become huge. The recent stories are (obviously) very current and immediate. People are sharing and commenting one them as they’re igniting people’s passions. The guitar post, on the other hand, is one that has enduring interest and every so often gets spread around forums and gets shared on Facebook again and again.

The fact that four of the top five posts are rather serious content (a trend that extends to the top 25 posts of the year) is a possible indication of a more serious tone on the site or that people are more inclined to share the more serious material.

What Are You Riled Up About?

There’s been 192,170 comments from 10,117 commenters. Of these, 21 commenters have posted over 1,000 comments each and responsible for 45,688 between them.

The five most commented posts were:

Where Did You Hear About us?

62% of the referral traffic comes from Facebook alone. They have firmly placed themselves as the distribution channel of choice for many people. Each change they make to how posts appear in news feeds has a massive and very visible impact. The last few changes have been massively detrimental to site like Broadsheet that can’t afford to pay Facebook to reach people who’ve already ‘liked’ the Broadsheet feed.

Twitter takes up 23% of the remaining referral traffic, which is less than the traffic coming from just the Facebook Mobile site.

Everything after the big two provide only tiny scraps of traffic.

The top five search terms (with the usual variations on broadsheet.ie removed) that brought people to the site were:

  • teletubbies
  • justin timberlake phoenix park
  • vikings
  • forbiddem fruit 2012 pictures
  • judge nolan

I’m at a loss to explain the teletubbies one.  After three years appearing in the top five, ‘niamh horan’ has slipped down to sixth. I almost feel like it’s an end of an era.

The Window You Look At The Internet Through

Chrome is the absolute king, with 44.5% (up 6%) of all pages viewed on it. this gain was mainly at the expense of both Internet Explorer (dropped 3% to 14.5%) and Firefox (down 4% to 17%). Safari stayed relatively stable at 14% (a 0.5% drop).

The other browser that gained share was the Safari in-app browser (i.e. if you view Broadsheet from with an app like Facebook or Tweetbot etc.) which is up 2% to 5%.

IE 6 and 7 are near extinction, with less than 1/2% of pagesviews from them. IE 8 though is still the most used of those browsers, although it is declining in favour of IE 10 and 11.

When it comes to whats powering the machines people are using, Windows still holds 50% (albeit that’s down 9%). iOS has overtaken OS X for second place with 21% (up 9%) compared to 15.5% (down 5%) followed by Android which has doubled to 10%. The rest is made up of a smattering of Linux, Chrome OS, Symbian Os and Windows Phone.

Mobile Apps

  • 5,325 (for a total of 29,901) on iOS
  • 4,555 (for a total of 6,819) on Android
  • 1,829 (for a total of 3,200) on Windows Phone

OS wise, only about 6% of installs are on a version of Android under 4.0, which is great news for me as I can aim the app to be 4.0+ plus in the next iteration. The downloads for it are a little disappointing but on the flip side, the app isn’t great and does next a bit of love and attention.

On iOS, less than 4% of active users in the last month had some version of iOS 6 on their device. Again, this is great new for me as it means I can drop support for 6 with minimal impact.

Where Are You Based These Days?

Unsurprisingly, there’s no change here. Visitors predominately come from Ireland (74%) followed by the UK (10%), US (4%), Germany (1%), Australia (1%) and then everyone else makes up the other 10%.

Anything Else?

If there’s anything else you’d like to know about, ask in the comments and I’ll see what I can dig out.

Previously:

Broadsheet Yearly Stats 3: We’re getting a bit old in the tooth now.
A Broadsheet New Year
Broadsheet – Entering the Terrible Twos
A year in the Broadsheet

The Three Tun Tavern

There’s been a lot of interest in the office to see what the inside of the The Three Tun Tavern was actually like since they started renovating it a few months ago. Since today was the official opening, 10 of us headed down for lunch out of sheer nosiness.

Inside it seemed brighter and airier than the previous incarnation, Tonic. The bar has been moved from the center of the building to along the back wall, opening it up a lot more space. Some of the internal walls have been removed as well so there’s more overlooking the central area. The over all impression though wasn’t of a brand new pub, rather one relatively recently renovated but lived in.

There were an awful lot of staff on but it’s not clear if that’s just because it was opening day or will be the norm.  They were certainly needed today as there were a constant stream of customers coming in, as well as a few curious people just having a nose around.

IMG_2790
Chicken & Ribs Combo

IMG_2792
Sirloin of Rump Steak

Food wise, the menu is rather extensive albeit standard pub grub fare. They’ve various food ‘clubs’ on during the week with themed specials.

Being lazy and since it was their Tuesday Steak Club, I (and most of the rest of the guys) ended up having a rump steak and chips for €9.95 (it’s normally €12.95). A few more chips wouldn’t have gone astray and the steak was a tiny bit overcooked for rare but the meat was flavourful. Considering the price (which included a soft drink) it was good value.

It was opening day though, so there was the inevitable hiccup with one of the meals which left one of the guys waiting 30 minutes for his veggie burger. It’s probably a little unfair to judge them on this though since it was opening day.

Overall the consensus was that the food was fine and filling but nothing spectacular. It’s another lunchtime option for Blackrock, but you wouldn’t be going out of your way to eat there.

IMG_2789

I will be back at least once more though as they’ve a good selection of craft beers on tap that I’d like to try (and retry Hobgoblin to see if my college days impression of it being vile is true).

Okay so it’s a Wetherspoon’s pub but to be honest, it’s not particularly different to many pubs already operating here. It’s probably going to do a decent lunchtime trade from the off but I wouldn’t like to make predictions about the evening.

Creating an iOS Framework and Today Extension

One of the exciting new things announced at the WWDC keynote this month was App Extensions. This really opens up inter-app communication as well as letting developers do interesting things with the notification center and sharing.

I’m convinced that pretty much every news organisation is going have a widget included in their app to show the latest couple of stories. To that end, I’m going show here how I added a Today widget to a simplified version of the Broadsheet app.

All the code talked about here is available on GitHub under the MIT license.

Caveats:

  • This all works in the simulator but has not been tested on a device yet  Kindly tested by Liam Dunne and it works (phew!)
  • Some of the cell layouts are a bit off but are good enough for informational purposes
  • This is still all in Obj-C rather than being all cool and done in Swift

Starting off

Our starting point is a simple app that grabs the JSON feed of the latest 10 posts Broadsheet.ie and displays them in a UITableView. Tapping on a cell brings you to a screen that displays the content of the post in a UITextView (this will sometimes be mangled due to the content).

To this app we’re going to add a Today widget that displays the latest two post and brings you back to the app when a cell is tapped.

The base app is in the (imaginatively called) ‘baseapp‘ branch.

Creating The Framework

First, we’re going to create a framework to hold the common files that will be used by both the app and the widget. We need to add a framework target to the project by doing the following:

  • Select the project
  • Editor->Add Target
  • Select ‘Framework & Library’
  • Select ‘Cocoa Touch Framework’ and hit ‘Next’
  • Give it a name (ours is called CBPKit) and hit ‘Finish’

The framework should be automatically included in your app target (check under ‘Link Binary With Libraries’ in ‘Build Phases’).

Move any files you want your framework to provide from your app to the framework (note that this doesn’t move the files on disk). In this case it is everything but the view controllers.

Select the headers and in the ‘Utilities’ pane select your framework and set the option on the right to ‘project’.

Select the implementation files and again in the ‘Utilities’ pane select your framework and unselect your project.

For displaying the images in the cells, I’m using SDWebImage. I could have just add the relevant SDWebImage files to the framework I just created, but this seems wrong to me as:

  • You may include another framework in the future that includes and exposes SDWebImage and then you’d have a clash
  • You release your framework and someone can’t use it because your included version of SDWebImage clashes with the version they want to use
  • SDWebImage may release its own framework in the future

Instead, I created a separate framework (with the exact same steps as above) called SDWebImageKit.

As I may want to use this framework in another project, I set the headers to ‘Public’ in the ‘Utilities’ pane and added each of the headers to the framework header file. This means that I can import all the headers at once using:

#import <SDWebImageKit/SDWebImageKit.h>

When you run the app now, it should look no different to before despite the internal changes.

If you switch to the ‘framework‘ branch, you’ll see this built on top of the previous branch.

Today Extension

Like the framework, we first add a Extension target to the project.

  • Select the project
  • Editor->Add Target
  • Select ‘Application Extension’
  • Select ‘Today Extension’ and hit ‘Next’
  • Put in a name (ours is called CBPTodayExtensionExample) and hit ‘Next’
  • Go into Build Phases and include both the frameworks from above in the ‘Link Binary With Libraries’

If you run the project now in the simulator and pull down the notifications drawer, you should see ‘1 New Widget Available’ at the very bottom. Tapping on ‘Edit’ will bring up all the available widgets and the new widget should be at the bottom with a green plus beside it. Tap the plus and it should be displayed with any other active widgets when you hit ‘Done’.

‘Hello world’ is a nice greeting but we want to replace this with a table containing 2 tappable cells containing the latest two posts from Broadsheet.

I don’t use storyboards, so I removed the default one added by the template. In info.plist I removed the key NSExtensionMainStoryboard and replaced it with NSExtensionPrincipalClass and put ‘TodayViewController’ as the value.

The cell I use in the app is a bit big for displaying in the notification center, so I added a set of new constraints to display a more compact cell. This means I can reuse the same data source and cell from the main app in my Today extension with just some minor modifications.

In the TodayViewController, there are two places that need to load data from the network – when the widget is created and when widgetPerformUpdateWithCompletionHandler is called. For the former, I load posts in viewDidLoad, so that they should be ready by the time the widget displays. When iOS thinks the widget will be displayed to the user after it has been first displayed, widgetPerformUpdateWithCompletionHandler is called giving the widget a chance to update the posts displayed.

In this example, both cells are always reloaded even if they already contain the posts returned. It probably shouldn’t do this and instead only replace the updated cells, but that’s a bit of extra polish I’ll leave for when building the full widget.

To get the app to open when a cell is tapped, a custom URL scheme is registered in the base app and this can be called using the UIViewController’s extensionContext to open the URL.

If you switch to the ‘todayextension‘ branch, you’ll see this built on top of the framework branch.

Finally

To keep everything tidy (and to satisfy my developer OCD), I used Synx to synchronise the files and folders to match those used in the project.

That’s all that’s needed to get a Today widget up and running. I’m delighted it’s so straight forward and I’m looking forward to seeing what people do with them.

Aside

There’s an issue with the debugger in Xcode attaching to the extension process properly. You can manually attach it by selecting ‘Debug’->’Attach to Process’ and finding the extension process under ‘System’. If your extension crashes on startup you won’t be able to this.

If you want to see logging from before you reattached the debugger, you can view it in the system log. You can open it from ‘Debug’->’Open System Log…’. If you filter it by your extension’s bundle id you’ll get the relevant entries.

Update 18/6/2014: In the release notes for Xcode 6 Beta 2 there’s a list of issues around the debugger and extensions. One pertinent to this app is that you can’t debug a today widget in the simulator but you can on device.

Update 7/7/2014: Xcode 6 Beta 3 came out today and fixes some of the debugging issues.

Recommended Reading and Viewing

AWS SNS and “File type is not supported or file has been corrupted”

I spent a rather frustrating few hours this morning trying to get push notifications set up on Amazon’s Simple Notification Service.

In the Getting Started with APNS, there is a section on converting the p12 formatted file exported from the key chain and converting it into pem format.

If you try and use this pem file in the management console, the following error is returned:

File type is not supported or file has been corrupted

What you actually need to upload is the p12 version.

Another post in the Wisdom of the Ancients series.

An Experiment In iOS App Pricing

I’ve a couple of paid apps and I am never really sure what to price them. On the one hand it’d be nice if I made anything from them but I also want people to actually use them. Last year, Michael Jurewitz wrote a series of fascinating posts on App Store Pricing. In particular the section on price elasticity stuck in my head.

My tax calculator app (unimaginatively called) TaxCalc.ie generally does best at two times of the year – in and around the budget announcements and in January (presumably as people are checking their first payslip).

On the run up to the 2014 Budget, I did a bigger than normal update to the app both modernising its internals and sprucing up the UI a bit. Based on the extra effort that was put in and keeping price elasticity in mind, I decided to increase the price on the 1st of October from 89c to €1.79 as a bit of an anti-sale.

For the period between the 1st of October 2013 and the 6th of January 2014 (€348.80 profit), I sold 320 copies compared to 315 the previous year (€151.20 profit). So not only did I sell more, I also more than doubled my profit.

I hadn’t really noticed this until I was doing my usual end of year comparisons at the start of January. As people seemed to be as willing to pay €1.79 as they were 89c, I decided to ratchet up the price to the next tier (€2.69) for the rest of January. The result was 72 sales compared to 101 in the same period of 2013. It may have been a 30% drop in downloads, but when you look at the proceeds, the app still generated more than twice amount at €118.08 as opposed to €54.54. I’d call that a successful experiment.

Will I increase the price again? Possibly for a week later in the year, but probably not. My gut feeling is that another price increase would would further drop download numbers to the point where it would make less money. More likely, I’ll either release the 2015 calculator as a standalone app or offer an update as an in-app purchase.

The tax calculator is a very niche app and and it’s really only providing me with beer money. Still, it was well worth tweaking the price to squeeze that little bit extra out of it to make the time spent developing it actually worth something.

tl;dr Increased the price of TaxCalc.ie from 89c to €1.79 and finally to €2.69 resulting in 2x profits while more or less maintaining the number downloads.

Broadsheet Yearly Stats 3: We’re getting a bit old in the tooth now.

So here we are again, another year, another gathering of stats from Broadsheet.ie. If I’m lucky, it’ll be in the Sunday Times again.

The Headline Figures

  • 4.9% increase in visits to 13.8 million
  • 5% drop in unique visitors to 2.9 million
  • 6.5% increase in pageviews to 30.1 million

So why the drop in unique users? We didn’t have another smash hit like the Meanwhile, At Smithfield Horse Fair. It provided a huge surge in once off visitors that we didn’t manage to replicate this year.

We’ve also not the had the massive growth of the previous couple of years, but that is somewhat expected. With the tiny team we have and a zero marketing budget, we’ve done extraordinarily well. Now we need to expand out from the core audience we’ve built.

Where Are You Based These Days?

There’s been no big change in where out visitors are from – Irish visitors account for 74% (up 2%), UK at 10% (no change), US 4.8% (down 0.3%), Germany 1.2% (up 0.13%) and Australia 1.12% (down 0.07%).

What Are You Reading?

Unlike previous years, the laughs are low on the top three stories so I’ll proceed without comment on those.

Instead I’m mention two of the more enduring pieces we ran from 2012 – Daisy: The Cutest Kitten In The World and the already mentioned Meanwhile, At Smithfield Horse Fair. Both of these pull in more once off visitors than most other posts from 2013.

How They’re Finding Us

Apart from the usual crowd that put in some form of ‘Broadsheet’, the top six search terms of 2013 were:

  • electric picnic 2013
  • property tax calculator
  • mikey clancy
  • tayto chocolate
  • niamh horan
  • bus porn

I’ve gone with six entries here rather than my usual five as, well, how could I resist exposing a term like “bus porn”? Niamh Horan continues to be popular for whatever reason.

The Window You Look At The Internet Through

Chrome continues to be the dominant browser choice with 39% (up 6%) of users viewing the site with it. Firefox stays in second place with 17.6% (down 4%), Safari takes third with 15.9% (down 0.1%) taking Internet Explorer’s place which now has 14.9% (down 2.9%).

iOS still accounts for 2/3rds of the mobile browser traffic to the site and Android taking most of the rest. The big winner though is Windows Phone from which pageviews exploded by 648% (for a total of 2.4% of the mobile traffic).

Apps

This year saw the release of an updated iOS app as well as new Android and Windows Phone apps.

There were 8,522 downloads on iOS (for about 30K total), 4,654 on Android and 2,310 on Windows Phone. From that, there’s between 2,500 and 3,000 active users a day producing 2.5 million sessions and 15 million screen views between them. It very much seems like people dip in and out of the apps a few times a day.

Anything Else?

If there’s anything else you’d like to know about, ask in the comments and I’ll see what I can dig out.

Previously:

A Broadsheet New Year
Broadsheet – Entering the Terrible Twos
A year in the Broadsheet

6 Tools In 5 Minutes, Or: How Not To Be A Caveman Developer


These are the slides from my talk at the XCake Xmas session.

Accessorizer:
http://www.kevincallahan.org/software/whatsNew.html
Download from the Mac App Store.

Objectify:
http://tigerbears.com/objectify/
Download from the Mac App Store.

Objective-Clean:
http://objclean.com/
Download from the Mac App Store.

Code Runner:
Download from the Mac App Store.

Reveal:
http://revealapp.com/

Tokens for Mac:
http://usetokens.com/