API data formats // mappiness blog

23 May 2011: We’ve launched the data download and API feature, and this post has now been superseded by our data dictionary and API guide.

Frankie Roberto asks:

@mappiness_app hello. Have you thought about publishing an example of the future data export format? (So that we can start making apps now).

A great question. We hadn’t!

Initially we plan to provide data in JSON and CSV formats (with KML and iCalendar to follow), and provisionally this will work as follows. (Note that if you don’t know what JSON stands for, you can probably ignore this post).

Examples

Here are examples of the JSON format and the CSV format data.

Requests and formats

Both formats will be available using a simple (https://) GET request with a secret key embedded in the URL — so the JSON data will be usable directly from a JavaScript web app. For security, this secret key will change if the user disables and then re-enables data download from within the app.

The CSV is UTF-8 encoded with Unix line-endings (\n) and strings unquoted except where empty or in the case of embedded commas/newlines.

The JSON is just a transformation of the CSV: an Array of Objects corresponding to CSV rows, with the CSV headers repeated as the keys of each Object.

The rows of the CSV and Objects of the JSON Array are in descending start time order, so the first row or Object corresponds to the most recent response.

The JSON API may default to returning only the most recent response, but offer a ?limit=N parameter for requesting the others. The JSON API will also offer a ?callback=callbackName parameter for cross-site scripting (JSON-P).

Data dictionary

The CSV headers and JSON Object keys are as follows:

start_time_epoch
start_time

What time did this response begin? This is the iPhone’s belief about UK local time.

The first variable is the number of seconds since 1 January 1970; the second is a string of the form “2011-03-13 12:11:26 +0000”.

noise_50pc
noise_90pc

Respectively the median of several readings of what the iPhone reports as the ‘average’ noise level, and the 90th percentile of what it reports as the maximum level. These are supposed to be decibel values, and are either negative or zero.

happy
relaxed
awake

The three feelings slider responses as decimal values scaled from 0 (not at all) to 1 (extremely).

in_out
home_work

Responses to the Place questions, each taking one of three possible string values. Respectively these are "in" / "out" / "vehicle" and "home" / "work" / "other".

Like subsequent response variables, these could be missing (null in JSON, or empty CSV cells) if the response was interrupted.

took_photo
photo_url

The first is a 0/1 indicator of a photo having been taken. If it’s 1 and the photo was made public, the second is a publicly-accessible URL for the photo (non-public photos are not currently available). If the first is 0 or the photo was not made public, the second will be missing (null or empty).

time_taken

How long it took from beginning to finishing the response, in seconds.

beep_time_epoch
beep_time
beep_sound

What was the server’s idea of UK time when the beep was sent (respectively as seconds since 1970 and as a string), and what was the name of the beep sound sent? If the response was volunteered with no beep having been sent, all three values will be missing (null or empty).

beep_sound will otherwise take one of the following string values:

Beeps
Bird
Bugle
Chimes
Ding (quieter)
Ding
Doorbell
Referee
Whistle

latitude
longitude
accuracy_m
altitude
alt_accuracy_m

GPS location parameters. Missing values or values of -1 indicate some form of error.

with_partner
with_children
with_relatives
with_peers
with_clients
with_friends
with_others

0/1 indicators corresponding to the answers to the People responses. If all are 0, either the user tapped ‘Alone or with strangers only’ or the response was interrupted. Otherwise this JSON mapping may be helpful:

{"with_partner":"Spouse, partner, girl/boyfriend", "with_children":"Children", "with_relatives":"Other family members", "with_peers":"Colleagues, classmates", "with_clients":"Clients, customers", "with_friends":"Friends", "with_others":"Other people you know"}

do_work
do_meeting
do_travel
do_cook
do_chores
do_admin
do_shop
do_wait
do_childcare
do_pet
do_care
do_rest
do_sick
do_pray
do_wash
do_love
do_chat
do_eat
do_caffeine
do_alcohol
do_smoke
do_msg
do_net
do_tv
do_music
do_speech
do_read
do_theatre
do_museum
do_match
do_walk
do_sport
do_gardening
do_birdwatch
do_hunt
do_compgame
do_games
do_bet
do_art
do_sing
do_other
do_other2

0/1 indicators corresponding to the Activities responses. If all are zero, the response was interrupted.

do_other is a ‘Something else’ response submitted by app versions before 1.0.2, while do_other2 is a ‘Something else’ response submitted from version 1.0.2 onwards.

The reason for the distinction is that five new activities were introduced in 1.0.2, and ‘Something else’ thus covers a slightly narrower range of something elses. The new activities were do_love, do_caffeine, do_msg, do_net and do_birdwatch.

The following JSON mapping may be helpful:

{"do_work":"Working, studying", "do_meet":"In a meeting, seminar, class", "do_travel":"Travelling, commuting", "do_cook":"Cooking, preparing food", "do_chores":"Housework, chores, DIY", "do_admin":"Admin, finances, organising", "do_shop":"Shopping, errands", "do_wait":"Waiting, queueing", "do_child":"Childcare, playing with children", "do_pet":"Pet care, playing with pets", "do_care":"Care or help for adults", "do_rest":"Sleeping, resting, relaxing", "do_sick":"Sick in bed", "do_pray":"Meditating, religious activities", "do_wash":"Washing, dressing, grooming", "do_love":"Intimacy, making love", "do_chat":"Talking, chatting, socialising", "do_eat":"Eating, snacking", "do_caffeine":"Drinking tea/coffee", "do_booze":"Drinking alcohol", "do_smoke":"Smoking", "do_msg":"Texting, email, social media", "do_net":"Browsing the Internet", "do_tv":"Watching TV, film", "do_music":"Listening to music", "do_speech":"Listening to speech/podcast", "do_read":"Reading", "do_thatre":"Theatre, dance, concert", "do_museum":"Exhibition, museum, library", "do_match":"Match, sporting event", "do_walk":"Walking, hiking", "do_sport":"Sports, running, exercise", "do_gardening":"Gardening, allotment", "do_birdwatch":"Birdwatching, nature watching", "do_hunt":"Hunting, fishing", "do_compgame":"Computer games, iPhone games", "do_game":"Other games, puzzles", "do_bet":"Gambling, betting", "do_art":"Hobbies, arts, crafts", "do_sing":"Singing, performing", "do_other":"Something else", "do_other2":"Something else"}

notes

App version 1.1, which will enable turning on data download, will also allow optional entry of free text notes with responses. This variable is a string containing those notes (it may be a missing value for responses from previous app versions, and it may be missing or an empty string if no notes were added).

Your thoughts?

We don’t guarantee these details won’t change between now and API launch. Please let us know if you think we should do anything differently.

Frankie Roberto

Thanks for this – really useful and clear.

Couple of questions:

1) Will there be a maximum value for ‘limit=’? If so, how can do pagination to get all the responses?
2) Will it be possible to add a since=datetime param in order to be able to do daily (or so) polling to get new responses?
3) What’s the recommended way for allowing people to give your app permission to get their data? oAuth? copy and paste of the ‘secret’?

Frankie
- George
  
  No problem!
  
  1) No, I don’t think we’ll set a maximum value — let’s say we’ll allow limit=all to retrieve everything. I guess to support paging for those who want it we should probably add offset=N too.
  
  2) That’s a nice idea: let’s say we’ll also provide before=X&after=Y parameters, then. Probably as integers representing epoch seconds. At some point I may think about adding web hooks for this kind of use case too.
  
  3) The app will provide a URL for a page that includes (predictable) links to the various data formats (and possibly also links to good third party apps). It will very likely look something like https://mappiness.me/4/1234/cx3a5i89sw3 — and have /data.json or /data.csv appended for the different data formats.
  
  So it’s probably simplest just to ask users for that base URL. (We’ll discourage users from emailing the URL to themselves, as this is insecure, so they’ll probably need to type the path component manually).
Mike Stenhouse

This looks excellent… What will the URLs look like? Were you planning on offering any filter parameters? “since” would be handy – fetch the items since a given date, and “before” would let me match on a single day or period.
- George
  
  Great. I think the reply to Frankie now covers these points?
Pingback: Frankie Roberto – Introducing Me-o-gram, a visualisation service for Mappiness()