Yesterday I was supposed to be working on a demo for tinfoil, but I got distracted looking at the New York Public Library digital collections API.
I ended up making ...yes, another Twitter bot. It's pretty simple - you tweet a word or phrase at it, and it does its best to send you back a picture from the NYPL collection that matches what you tweeted.
Mostly it uses things I've had experience with before - packages random-js and twit, and the node file system 'fs'. There were a few new things I tried though - saving and retrieving image files using fs, and tweeting images.
There were also a couple of bugs I created by not understanding what 'types' I was using. If you're interested in learning nodejs and JavaScript, this post might help save you a lot of frustration!
The first problem I came up against was that recently heard about the npm package chalk and wanted to play around with it. Chalk allows you to selectively add colours in the console. It's useful if you want to differentiate between different outputs (e.g. you might make error messages red). What I've learned as a result is useful, but it was pretty painful.
When first using an API, I find I always need to look hard at the JSON object it sends back, because the API documentation often makes perfect sense once you understand the data structure, and just as often makes little sense before you understand the data structure! To see what the various parts were, I simply requested some data (i.e. sent a query) from the API, then logged the data to the console:
console.log(data)
This resulted in a huge block of text, which was pretty difficult to read, but it was easy enough to find that the first 'key' was, appropriately enough, nyplAPI. So, knowing my JSON dot notation I confidently changed my instruction to:
console.log(data.nyplAPI)
...and was rewarded with undefined. What the hell?
Sometimes strings look like objects
The problem here is that I was thinking that the data returned by the API was an object. It certainly looked like one. But it was actually a string - part of the reason it was so hard to read, coming through as a big block without spacing. I needed to turn it back into an object to make it usable, using JSON.parse():
var parsed = JSON.parse(body);
Now I could use console.log(parsed.nyplAPI) and get a result:
{ nyplAPI:
{ request:
{ search_text: 'fun',
filter_query: 'use_rtxt_s:("PDCDPP" OR "PDNCN" OR "PDREN" OR "PDEXP" OR "PDADD" OR "PDUSG" OR "PPD" OR "PPD100" OR "CC_0")',
filter_type: 'fuzzy',
perPage: '50',
page: '1',
totalPages: '4',
startTime: 'Beginning of Time',
endTime: 'Till Now' },
response: { headers: [Object], numResults: '152', result: [Object] } } }
Great! Now I can use chalk to make it a pretty colour:
console.log(chalk.yellow(parsed.nyplAPI));
Nope. Instead of the same result with yellow text, the console logged [object Object]. What on earth was going on here? At the time, I simply gave up on using chalk, but it turns out the answer is both simple and complex. I could easily have resolved my problem by turning it back into a string using JSON.stringify() - the opposite of the JSON.parse() manuever we used a moment ago:
console.log(chalk.yellow(JSON.stringify(parsed.nyplAPI)));
So how come console.log(parsed.nyplAPI) works but console.log(chalk.yellow(parsed.nyplAPI)) doesn't?
Well, [object Object] is what you should normally expect to be logged by the console in this situation. It turns out, however, that nodejs has a feature called util-inspect, which is automatically called when you do a console.log() on an object. util-inspect automatically expands the object down to two levels, which we saw above. But it can only expand objects when they are standalone arguments. When we used chalk, we put the object inside a function, so console.log() reverts to normal behaviour and returns the result of the function, which is [object Object]. Clear as mud? Here's a breakdown:
console.log(body); // returns a string that looks like an object
console.log(typeof body); // returns string
console.log(body.nyplAPI) // returns undefined
var parsed = JSON.parse(body);
console.log(typeof parsed); // returns object
console.log(chalk.cyan(parsed)); // returns [object Object]
console.log(parsed); // returns an object tree to 2 levels
var chalkString = chalk.yellow(JSON.stringify(parsed))
console.log(typeof chalkString); // returns string
console.log(chalkString); // returns a string that looks like an object
The moral to the story is that instead of using chalk I should have just used console.dir(). Also, just because something looks like something, doesn't mean it is. It seems I hadn't learned my lesson, because as I prepared to add the finishing touch (adding the Twitter component) to the bot, I was about to make a very similar mistake.
This bit of code uses the twit package (variable name 'T' here) to check the bot's mentions, and then respond appropriately:
function initiate() {
getMentions(lastTweet);
};
function getMentions(id) {
T.get('statuses/mentions_timeline', {since_id: id, include_entities: false}, function(err, data, response) {
if (data.length > 0) {
for (i in data) {
var currentId = data[i].id;
var tweet = data[i].text;
var name = tweet.slice(0,13).toLowerCase();
// only resond to @s, not mentions
if (name === '@picturesofny') {
var query = tweet.slice(14);
var user = data[i].user.screen_name;
// only respond to new stuff
if (currentId > lastTweet) {
getUrl(query, user);
setLast(currentId);
} else {
console.log("no new tweets");
}
}
}
} else {
console.log("no tweets");
}
});
};
It works fine, except that it never updates lastTweet, so every time it loops it replies to every mention again, and again, and again... Can you see why?
Sometimes strings look like numbers
Yes, it's our old friend "looks like something but is actually something else". In this case, I was treating a string as if it was an number. The code tries to evaluate whether currentId (the id of the tweet currently being considered) is larger than lastTweet (the id of the last tweet to be recorded as having been replied to). The problem with this is that it's like evaluating whether "I like doughnuts" is larger than "I like coffee". Both of the things being considered are strings rather than numbers. Luckily, this is pretty easy to fix. We just use parseInt(), which turns strings that look like numbers into actual numbers:
// only respond to new stuff
if (parseInt(currentId) > parseInt(lastTweet)) {
getUrl(query, user);
setLast(currentId);
} else {
console.log("no new tweets");
}
Hooray, it works!
The lesson here is that typeof is a very useful debugging tool, and it's important to know what format your data is in before you try to manipulate it.
You can check out the new bot at @picturesofNY, and the code on GitHub.