Tuesday, July 17, 2012

Plotting the Frequency of Twitter Hashtag Usage Over Time with R and ggplot2

The 20th annual ISMB meeting was held over the last week in Long Beach, CA. It was an incredible meeting with lots of interesting and relevant talks, and lots of folks were tweeting the conference, usually with at least a few people in each concurrent session. I wrote the code below that uses the twitteR package to pull all the tweets about the meeting under the #ISMB hashtag. You can download that raw data here. I then use ggplot2 to plot the frequency of tweets about #ISMB over time in two hour windows for each day of the last week.

The code below also tabulates the total number of tweets by username, and plots the 40 most prolific. Interestingly several of the folks in this list weren't even at the meeting.


I'll update the plots above at the conclusion of the meeting.

Here's the code below. There's a limitation with this code - you can only retrieve a maximum of 1500 tweets per query without authenticating via OAuth before you receive a 403 error. The twitteR package had a good vignette about how to use the ROAuth package to do this, but I was never able to get it to work properly. The version on CRAN (0.9.1) has known issues, but even when rolling back to 0.9.0 or upgrading to 0.9.2 from the author's homepage, I still received the 403 signal. So my hackjob workaround was to write a loop to fetch all the tweets one day at a time and then flatten this into a single list before converting to a data frame. You still run into the limitation of only being able to retrieve the first 1500 for each day, but #ISMB never had more than 1500 any one day. If you can solve my ROAuth problem, please leave a comment or fork the code on GitHub.




5 comments:

  1. Very useful post, thanks. So far I have the exact same issue as you using ROAuth 0.9.2. Also, I think the "until" parameter for searchTwitter() is very confusing. It seems to mean "up to but not including", whereas my brain would interpret it as "up to but not later".

    ReplyDelete
  2. A week ago, I was using the twitteR package as well and the ROAuth problem seems to be rampant.

    I stumbled on the maintainer of the library's post, and it stated that he and Hadley are working to fix it

    ReplyDelete
  3. I'm glad to see I'm not the only one having problems with ROAuth. Glad Jeff and Hadley are working on it.

    ReplyDelete
  4. I'm getting this message:

    - "Rate limited .... blocking for a minute ..."

    Any ideas?

    ReplyDelete
  5. This is working well for me: https://github.com/sferik/t

    ReplyDelete

Creative Commons License
Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.