Collaborative lesson development with GitHub

If you're doing any kind of scientific computing and not using version control, you're doing it wrong. The git version control system and GitHub, a web-based service for hosting and collaborating on git-controlled projects, have both become wildly popular over the last few years. Late last year GitHub announced that the 10-millionth repository had been created, and Wired recently ran an article reporting on how git and GitHub were being used to version control everything from wedding invitations to Gregorian chants to legal documents. Version control and GitHub-enabled collaboration isn't just for software development anymore.

We recently held our second Software Carpentry bootcamp at UVA where I taught the UNIX shell and version control with git. Software Carpentry keeps all its bootcamp lesson material on GitHub, where anyone is free to use these materials and encouraged to contribute back new material. The typical way to contribute to any open-source project being hosted on GitHub is the fork and pull model. That is, if I wanted to contribute to the "bc" repository developed by user "swcarpentry" (swcarpentry/bc), I would first fork the project, which creates a copy for myself that I can work on. I make some changes and additions to my fork, then submit a pull request to the developer of the original "bc" repository, requesting that they review and merge in my changes.

GitHub makes this process extremely simple and effective, and preserves the entire history of changes that were submitted and the conversation that resulted from the pull request. I recently contributed a lesson on visualization with ggplot2 to the Software Carpentry bootcamp material repository. Take a look at this pull request and all the conversation that went with it here:

https://github.com/swcarpentry/bc/pull/395

On March 27 I forked swcarpentry/bc and started making a bunch of changes and additions, creating a new ggplot2 lesson. After submitting the pull request, I instantly received tons of helpful feedback from others reviewing my lesson material. This development-review cycle went back and forth a few times, and finally, when the Software Carpentry team was satisfied with all the changes to the lesson material, those changes were merged into the official bootcamp repository (the rendered lesson can be viewed here).

Git and GitHub are excellent tools for very effectively managing conflict resolution that inevitably results from merging work done asynchronously by both small and very large teams of contributors. As of this writing, the swcarpentry/bc repository has been forked 178 times, with pull requests merged from 71 different contributors, for a total of 1,464 committed changes and counting. Next time you try reconciling "tracked changes" and comments from 71 contributors in a M$ Word or Powerpoint file, please let me know how that goes.

In the meantime, if you're collaboratively developing code, lesson material, chord progressions, song lyrics, or anything else that involves text, consider using something like git and GitHub to make your life a bit easier. There are tons of resources for learning git. I'd start with Software Carpentry's material (or better yet, find an upcoming bootcamp near you). GitHub also offers courses online and in-person training classes, both free for-fee (cheap). You can also learn git right now by trying git commands in the browser at https://try.github.io.

Analysis of #ASHG2013 Tweets

I archived and anlayzed all Tweets with the hashtag #ASHG2013 using my previously mentioned code.

Number of Tweets by date shows Wednesday was the most Tweeted day:

The top used hashtags other than #ASHG2013:

The most prolific users:

And what Twitter analysis would be complete without the widely loved, and more widely hated word cloud:

Edit 8:24am: I have gotten notes that some Tweets were not captured in this archive. This year's ASHG was very actively Tweeted. Unfortunately there are API limits restricting how many results I can return using the t Twitter command line client. A search during a particularly active time of day might have truncated some search results. Please feel free to send me a pull request if you think there's something I can do to improve the automated search code!

Useful Unix/Linux One-Liners for Bioinformatics

Much of the work that bioinformaticians do is munging and wrangling around massive amounts of text. While there are some "standardized" file formats (FASTQ, SAM, VCF, etc.) and some tools for manipulating them (fastx toolkit, samtools, vcftools, etc.), there are still times where knowing a little bit of Unix/Linux is extremely helpful, namely awk, sed, cut, grep, GNU parallel, and others.

This is by no means an exhaustive catalog, but I've put together a short list of examples using various Unix/Linux utilities for text manipulation, from the very basic (e.g., sum a column) to the very advanced (munge a FASTQ file and print the total number of reads, total number unique reads, percentage of unique reads, most abundant sequence, and its frequency). Most of these examples (with the exception of the SeqTK examples) use built-in utilities installed on nearly every Linux system. These examples are a combination of tactics I used everyday and examples culled from other sources listed at the top of the page.

The list is available as a README in this GitHub repo. This list is a start - I would love suggestions for other things to include. To make a suggestion, leave a comment here, or better - open an issue, or even better still - send me a pull request.

Useful one-liners for bioinformatics: https://github.com/stephenturner/oneliners

Alternatively, download a PDF here.