Steel-Toed Birds
Apr 3, 2011
The revolutions of 2011 in the Arab world have been televised all over the planet. Here in Philadelphia, I could sit on my sofa and watch in real time as hundreds of thousands of people assembled in Tahrir Square in Cairo, people who have been shut out of political life for generations, even centuries, yet who somehow knew just where to go to alert the world to their cause, and when to go there, and what to say and how to say it
The choreography behind those demonstrations, it soon became evident, was digital: tweets and texts and emails and Facebook updates. Is new media creating a new world order? Researchers struggled to collect the tweets and analyze them semantically, in hopes of more closely apprehending the social and intellectual underpinnings of the Arab revolutions. All that was missing was an app.
Tweets, it turns out, aren't all that easy to assemble for analysis. By design, they aren't available after seven days. To do the job properly requires software optimized for twitter mining; it has to scan the twitter universe constantly in search of tweets sent to or from certain users and/or containing certain hashtags or vocabulary; the tweets identified in this way then have to be archived in a searchable database.
Ted at Inner File Software was the twitter miner of the hour. He sat down at his machine and went to work on building the world a better twitter miner. The open-source software he designed, which runs in conjunction with Drupal, a web content management system to which Ted is a contributor, is only a couple of weeks old but has already extracted and stored more than a million tweets.
Best of all, he notes: "No digital canaries were harmed in the making of this twitter mine."
I wouldn't be much of a mother, would I, if I didn't suggest that you go on ahead and check out the company Ted is building around this fantastical new tool for twitter data mining and semantic analysis? Also, the open source twitter miner project.