Andrew Trench notes from the revolution

Analysing the flow of social conversations with Python and D3

There's a great book by Matthew Russell called Mining the Social Webwhich I have been reading for several months now. The book hits the spot for me on multiple fronts because I am interested in network analysis, programming and news which are at the core of Russell's book.

I'm fascinated with trying to gain insight into how social networks (in their broadest sense) operate as this is something that occupies a large part of our time as an investigative unit. Gaining access to networks of power or understanding the linkages betweek key players is often at the heart of breaking open a major story or at least understanding who you need to be looking at.

I've been hacking out various Python scripts over the years which allow me to rapidly trawl through public information sources and to attempt some programmatic mapping of relationships and connections using brilliant Python libraries like networkx which can produce various graphs applying useful algorythmns like degree centrality, betweenness and so on.

For months I have also dabbled inD3, the powerful javascript library by Mike Bostock (now of the New Year Times team) but, boy, has it been tough! I find javascript challenging compared to the easy elegance of Python. It's like trying to write in Latin if you only know English! But a couple of weeks back I put my head down after fooling around with one of Russell's Python scripts designed to mine Twitter's search API and build a relatively interactive visualisation using the Protovis javascript library.

It worked pretty smoothly but Protovis had some severe limitations in terms of functionality and provided a major hurdle for me to get over since my javascript is so poor.

Instead I decided to try and remake the visualisation using D3 which has more scope for interactivity that I can see - and is also significantly better documented. After two weeks of missioning I finally worked it out and you can see the results over here (click image below) where my wife, The Grubstreet Gal, has used it to represent a Twitter debate over the National Press Club's controversial decision to name the rhino as newsmaker of the year.

I'm really pleased at how it worked out.

Basically what happens here is this:

1. My Python script (with major acknowledgements to Russell's open source original) checks in with the Twitter search API using an entered search term or hashtag;

2. It downloads about 600 messages in a couple of minutes (depending on the popularity of the dicussion) and then parses each message looking for the "RT" or "via" phrases using some clever regular expressions;

3. Once it finds a tweet that has been retweeted then it analyses it to identify the source of the tweet and attempts to extract than information (this is not 100% accurate. as you will see in the visualisation. but I would say is around 98% good because there can be misspellings in retweets or shortenings of words and so on);

4. As it locates these entities the relationships, or edges, between them are recorded in a networkx DiGraph (directed graph) which starts to give the network of the conversation some shape;

5. I also included code to capture the original tweet and save it as an attribute of each node so that on mouseover on the links between two parties in the interactive you can see what the original tweet was. This was one of the most difficult things for me to work out how to do and special thanks to www.d3noob.org and his excellent D3 guide for showing me the way!

6. Next the DiGraph is output as a json file which can then be imported as a data source to D3 which uses Bostock's genius to represent the network.

You will see that it can take a little while for the interactive to settle down thanks to complex physics involved in calculating the relationships and positions between the nodes.

This was a a major learning curve for me and I can't wait to tackle a few more of these. I'd like to apply some of this to natural language text processing, for example, to experiment with programmatically trying to visualise linkages between people in a corpus of text like a news archive or something. Also very handy in producing illustrations of power elites and so on on the fly.

Would love to hear your views and happy to share my (very hacky) code with anyone who is interested.