JMRI Code Swarm

Download the Video

You can download the JMRI Code Swarm here. Note that this is a large file (~15M) and is in zip format. After downloading, you will need to unzip it to extract the mpeg file, then play it in your favourite video player. It has been tested on Windows XP with Windows Media Player(V9) and Irfanview, on Ubuntu with Totem (after installing the Gstreamer codec and its plugins) as well as on the Mac.

The video is copyright JMRI and the musical track is copyright AdHoc, see license details.

Introduction

What the heck is a Code Swarm?

Code Swarm is a new open source visualization package that was released this past summer (2008). It can be used to provide a dynamic video history of a software project, in this case JMRI.

Each time a developer "commits" (i.e. saves) a new file or a change to an existing one, a colour-coded dot appears on the screen. The dots coalesce around the developer's name and the names move around the screen as others join in. The file dots fade away if there is no further activity on them, or flare up if changed again. The names fade as well if that developer has no more activity, or change to bold font if more contributions are made.

Although it sounds simple, it can be quite mesmerizing as the developers and files swirl around and it gives a very clear view of the increasing JMRI activity, especially over the last two years.

The resulting video gives a lively perspective of how JMRI has developed and who has been involved. The 4 minute JMRI Code Swarm covers the beginning, with the very first commits in June of 2001, through to early January 2009, a period of about seven and a half years and encompassing nearly 23,000 commits.

Here is an animated gif sample of about 1.5 secs from the video (covers about 2 weeks in April 2008):

After viewing the JMRI code swarm, one thing is abundantly clear, and that is the great debt we owe Bob Jacobsen. The large number of commits he has made is very evident and is an order of magnitude higher than any other developer. Knowing how quick Bob is to highlight others' contributions, I'm sure he would be the first to point out that his commits also include those of many others on the Acknowledgements list who submit files or patches and don't commit directly. Unfortunately, the code swarm can't separate out these contributions and attribute them to the true originators, but even if it could, Bob's own contributions are still huge.

Enjoy!

Technical Details

Warning! Any non-Geeks who have wandered into this section by mistake should stop now before any permanent damage is done to your intellect or personality. Proceeding further may result in glazed eyes, confusion and disorientation. Serious cases could result in an all-consuming interest in programming with the accompanying loss of social skills and deterioration in personal hygiene. Do not read while operating heavy machinery. Reader discretion is advised.

Data Source

• All of this was done in Windows XP, but the process will be essentially identical for other platforms.
• First, the project repository HEAD is checked out of CVS into a local directory. This can be done as a read-only version as described here. For Windows based systems, an excellent tool for this is TortoiseCVS
• Next, the commit data is extracted from the local repository into log files. This is obtained by issuing the command line statement:
  
  cvs log > activity.log
  
  in each of the following subdirectories of the repository:
  • help
  • java
  • jython
  • lib
  • resources
  • web
  • xml
  
  
• The activity.log files now in each subdirectory have to be combined into a single file for processing. Pick one of them as the master and save a copy so you can go back if you make a mistake while combining the files. Then open each of the other files in turn and copy the entire file contents with the exception of the first two lines. (The first line contains the file name and the second line is blank). Paste the copied contents at the end of the master file and repeat for each of the activity.log files.

Code Swarm Frame Generation

• See this page for instructions on downloading the Code Swarm software using an SVN client and building from source. I use Eclipse, so I downloaded the Eclipse module from SVN and simply ran the software from within Eclipse.
• Once the Code Swarm software is ready and the log files have been combined, the video generation process can begin. The basic process described here is used to generate the code swarm frames. The following steps give more detail and assume the code swarm software has been downloaded into a directory called codeswarm :
  • Convert the master activity.log file to a properly formatted xml file. To do this, copy it to the codeswarm/convert_logs directory. (Note: Python must be installed and working before proceeding. If you don't have it, install it now). In a command session, change to the convert_logs directory and issue the command:
    
    python convert_logs.py -c activity.log -o activity.xml
    
    This will generate a file called activity.xml in the codeswarm/convert_logs directory
    
    
  • This step is optional. Open the activity.xml file in a text editor and do a Replace All for each user id to replace it with the committer's name. The names are available on our Sourceforge site.
  • One statement in the source code for code_swarm can be changed to make the Legend more visible, especially when compressed video versions of the swarm will be generated. In the code_swarm.java file, find the drawLegend() method and change the line
    textFont(font) to
    textFont(boldFont)
  • The code swarm config file sample.config, which comes with the program in the codeswarm/data directory, has the following options changed (all others are left at default):
    
    InputFile=convert_logs/activity.xml
FontSize=14
BoldFontSize=18
comment out: MillisecondsPerFrame=21600000
uncomment and set: FramesPerDay=2
ColorAssign1="Documentation","(.*doc.*)|(.*htm.*)|(.*html.*)|(.*shtml.*)|(.*css.*)|(.*properties.*)", 0,0,255, 0,0,255
ColorAssign2="Java/Jython Files","(.*java.*)|(.*py.*)", 255,255,0, 255,255,0
ColorAssign3="Resource Files","(.*xml.*)|(.*dtd.*)|(.*gif.*)|(.*xsl.*)|(.*png.*)|(.*jpg.*)", 255,0,0, 255,0,0
comment out all other ColorAssign statements
    TakeSnapshots=true
DrawFilesSharp=true
DrawFilesFuzzy=false
DrawFilesJelly=true
ShowEdges=true
EdgeLength=50
PersonSpeed=1.50
PersonMass=20.0
HighlightPct=10

• The ColorAssign statements above in the config file colour-codes the files to be displayed as:
  

  Documentation

  in Blue; files: all help files, but not web site (doc, html/css) and language translations(properties)

  Java/Jython

  in Yellow; files: all program files (java) and scripts (py)

  Resources

  in Red; files: all decoder and programmer definitions (xml/dtd/xsl) and images (gif, png, jpg)

  All other files types are considered Misc and are shown in gray.

• Create an empty folder under the codeswarm directory called frames
• Execute code_swarm, which will open a graphics window showing the frames as they are generated and saved in the codeswarm/frames directory. They are saved as a series of .png files, with names in the form of code_swarm-#####.png where ##### starts at 0000 and counts up to the number of frames. This naming format is required for the benefit of the software that is used to convert the series of still pictures into a video.

Music

• If music is desired obtain it now.
• For this video, the musical track used is Sumbala from the album Toutes directions, copyright AdHoc and used under the Creative Commons license (attribution; non-commercial; share-alike). The music can be found at Jamendo.

Video Generation

• After code_swarm has created the frames, ffmpeg can be used to combine the frames and the mp3 file of the music into a single mpeg file. For Windows, a pre-compilied binary version is available here. It needs to be unzipped to a folder before use (for the purposes of these instructions, the folder name is assumed to be ffmpeg).
• Copy the codeswarm/frames directory to the ffmpeg/bin directory. Also, copy the mp3 (sumbala.mp3 in this case) file to the same ffmpeg/bin directory. Copying these files isn't really necessary but it keeps the path names short in the following command.
• In a command session, change to the directory containing the ffmpeg/bin directory and issue the following command:
  
  ffmpeg -i sumbala.mp3 -f image2 -r 24 -i frames\code_swarm-%05d.png -qmax 15 jmricodeswarm.mpg
  
  This creates an mpeg file called jmricodeswarm.mpg in the same directory that is a combination of the video (made from the individual frames) and the musical sound track.
• The quality of the video can be changed by modifying the -qmax 15 setting in the above command. This value was chosen to give a balance between quality and file size (decreasing -qmax 15 improves quality but increases file size). For a high quality but much larger file, change -qmax 15 to -sameq

---