figure word cloud from the sakai developer list we will be using data from free email list archiving service called this service is very popular with open source projects because it provides nice searchable archive of their email activity they also have very liberal policy regarding accessing their data through their api they have no rate limitsbut ask that you don' overload their service and take only the data you need you can read gmane' terms and conditions at this pageit is very important that you make use of the gmane org data responsibly by adding delays to your access of their services and spreading long-running jobs over longer period of time do not abuse this free service and ruin it for the rest of us when the sakai email data was spidered using this softwareit produced nearly gigabyte of data and took number of runs on several days the file readme txt in the above zip may have instructions as to how you can download pre-spidered copy of the content sqlite file for majority of the sakai email corpus so you don' have to spider for five days just to run the programs if you download the prespidered contentyou should still run the spidering process to catch up with more recent messages the first step is to spider the gmane repository the base url is hard-coded in the gmane py and is hard-coded to the sakai developer list you can spider another repository by changing that base url make sure to delete the content sqlite file if you switch the base url the gmane py file operates as responsible caching spider in that it runs slowly and retrieves one mail message per second so as to avoid getting throttled by gmane it stores all of its data in database and can be interrupted and restarted as often