visualizing data as needed it may take many hours to pull all the data down so you may need to restart several times here is run of gmane py retrieving the last five messages of the sakai developer listhow many messages: nealcaidin@sakaifoundation org re[building samuelgutierrezjimenez@gmail com re[building da @vt edu [building sakaimelete oracle shedid@elraed-it com [building sakaisamuelgutierrezjimenez@gmail com redoes not start with from the program scans content sqlite from one up to the first message number not already spidered and starts spidering at that message it continues spidering until it has spidered the desired number of messages or it reaches page that does not appear to be properly formatted message sometimes gmane org is missing message perhaps administrators can delete messages or perhaps they get lost if your spider stopsand it seems it has hit missing messagego into the sqlite manager and add row with the missing id leaving all the other fields blank and restart gmane py this will unstick the spidering process and allow it to continue these empty messages will be ignored in the next phase of the process one nice thing is that once you have spidered all of the messages and have them in content sqliteyou can run gmane py again to get new messages as they are sent to the list the content sqlite data is pretty rawwith an inefficient data modeland not compressed this is intentional as it allows you to look at content sqlite in the sqlite manager to debug problems with the spidering process it would be bad idea to run any queries against this databaseas they would be quite slow the second process is to run the program gmodel py this program reads the raw data from content sqlite and produces cleaned-up and well-modeled version of the data in the file index sqlite this file will be much smaller (often smallerthan content sqlite because it also compresses the header and body text each time gmodel py runs it deletes and rebuilds index sqliteallowing you to adjust its parameters and edit the mapping tables in content sqlite to tweak the data cleaning process this is sample run of gmodel py it prints line out each time mail messages are processed so you can see some progress happeningas this program may run for while processing nearly gigabyte of mail data loaded allsenders and mapping dns mapping