Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. ||||| Mitt Romney just gave a more articulate defense of Obamacare than President Obama ever has. He continues to believe that the individual mandate is a good idea, despite the fact that the “free-rider” problem is a myth. His effort to make a distinction between Romneycare and Obamacare was not persuasive: If anything, he convincingly made the opposite case, that Romneycare and Obamacare are based on the same fundamental concept. 
 
 In recent months, Romney has claimed that Romneycare was a specific solution for the specific needs of Massachusetts. But in his remarks, he did not name one specific aspect of the Massachusetts health-care environment that is unique to that state. If there are no important differences between the Massachusetts health-care system and that of other states, why shouldn’t he believe that Romneycare should be the model for every other state? 
 
 Indeed, here is what Romney wrote in the Wall Street Journal on April 11, 2006, the day before he signed his signature health-care legislation: 
 
 How much of our health-care plan applies to other states? A lot. Instead of thinking that the best way to cover the uninsured is by expanding Medicaid, they can instead reform insurance. 
 
 The final straw, for me, was when Romney said that “there’s not a lot that I want to borrow from France and Switzerland.” I’m with him on France, but Switzerland (despite its own individual-mandate issues) has the most market-oriented health-care system in the developed world. I would love to borrow Switzerland’s exceptionally low levels of state-health spending, which they manage to achieve while providing universal coverage and high-quality care. Their tax rates aren’t shabby either. 
 
 — Avik Roy is an equity research analyst at Monness, Crespi, Hardt & Co., and blogs on health-care policy at The Apothecary. You can follow him on Twitter at @aviksaroy. ||||| TheWashingtonPost Romney’s speech: He digs deeper 
 
 This morning the Wall Street Journal editorial board accused Mitt Romney of being a technocrat and not a conservative. In his speech in Ann Arbor, Mich., today, Romney proved the editorial board correct. You could tell it wasn’t going to be a good outing when it became clear that a tiny room of about 100 invited guests was the setting. Message: Only handpicked friends could be counted on not to boo or laugh. 
 
 It was really two speeches. The first was a defense of RomneyCare. But it was one that will only deepen conservative criticism. Romney proudly defended his plan and the individual mandate that is now the object of the entire Republican Party’s ire. His explanation — mandates were designed to stop the free-rider problem — is exactly the justification for the same feature in ObamaCare. The mandate, conservatives believe, is an infringement on individual freedom. (It also didn’t work to reduce insurance cost in Romney’s state of Massachusetts.) 
 
 Romney trotted out the federalism argument once again, calling ObamaCare a “power grab.” But conservatives for nearly two years have been arguing that NO government should require individuals to purchase something they don’t want. That Romney still doesn’t understand this is a bit shocking. 
 
 The line of the day, and the one that will haunt him throughout his campaign, was “I believed I did what was right for my state.” Yikes. 
 
 The immediate Twitter reaction among conservative pundits and some keen-eyed liberal ones (I will include my colleague Greg Sargent in that second club) was withering. The consensus: If possible, Romney made things worse. If anything, Romney proved to be a far more articulate spokesman for the individual mandate than Obama. And if conservative pundits hated it this much, can you imagine what Tea Partyers will think? 
 
 Who thought up the idea of this sort of speech? I imagine the candidate did. No professional campaign adviser who knows the tiniest bit about GOP politics would have advised him to take this route. 
 
 The second part of the speech was a long discourse on what Romney would do for the country. Guess what? He says that it would be “different” than what he did in Massachusetts. He ran through a number of conservative ideas that have shown up in other plans ( purchase insurance across state lines, equal tax treatment for individual and employer-provided insurance, etc.) But, of course, the takeaway is he didn’t choose such a plan for his own state. 
 
 In the Q-and-A afterward he got mostly softball questions. But at one point he spoke admiringly of the French health-care system. I’m not kidding. 
 
 Romney is entirely lacking in self-awareness and understanding of the current Republican primary electorate if he thinks this speech is going to help. I’m sure his primary opponents, like many pundits, are dumbstruck that such a capable man could be so dense when it comes to his chosen profession. ||||| Santorum: “I greatly respect Governor Romney and admire many of his personal and professional accomplishments, but his work to institute the precursor to national socialized medicine is not one of them. Both Romneycare and Obamacare infringe upon individual freedom and exponentially increase the government’s healthcare cost burden. Romneycare has, in fact, not made healthcare better or saved costs in Massachusetts. It’s done just the opposite.”
– So did Mitt Romney help his cause with the right with his health care speech today? The early reaction isn't pretty for him on that front: Avik Roy, National Review: "His effort to make a distinction between Romneycare and Obamacare was not persuasive: If anything, he convincingly made the opposite case, that Romneycare and Obamacare are based on the same fundamental concept." Jennifer Rubin, Washington Post: "The consensus: If possible, Romney made things worse. If anything, Romney proved to be a far more articulate spokesman for the individual mandate than Obama. And if conservative pundits hated it this much, can you imagine what Tea Partyers will think?" Ezra Klein, Washington Post: The speech was "as thoroughgoing a defense of the individual mandate as I’ve heard in months. The White House would’ve been wise to take notes." Potential rival Rick Santorum, at Time: "Both Romneycare and Obamacare infringe upon individual freedom and exponentially increase the government's healthcare cost burden. Romneycare has, in fact, not made healthcare better or saved costs in Massachusetts. It's done just the opposite."