Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. ||||| Q: Will Barack Obama be a one-term president? 
 
 A: Yes, he might last that long. 
 
 Honest to goodness, the man just does not get it. He might be forced to pull a Palin and resign before his first term is over. He could go off and write his memoirs and build his presidential library. (Both would be half-size, of course.) 
 
 I am not saying Obama is not smart; he is as smart as a whip. I am just saying he does not understand what savvy first-term presidents need to understand: 
 
 You have to stay on message, follow the polls, listen to your advisers (who are writing the message and taking the polls) and realize that when it comes to doing what is right versus doing what is expedient, you do what is expedient so that you can get reelected and do what is right in the second term. If at all possible. And it will help your legacy. And not endanger the election of others in your party. And not hurt the brand. Or upset people too much. 
 
 Recently, President Obama decided he had to weigh in on the controversy surrounding the proposed construction of a Muslim community center and mosque near ground zero in New York. 
 
 It is a controversy Obama could have ducked (he had been doing so for weeks), but he finally decided he needed to lend his voice and the weight of the presidency to speaking out for what is right. 
 
 So on Friday night, he said: “As a citizen, and as president, I believe that Muslims have the same right to practice their religion as everyone else in this country. That includes the right to build a place of worship and a community center on private property in lower Manhattan in accordance with local laws and ordinances.” 
 
 See what I mean about not getting it? 
 
 John Feehery, a Republican consultant, told Sheryl Gay Stolberg of The New York Times, “This is not a unifying decision on his part; he chose a side. I understand why he did this, but politically I think it’s a blunder.” 
 
 You could not put the conventional wisdom more clearly: It is far better for a president to do nothing than to choose a side. Even if the side he chooses is the right one from an ethical or moral perspective, it is a “blunder” politically because inevitably it will upset some people. 
 
 The problem for Obama is that he appears to have taken seriously all the “change” stuff he promised during his campaign. And he has been unable to make the transition from candidate to president. ||||| Last Friday, at the start of Ramadan, President Obama presided over the White House's annual iftar dinner and made some rather bland remarks about religious freedom. The context, of course, was the controversy over the proposed mosque in Lower Manhattan, which is not, as Obama insisted, about freedom of religion but about religious tolerance. And then, having once again gotten high praise for so very little, he went to bed a panicked man and reached, trembling, some hours later, for a political morning-after pill to take back some of what he had said. Whew, for a moment there he was pregnant with principle. 
 
 No more. "I was not commenting, and I will not comment, on the wisdom of making the decision to put a mosque there," Obama said in revising and extending and eviscerating his remarks of the previous night. He had merely been commenting on freedom of religion. Turns out he's for it. 
 
 The president muddled his message. Does he not grasp that questioning the "wisdom" of the mosque's placement is predicated on thinking that 9/11 was a Muslim crime? Does he not understand that the issue here is religious prejudice, not zoning? The answer, of course, is that he does. But unlike Henry Clay, he would rather be president than right. 
 
 The very ugly controversy over the planned Islamic center -- not at Ground Zero, mind you, and not even within eyeshot -- has managed to make fools or knaves out of some pretty smart people. Some of them have embarked on a fruitless hunt for the perfect analogy. The winner, as you might have imagined, goes to that evil cherub Newt Gingrich, formerly of Georgia but now of any meeting hall with a spotlight. He said approving the mosque "would be like putting a Nazi sign next to the Holocaust museum." 
 
 Gingrich keeps trying. Earlier he had argued that since there are no churches or synagogues in Saudi Arabia, "there should be no mosque near Ground Zero." But the mosque is not Saudi Arabian; it is Islamic, a distinction not all that hard to keep in mind. The comparison to a Nazi sign at the Holocaust museum is equally specious. Every Nazi was dedicated to the persecution and/or murder of all Jews. This is not the case with Islam and the attack on the World Trade Center. That attack was conducted by a handful of fanatics, not an entire religion. 
 
 Others have joined in the false analogy contests. The most surprising is Charles Krauthammer, my longtime colleague on The Post's op-ed page. In a belabored analogy, he said that while "no one objects to Japanese cultural centers, the idea of putting one up at Pearl Harbor would be offensive." Yes, indeed. But all of Japan attacked Pearl Harbor and declared war on the United States. It was not a rogue act, committed by 20 or so crazed samurai, but an attack by a nation. You can look that up. 
 
 Krauthammer, though, could not be stopped. He likened the mosque to a "commercial tower over Gettysburg," then to the attempt to establish a convent outside of Auschwitz and, inevitably, to "a German cultural center at, say, Treblinka." Enough said. We all have bad days. 
 
 If it is not false analogies that pollute this debate, it is false populism. The people are opposed. John Boehner, the House minority leader, says so, and so does Rep. Peter King, the Long Island Loud Mouth who is clearly running for something. They are right -- but so what? Would they have liked Lincoln to have deferred to popular sentiment in the South regarding slavery? Would they have liked Truman to have polled the Army about desegregation? Minority rights are embedded in our Constitution. It was the perceived lack of them that caused the states to seek some immediate amendments, what we now call the Bill of Rights. King, Boehner and the rest of the GOP mob are showing a fearless willingness to pander to majority prejudice. Newt has mounted a crusade against radical Islam. No Saracen will be safe. 
 
 The inclination to go from the particular to the general -- to blame a people for the acts of a few -- is what has always fueled pogroms and race riots. History shows that it is a natural tendency and it will literally run riot if not controlled. It is the solemn obligation of elected leaders to restrain such an urge -- to be moral as well as political leaders. Obama almost pulled that off, but he flinched. 
 
 Yes, he couldn't. 
 
 cohenr@washpost.com 
 
 For more Post opinions on the New York mosque, read Michael Gerson's Obama's mosque duty, Eugene Robinson's Republicans pander over 'Ground Zero mosque', Dana Milbank's Religious tolerance, then and now, Charles Krauthammer's Sacrilege at Ground Zero and the editorial Obama needs to show leadership on mosque debate. ||||| A column (h/t, MEMRI ) in the August 16, 2010 London daily Al-Sharq Al-Awsat by Abdul Rahman Al-Rashid, director of Al-Arabiya TV and the paper's former editor, “A House of Worship or a Symbol of Destruction?” should mean the end of plans for a mosque near Ground Zero. Mr. Al-Rashid supports President Obama’s stand for the mosque in principle (as he supports Obama-like or even beyond-Obama-like policies with respect to the Middle East). He’s no neocon. But his practical case against building the mosque is irrefutable. It should lead well-meaning liberals to join with us dastardly conservatives (well, it would be too painful for them to join with us—they can simply act in parallel, on their own, while continuing to denounce us) in calling for the organizers to shelve the plans for a mosque at this site. Or will the organizers pull the plug even before their supporters get around to urging them to do so? 
 
 "I cannot imagine that Muslims want a mosque on this particular site, because it will be turned into an arena for promoters of hatred, and a symbol of those who committed the crime. At the same time, there are no practicing Muslims in the district who need a place of worship, because it is indeed a commercial district....The last thing Muslims want today is to build just a religious center out of defiance to the others, or a symbolic mosque that people visit as a museum next to a cemetery....[T]he battle against the 11 September terrorists is a Muslim battle...and this battle still is ablaze in more than 20 Muslim countries. Some Muslims will consider that building a mosque on this site immortalizes and commemorates what was done by the terrorists who committed their crime in the name of Islam. I do not think that the majority of Muslims want to build a symbol or a worship place that tomorrow might become a place about which the terrorists and their Muslim followers boast, and which will become a shrine for Islam haters whose aim is to turn the public opinion against Islam.” 
 
 Feel free to read the whole thing here. 
 
 This will be over soon. There will be no thirteen-story mosque near Ground Zero.
– The proposed mosque and Islamic center near Ground Zero continues to dominate pundit-land. Here's a sampling: Richard Cohen, Washington Post: Obama's backtracking was a shame. People will always blame the actions of a few on the many, as they are doing now with Islam. "It is the solemn obligation of elected leaders to restrain such an urge—to be moral as well as political leaders. Obama almost pulled that off, but he flinched." (At Politico, Roger Simon defends Obama with a tongue-in cheek criticism.) Jonah Goldberg, Los Angeles Times: The mosque was a dumb idea to begin with, and pragmatic politicians shouldn't have allowed it to advance this far. Also, Obama's speech was the "opposite of statesmanship. By elevating an already stupid idea and a poisonous debate, he forced everyone to take a side on a polarizing issue." Abdul Rahman Al-Rashid, Asharq Alawsat (UK): "I cannot imagine that Muslims want a mosque on this particular site, because it will be turned into an arena for promoters of hatred, and a symbol of those who committed the crime." William Kristol, Weekly Standard: "This will be over soon," he writes, citing the above column by al-Rashid (the director of director of Al-Arabiya TV) as an irrefutable argument. "There will be no thirteen-story mosque near Ground Zero."