Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What�s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa�s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available �warts and all� for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you�re hoping to do with it. We may not be able to say �yes� to all requests, since we�re just figuring out whether this is a good idea, but everyone will be considered. ||||| Investors in bank stocks are about to get a big cut of the profits again. 
 
 Financial analysts say the nation’s largest banks are ready to begin restoring their dividends in the first half of the year, after a three-year pause to repair their damaged balance sheets. The reversal could put billions of dollars in the pockets of pension funds and retirees who had viewed bank shares as dependable sources of income. 
 
 Clues to how big a payout is in store could come as early as Friday, when JPMorgan Chase announces its 2010 financial performance, the first of many earnings reports to come over the next week from the likes of Bank of America, Citigroup, Goldman Sachs and Wells Fargo. 
 
 If the big banks deliver a second straight year of rising profits, as many analysts expect, the conditions would be in place for regulators to approve dividend increases by as early as March. 
 
 As the financial crisis worsened in 2008 and 2009, all but a handful of financial institutions cut their once-lucrative dividends to just pennies a share, hurting ordinary investors who had come to see them as sources of income. JPMorgan, for example, now has a dividend of 20 cents a share annually, down from $1.52 before the crisis. 
 
 Over all, the financial sector of the Standard & Poor’s 500-stock index paid out $51 billion in dividends in 2007. By 2010, that figure had shrunk to $19 billion. 
 
 “It’s a significant milestone,” said Gerard Cassidy, a veteran bank analyst at RBC Capital Markets. “The return of dividends signals that the banks are back, and the Federal Reserve wants to inspire confidence in the marketplace so that banks lend more.” 
 
 The financial industry has returned to health much faster than expected, helped by an alphabet soup of federal aid programs totaling more than $3 trillion, ultralow interest rates and a surging stock market. 
 
 Banks are expected to record $70 billion in profits in 2010, according to Foresight Analytics, a financial research firm. That would be up from $12.5 billion in 2009 but remains about half the level reached in 2006, before the housing market collapsed and the financial system almost came undone. 
 
 The earnings reports for the fourth quarter of 2010 are also likely to show that corporate and consumer lending is starting to come back while losses on bad loans are continuing to ease. 
 
 Wall Street’s trading businesses are expected to turn in a strong performance because of an increase in deal-making activity late in the year. 
 
 This week the Federal Reserve began another round of so-called stress tests of the nation’s 19 largest banks, evaluating their ability to remain financially healthy in the face of a still-anemic economic recovery and tough new regulations that will cut deeply into revenues. Unlike the first round of tests, the findings this time will not be made public. 
 
 Before approving a dividend increase, regulators must sign off on a bank’s stress test and conclude that the bank can meet the higher capital requirements put in place by new international agreements and the recent overhaul of financial regulations in the United States. They also must have fully repaid any federal bailout funds they accepted at the height of the crisis. 
 
 Advertisement Continue reading the main story 
 
 Advertisement Continue reading the main story 
 
 While the return of dividends will be welcomed by ordinary investors, it remains a delicate issue for the banks as well as regulators and politicians in Washington, said Chris Kotowski, a bank analyst with Oppenheimer. 
 
 Many voters are still angry about the government-led bailout that rescued banks after the collapse of Lehman Brothers in 2008. More recently, the return of bonuses on Wall Street has stirred outrage. 
 
 “It’s purely a matter of making it palatable to the public,” Mr. Kotowski said. “Banks are fully capable of doing it. But everyone’s afraid of headlines that say just two years after the bailout, the fat cats are getting dividends again.” 
 
 Partly as a result, he said, dividends will probably be restored in stages, and it could be take until the end of 2012 for them to return to historical norms. 
 
 For decades, shares of banks, along with utilities, were the favored choice of retirees and other conservative investors who looked forward to a steady payment each quarter. 
 
 That all changed when the financial crisis struck, forcing Citigroup to cut its dividend as it braced for a wave of huge losses tied to loan defaults. 
 
 Although the federal bailout program did not require banks to lower their dividends in most cases, regulators all but forced many banks into making cuts by insisting that they hold more capital in reserve to cushion against losses. 
 
 By the spring of 2009, several of the largest banks — including JPMorgan, Bank of America and Wells Fargo — cut their dividend to just pennies a share each quarter. 
 
 Just as the banks cut their dividends at different rates over the course of months, the timing of dividend increases will probably also vary widely across the industry. The strongest banks, including JPMorgan, State Street, U.S. Bancorp and Wells Fargo, should be in the first wave this spring, several analysts said. 
 
 For Bank of America and Citigroup, which continue to suffer steep losses on mortgages and consumer loans, the analysts said higher dividends probably would not come until later this year or early next year. 
 
 Several regional lenders, including Fifth Third Bank, KeyCorp, SunTrust and Regions Financial, are barred by regulators from raising their dividends. None of these banks have repaid their bailout money in full. 
 
 In particular, analysts and investors are eagerly anticipating what the chief executive of JPMorgan, Jamie Dimon, will say on the company’s earnings call on Friday, looking for any hint of the bank’s dividend plan. JPMorgan emerged from the financial crisis in far better shape than most of its rivals, and Mr. Dimon has been outspoken about his desire to raise his company’s payout. 
 
 “We’re going to be building up a lot of excess capital,” he said in a CNBC interview on Tuesday. “So, we would like to restart a dividend.” 
 
 Eventually, JPMorgan’s restored dividend could equal $1.50 a share annually, said Michael Scanlon, senior equity analyst with Manulife Asset Management in Boston. That would equal a yield of 3.3 percent based on its closing price of $44.45 on Thursday. 
 
 That would be up from 0.5 percent now. More important, it would catapult the yield on JPMorgan shares to far above the 2.26 percent yield on certificates of deposit, a popular vehicle for investors seeking income. 
 
 “It won’t come right out of the chute at that level,” Mr. Scanlon said. “But it’s definitely the end of a long drought.”
– America’s top banks are expected to report yet another quarter of profits, putting them in prime position to start paying dividends again, analysts tell the New York Times. JPMorgan got things rolling today by reporting a 47% increase in profits, notes the Wall Street Journal. It will be followed by Bank of America, Citigroup, Goldman Sachs, and Wells Fargo next week. Analysts expect their combined 2010 profits to come in at around $70 billion, up from $12.5 billion in 2009. This week, the Fed began another round of “stress tests” on the banks. If the results are good, regulators will give banks the OK to reinstate dividends, putting money back in the pockets of investors, pension funds, and retirees. “It’s a significant milestone,” said one analyst. “The return of dividends signals the banks are back, and the Federal Reserve wants to inspire confidence in the marketplace so that banks lend more.”