Why does the B-2 have a smaller payload than other bombers of comparable size?
The simple answer is that it is optimized for being low observable/stealthy, and long range, rather than for maximum payload.

When you design an aircraft, you make a variety of trade offs depending upon which features are most important for that design.  The design of the B-2, which reaches back into the 1980's1, put a premium on the ability to avoid being detected by radar.  This informed the shape.  The USAF's strategic purpose for that aircraft was to enable it to reach globally, so that it could be launched form the continental US or a forward deployed base.  This put a premium on range/fuel capacity.  

@Durandal points to another detail/difference: The B52 and B1 both can carry ordnance on external hard points (and this is a must to get up to the cited payloads). The B2 has no such hard points because it would nullify its stealth capability. There is simply no room to put more munitions within the bomb bay.  

With the advent of precision weapons, which were in parallel development to the B-2, the raw tonnage wasn't as important as getting to the target area, and getting out "without being seen." 

Because aircraft are all constrained somewhat by the cube/square law in terms of size, the trade offs/compromises on that model arrived at a payload that fit in with the rest of the requirements.  

Both the B-52 and B-1 were build under a different design paradigm, under different requirements combinations, and were "pre-stealth" bombers: the B-52 was meant to get the most range and raw payload together, while the B-1 combined speed (supersonic, back when speed was perceived to be at a premium) and payload foremost.  While both have since benefitted from the use of precision munitions, the original designs (all nuclear weapons carriage considered) had to account for the problem of non-precision munitions shortcomings with accuracy.  (Experiences in WW II informed this problem, as did experience in Viet Nam).  You need to drop a pattern of "x many bombs" to have a high probability of hitting a given target. 



1 Insofar as the "flying wing" idea for the shape of the B-2, the initial attempts in that direction go back to the 1930's and the Ho_229 as so @paracetamol kindly reminds us.  Also, and FWIW, it is related to a DARPA  project that goes back to the 1970's (stealth / radar resistant in general) that was compromised publicly by President Carter during a press conference.  Actually, those paying attention had heard of it from Kelly Johnson: In fact, that existence was far from secret. On July 23, 1976, readers of Aerospace Daily were informed that Clarence L. (Kelly) Johnson, "the nation's premier aircraft designer" and architect of the U-2 and SR-71 spy planes, was bulding "a new 12,000-pound, one-man Stealth aircraft" at the Lockheed plant in Burbank, Calif. The magazine claimed that the $90 million program, sponsored by the Defense Advanced Research Projects Agency, was designed "to reduce aircraft visibility -- optical, acoustic, and radar signatures -- through new technology."