Why was the U-2 so different from the SR-71?
Because the requirements were different. When U-2 was developed, the requirement was for an aircraft which would fly high at 70,000' due to the (mistaken) belief that the Soviets would be able to detect and engage aircraft only below that altitudes.

Once the U-2 entered service however, it became obvious that they could detect U-2 (they complained about the overflights, though misidentifying the aircraft), and once an aircraft was shot down, it became obvious that U2 isn't going to be of much help.

SR-71, which came later, was the result of a requirement which expected it to not only fly higher, but more importantly, faster. Interestingly, the Soviets went the same way with the Mig-25R. 

The main reason for the retirement of spy aircraft was the advent of reconnaissance satellites, which are immune to being shot down (not that it is impossible, but no one has done it except in tests). Spy aircraft are still operated only over areas where the threat from anti-aircraft systems are extremely limited or non-existent. At the time of its retirement, the USAF accepted that even the SR-71 is not invulnerable:


  In congressional testimony, Air Force Chief of Staff Gen. Larry D. Welch identified the increased survivability of reconnaissance satellites, SR-71 vulnerability to the Soviet SAM-5 surface-to-air missile and the cost of maintaining the SR-71 fleet. 


As for why U2 has outlived the SR-71, the main reason is the operating costs- the operating costs of SR-71 is quoted to be anywhere from \$85,000 to \$200,000 per hour, while the U-2 costs much less than that (incidentally, U-2s operating cost is less than that of its proposed replacement, the RQ-4 Global Hawk).

From the same source as above:


  The Air Force decision to retire the Blackbirds in 1990 is based on several factors. ... The cost factor is the most significant to the Air Force because it limits expenditures in other areas. Reagan Administration Air Force Secretary Edward C. Aldridge Jr. estimated that the money used to operate the SR-71 fleet could operate and maintain two tactical fighter wings.