What challenges restrict the resolution of spacefaring digital cameras?
There is one overriding requirement for deep-space missions: reliability. In general NASA Preferred Parts are quite stodgy, because the overriding need is for a mature, well-understood technology. Cutting-edge technology that doesn't work is frowned upon under the circumstances. So 10-year-old image sensors are about what you expect.

Additionally, if you read the JunoCam article you linked, you'll see (second paragraph, first sentence) that data transfer rates are quite slow, on the order of 40 MB per 11 days. Increasing image size cuts down the number of images which can be acquired, and I expect that a lot of effort went into determining the tradeoff between number of images and image resolution.

For what it's worth, NASA has been pushing for better data rates for its programs, but the limited power and long ranges involved make this a non-trivial problem. The LADEE mission a couple of years ago incorporated the LLCD (Lunar Laser Communication Demonstrator) which worked quite well, and this holds great promise (optical communication limit of 1 bit/photon at the receiver), so future missions may be able to do a lot better.