Can a Shimano XT (mountain) hydraulic disc brake caliper be used with an Ultegra STI road brake lever?
Shimano recommends a totally different mountain brake levers for the 2-piston XTR brake (the M9100) vs the XTR 4-piston model (the M9120). Why would you need a different brake lever for these different calipers with the same size hose? 


The key is that there are two brake options available: "race" and "trail". "Race" features 2-pot calipers and lever without Servo Wave™, which yields different modulation curve (more linear?). The "trail" version is 4-pot and with Servo Wave™ levers, which has more of an on-off modulation.

As far as I know, all Shimano calipers/levers have very close displacement/pressure values to be deemed compatible. On one hand, I did not hear about incompatibilities between any pre-9100 parts, and on the the other there's a lot of reports of mix-matching working with great success. Unless there are reports stating otherwise, I'd bet on compatibility trend. In the end, calipers are not that expensive to experiment with and there are cheaper options (BR-M520, BR-M640).


  How could you know which post-mount caliper (the M9100 vs M9120) would work with an STI ST-R8020 brifter?


What Shimano "recommends" and what actually works are not necessary same things. Shimano is a conservative company bound by legal responsibilities. If you want 4-piston calipers to pair with STIs, take a look at Hope RX-4 SH (explicitly compatible, I run these) or Shimano calipers mentioned above.

As with all things new, there's not much feedback, you might as well get that caliper and report results here as an answer.