Light Fixture box with 3 sets of wires (confusing wiring)
What you've got here is power in to the fixture box, power out to whatever is next and a "switch leg" cable to the switch. You are probably right that the wires from the fixture were connected to the blacks with the wire nuts.  Unfortunately this isn't best practice, dangerous at best.  This setup leaves the fixture "hot" all the time and effectively interrupts the neutral.  

The way to make it right is to connect the right black wire with the 2 on the left.  Disconnect the white wires and code tape black the white wire coming back from the switch.  Connect that to the black wire on the fixture and the other wire (white) from the fixture to the other white wires.   Best practice would be pigtail the whites so you only have one wire to deal with when installing the fixture.  

Oh and get a wire nut on those ground wires while you're in there!