Prosperity lost from players leaving is because of their Build Score. From their Manor. Basically what they have set up for you to veiw. (Not what they have stocked in the grounds) In essence, Build score which provides prosperity is actually Player prosperity. Not Camp prosperity. It is on loan to the camp while they are living their. The higher their build score, the more prosperity they have, Which is loaned to the total prosperity of the camp, you will lose this players prosperity when kicked or they leave.p

Ive done the testing on upgrades. If you place a wall, you gain 1 prosperity, if upgrage a wall from level 1 to 2. Or 2 to 3, or 7 to 8 You gain 1 prosprerity, consecutively 1 prosperity per level (8 prosperity per level 8 wall). 

I forget the numbers of buildscore, but I think it was 2 per level 1 to 7, then 7 to 8, was 4 which provided a total of 18 build score per lvl 8 wall.

My apologies if this is confusing.