how many spores are needed to "infect" a wall?
Yes, the box likely had many fungal spores attached to it since it smelled “moldy”. This doesn’t mean that your room will definitely be contaminated because spores often wait for ideal conditions to germinate. There are fungal spores in the air we breathe all the time, so if those haven’t already colonized your wall then adding the ones from the box are unlikely to cause any problems. 

If you do see spots appear, they can be easily removed with a damp paper towel, generic home cleaner, or diluted bleach.