What could cause a ground/soil smell on the first floor?
This is very likely a drainage issue. I would check the following things:


Does water drain away from the house
near this room?
Are plants growing
right up next to the foundation?


You might want to put a buffer of 10" of rock (4-6" deep) next to the house. 

Since the house has vinyl siding you need to peek under the siding and see if you have moisture collecting under the siding. Spray the side of the house gently with water or go out after a rain and pull back some of your siding and check for water seepage. The smell could be the result of smells outside but it could be moisture in the walls. 

The cinder blocks can also be an issue. Concrete holds a surprising amount of moisture. In addition to the things mentioned above, you likely would benefit from a dehumidifier in this room. Check the humidity with a cheap humidity tester and if the room is over 65% humidity you should consider dehumidifying.