What is the risk of using only ESD gloves?
While you are not likely to give a direct shock to components, ESD control also requires minimizing all electric fields. Triboelectric charging (charging from friction) can generate electric fields with a potential of  thousands of volts which can travel through the air. To minimize this potential source of ESD you must control materials around sensitive electronic components and the humidity of the air (more humidity makes the air more conductive). The best way is to use grounding mats, straps ect.

An insulator between two potentials doesn't do much when you have fields that reach into the kV range, in short the best way is to make everything the same potential by grounding. 

This being said ESD is about minimizing risk. Industries like medical and aerospace have zero tolerance for risk, you can't have a billion dollar satellite fail because someone zapped a component that cost a few dollars. If a production line is having failures or cannot tolerate the chance of a product failing in the field then ESD is a risk factor. The hard part is ESD failure is dependent on the component and the field and both of these are highly variable and makes the risk difficult to determine. The biggest problem ESD is an unknown risk factor so the best way to eliminate the risk is to ensure all fields are zero. 

It sounds like your application will tolerate some risk, you need to balance the costs of ESD control with the costs of failure, since I don't know what your failure rate is I cannot speak for one or the other.  

NASA\JPL implement some of the strictest ESD controls, check out this article on best practices. You will have to decide on what works for you, but make sure you understand how ESD problems arise and how they need to be controlled, then find out what works for you because ESD control comes with a time and capital costs.