How to transform the CSV link in the Views Data Export module in a non-image link display?
I agree that this CSV button needs some refinements, similar to what you are describing in your question. And quite a few others seem to think likewise. 

To actually solve your issue, consider looking at the module's issue # 2261167, which is about an Option for non-image link display (issue status = needs review). Comment # 26 of that issue contains a patch which adds 2 new settings as shown in this screenprint:



Here are some more details about these 2 new settings: 


a checkbox to (if selected) provide the download link as a text link instead of an icon/image link.
a textfield to add a class to the link, so that you can give the link a class like "button" (see picture) to your text link


Do I'd definitely consider this patch.

As an alternative (2nd choice?), you may also want to look at issue # 1379672, which is about Export Button, make it styleable with CSS. It also includes a patch that you may want to consider (eg if the previous one for some reason doesn't fit).