Can mathematical ideas applied to CS problems count as "undergraduate research"?
Undergraduate research is whatever research that you do as an undergrad, period. So if you do something that ends up being publishable, or feeds into another research project in a real way, it's undergraduate research.

However, for practical purposes, you want to get some faculty involved in your research project. It will help with funding for travel (you do want to submit to conferences rather than journals in CS), it will make the work better, and you will end up with a good letter writer afterwards. And if you think your project won't be interesting to faculty in your current school, what makes you think it will later on swoon an admission committee?


  would it look good (to future graduate schools to which I will apply) if I, for example, write a paper or two based on things I do applying mathematics to Computer Science and programming, e.g.- solving optimization issues in GPU processing of large data sets, etc?


Sure - if the paper is good. Solving optimization issues does not sound bad at all to me on first glance, but of course it is impossible to tell without seeing the paper or knowing in much more detail what you would want to do.


  Even if the papers aren't so interesting, would it at the very least look good on my "resume", so to speak?


I guess that depends on what you mean with "so interesting". It does not have to be A* / Transactions level quality, but the paper certainly needs to be published in a reasonable, peer-reviewed, non-spam venue. Papers in pay-to-publish OA journals don't help at all.

This seems to me to be an unfortunate side effect of the "prior research experience required" mantra that many admission committees nowadays seem to have - many students seem think that sending crap papers to pseudo-venues will help them get accepted. They don't. A paper that is substantially below the level that a prof. would want to submit himself won't help you.


  For the record, I'm hoping to go to graduate school to study mathematics, not computer science, which is why I can't quite answer this question myself.


If you want to continue in maths, a maths paper will help more than a CS paper. Still, a good CS paper will be much better than no paper at all, I suppose.