When to use a preprocess function instead of changing the template
From the docs


  The main role of the preprocessors is to set up variables to be placed within the template (.tpl.php) files.


And building on that:


  There can be numerous preprocessors for each theming hook. Every layer from core, modules, engines and themes can have one, each progressively building upon the variables before being rendered through template files. This keeps the markup clean and easy to work with inside templates by placing most of the logic inside these preprocessors.


So if you're setting up a variable that you want to be rendered in a template, your code will go in a preprocess function.

When it's time to render the variables that have been set up, wrap them in specific markup and so on, you'll be editing template files.