Spritesheet cut to individual frames
First of all, this kind of manipulation is done using CSS. E.g. http://blog.teamtreehouse.com/css-sprite-sheet-animations-steps

Nevertheless, let's see what can be improved:

Instead of nth-child selectors, use classes instead

It provides some meaning to the images, and you can replace the src easily afterwards:

&lt;div id='animation'&gt;
    &lt;img class='first-frame' src='...'&gt;
    &lt;img class='first-frame-glow' src='...'&gt;
    &lt;img class='second-frame' src='...'&gt;
    &lt;img class='second-frame-glow' src='...'&gt;     
&lt;/div&gt;


Refactor the code into a general function that handles the styling

var style_switcher = function(first, second, third) {
    document.querySelector(first).style.display = "initial";
    document.querySelector(second).style.display = "none";
    document.querySelector(third).style.display = "none";
};


Use named constants

var TIMEOUT = 300;
var INTERVAL = 600;


This will make things easier when you want to change the values.

Put everything together

window.setInterval(function(){
    if(animation_hovered == false){
        style_switcher('.first-frame', '.second-frame-glow', '.second-frame');
    }else{
        style_switcher('.first-frame-glow', '.second-frame-glow', '.second-frame');
    }
    window.setTimeout(function(){
        if(animation_hovered == false){
            style_switcher('.second-frame', '.first-frame', '.first-frame-glow');
        }else{
            style_switcher('.second-frame-glow', '.first-frame', '.first-frame-glow');
        }
    }, TIMEOUT);
}, INTERVAL);


If there is more common behavior, you can abstract that into a function.

Good luck!