Is my performance affected by invisible items?
Like is it still running in the background while invisible?
-> if an object is in the renderlayer, whatever the color (even invisibility) blender will still compute it ( so it will decrease performance).
If you don't need this object you can put in in another renderlayer that you don't use.

but control it's invisibility somehow
-> you can use a keyframe on the color of the material to make visible/invisble at the moment you want