How many vibrations are expected for butatriene and cyclobutadiene?
Perhaps an explanation of where these mysterious equations come from will help you understand how to apply them properly.

I only need you to make one, single (correct) assumption: all atoms in a molecule are capable of moving in each of the three Cartesian coordinate directions.

That is, each of those little hydrogens can wiggle around in the x, y, and z coordinates as much as it pleases.

That means that we have a total $3N$ ways that the whole molecule can move because each of $N$ atoms can move in three directions.

Now, let's consider how molecules move. If every one of those atoms happened to be moving in the same direction, what would that look like?

Well, that would be translation of the molecule (i.e. moving). Any molecule can do this in three directions.

So, excluding translation we have $3N-3$ ways of moving around.

Now, another thing we know molecules do is rotate. By the same reasoning, we can say that a molecule can rotate in all three coordinate directions.

This leaves us with $3N-6$ ways to move around.

But, what if that molecule was linear? Well, then that molecule can only rotate in two unique directions because spinning around like a cylinder is known to be a zero energy process. Another way of saying that is that it doesn't happen.

So for a linear molecule there $3N-5$ ways of moving around still because we only subtracted out two rotational modes.

Of course, now we see that the only other way our molecules can move is by vibrating.

Thus, for a linear molecule, $$Number\ of\ normal\ modes=3N-5$$and for a nonlinear polyatomic molecule,$$Number\ of\ normal\ modes=3N-6$$

So, using these equations which now hopefully make some more sense, it's quite obvious that we do indeed include the hydrogens in our $N$ and find there are 18 normal modes of vibration for both molecules.

And, if there were an isomer of the formula $\ce{C4H4}$ which was linear then there would be 19 normal modes, but there is no such molecule so, yes all molecules with the molecular formula $\ce{C4H4}$ have 18 vibrational modes.