Why is the compound 3,4-dihydroxy-3-cyclobutene-1,2-dione more acidic than acetic acid?
This molecule is commonly known as Squaric acid. The Wikipedia entry gives the possible resonance structures of the conjugate base (added here for convenience):


You can see that there are four possible resonance structures for the conjugate base, resulting from the proximity of two ketone groups and two hydroxyl groups around the ring. This proximity allows movement of electrons around the square ring and between the ring and each oxygen atom.

On the other hand, acetic acid (or rather, the conjugate acetate anion) has only two resonance structures, as seen here:


Since the principle is that a molecule with more resonance structures is more stable than a molecule with less resonance structures, squaric acid anion is more stable than acetate anion. Since a more stable conjugate base makes an acid stronger, squaric acid is considered to be stronger than acetic acid. 

This conclusion is also evident by the Pka values. This table provides Pka values for many acids, including squaric, acetic and formic acid. You can see that squaric acid has two values for two acidic protons - 1.5 for the first proton and 3.5 for the second proton. acetic acid has a Pka value of 4.75 and formic acid has a Pka value of 3.75.
This means that the strongest acid of the three is squaric acid, followed by formic acid, and the least acidic of them is acetic acid.

Squaric acid > Formic acid > Acetic acid