When is a first-order formula is existential and when is it universal?
The key to the usage is to transform the formula to an equivalent formula that is in prenex form.

A formula $F$ is in prenex form if $F \equiv Q_1 x_1 ...Q_m x_m\; G$ where $G$ is a quantifier free formula and $Q_1,...,Q_m$ are quantifiers. 

A formula $F$ is universal if it is equivalent to a formula in prenex form that has only universal quantifiers.

There is an important model theoretic property of universal formulas: a universal formula that is satisfiable in some structure $S$ is also satisfiable in any substructure $P$ of $S$.


  Why is $\forall x \exists y(X[xy]∨X[yx])$ universal and $\exists x \forall y(X[xy]∨X[yx])$ existential? I mean for the second formula all $y$ are within the scope of $\forall y$, so shouldn't it be universal as well?


Either sentence is neither universal nor existential. One can easily come up with an example structure where $F\equiv\forall x \exists y(X[xy]∨X[yx])$ is true but is not true for a substructure.


   Let $S=\{0,1\}$ and $X[xy]\equiv x\not=y$, then F is true in $S$ but is not true for any one-element substructure of $S$.


Now for the second question


  And if you have a formula like this: $H\equiv(\exists x X[xy])⊙X[yx]$, where ⊙ denotes an arbitrary binary connective, would this be a binary connective formula or an existential formula?


In order to find the prenex form we rename the variable $x$ in the scope of the existential quantifier to get $H'\equiv(\exists z X[zy])⊙X[yx]$.

Now suppose $⊙$ is disjunction or conjunction then we can consider an equivalent formula $H''\equiv\exists z X[zy]⊙X[yx]$ that is in prenex form and is an existential formula. 

However if $⊙$ were an implication then the formula $H$ would be equivalent to $H'''\equiv\forall z X[zy] \Rightarrow X[yx]$. The formula $H'''$ is clearly a universal formula.