Is that edge orientation optimization problem NP-hard?
We answer OP's last question: can an approximate solution to IQP be obtained by randomized rounding?
We show that the natural randomized-rounding scheme gives a 2-approximation, and a $(1+1/\overline d)$-approximation in graphs with average degree $\overline d$. (Note that any connected graph has average degree at least $2-2/|V|$.)
For the record, here's the rounding scheme. Given a fractional solution $x$ for QP, define 0/1 solution $x'$ as follows: for each edge $(u,w)$ independently: take $x'_{uw} = 1$ and $x'_{wu}=0$ with probability $x_{uw}$; otherwise take $x'_{uw} = 0$ and $x'_{wu} = 1$.
Let $f(x)$ be the objective function of QP.
Lemma 1. For any feasible $x$,
$$E[f(x')] = f(x) + 2\sum_{(u,w)\in E} x_{uw}x_{wu} \le f(x) + |E|/2.$$
Proof. This is a standard calculation.  Note that
$$f(x) = \sum_{u\in V} \Big(
\sum_{w\in N(u)} x^2_{uw} + 2\sum_{w'\in N(u), w'\ne w} x_{uw}x_{uw'}\Big)$$
and by linearity of expectation, the independence of $x'_{uw}$ and $x'_{uw'}$, and $E[x'^2_{uw}] = x_{uw}$,
$$E[f(x')] = \sum_{u\in V} \Big(
\sum_{w\in N(u)} x_{uw} + 2\sum_{w'\in N(u), w'\ne w} x_{uw}x_{uw'}\Big).$$
Using this and $x_{uw} = 1 - x_{wu}$,
$$
E[f(x')] - f(x) 
= \sum_{(u,w)\in E} x_{uw} - x^2_{uw} + x_{wu} - x^2_{wu}
= 2\sum_{(u,w)\in E} x_{uw} x_{wu}.$$
Lemma 2. For any feasible $x$, $E[f(x')] \le 2 f(x)$. That is, the rounding scheme gives a 2-approximation.
Proof.
By inspection of the expansion of $f(x)$ in the proof of Lemma 1,
$$f(x) \ge \sum_{(u,w)\in E} x^2_{uw} + x^2_{wu}.$$
Applying $p^2 + (1-p)^2 \ge 2p(1-p)$ for $p\in[0,1]$ (with $p=x_{uw}$)
to each term gives
$$f(x) \ge 2\sum_{(u,w)\in E} x_{uw}x_{wu}.$$
This and Lemma 1 imply $E[f(x')] \le f(x) + f(x) = 2 f(x)$. $~~~\Box$
The bound is tight in that for any graph with maximum degree 1,
the fractional solution $x_{uw} = x_{wu} = 1/2$ obtains value $|E|/2$,
but any integer solution has value $|E|$.
(Also note that $|E|$ is a lower bound on OPT,
using $\sum_{u\in V} d^2_{out}(v) \ge \sum_{u\in V} d_{out}(v) = |E|$.)
Lemma 3. In any graph with average degree $\overline d = 2|E|/|V|$, the rounding scheme gives at least a $(1+1/\overline d)$-approximation.
Proof.  Fix an optimal solution. Let $d'_v$ denote the out-degree of vertex $v$ in the optimal solution,
so $d'$ is a vector in $\mathbb R_+^{|V|}$.
Let $\mathbf 1$ denote the all-ones vector in $\mathbb R^{|V|}$.
By Cauchy-Schwarz,
$$\text{OPT} = d' \cdot d' 
\ge (d' \cdot \mathbf 1)^2/(\mathbf 1\cdot \mathbf 1) 
= \textstyle \big(\sum_v d'_v\big)^2/|V| 
= |E|^2/|V| = |E|\overline d/2.$$
By this and Lemma 1
$$E[f(x')] \le f(x) + |E|/2 \le f(x) + \text{OPT} / \overline d \le \text{OPT}(1+1/\overline d).$$
$~~~\Box$