Does fly-by-wire permit smaller tail fins (area and thickness) for jetliners?
After much google-fu, for the 777 it's an astounding yes (that someone was right after all).
From a 1996 ICAS paper (see PDF pages 2, 4, and 5) about the 777's innovations, the use of relaxed stability and FBW laws indeed helped reduce the tail area and its weight:



The paper mentions a wheel/rudder crosstie. The wheel here being the yoke. Before the 777 Boeing sized the tail based on controlling an in-flight engine-out using only the yoke. This digital crosstie allowed the smaller fin:

Vertical Stabilizer. Conventional sizing of the vertical stabilizer is based primarily on meeting requirements associated with static directional stability. Sideslip characteristics and the pilot’s ability to control the effects of in-air engine failure using wheel only have sized the vertical stabilizers on previous Boeing airplanes. With the wheel/rudder crosstie, the same desired sideslip and engine-out control characteristics can be achieved with the smaller vertical stabilizer. The rudder input due to the crosstie compensates for the reduced area.


The even smaller fin of the quad-A345/6 remains open.
It turns out my guess had merit. The A340 does use the roll surfaces to supplement the yaw:

The flexibility of fly-by-wire was used to optimize the VMCG speeds. In fact, the rudder efficiency was increased on the ground by fully and asymmetrically deploying the inner and outer ailerons on the side of the pedal action as a function of the rudder travel: the inner aileron is commanded downwards and the outer aileron complemented by one spoiler is commanded upwards.$^1$

And watching the engine-out test video again, this becomes clearly visible:



Note: either the author mixed inner with outer, or Airbus updated it since or flipped them for the -5/600, either way, it doesn't change the answer.

$^1$: C. FAVRE (1994) Fly-by-wire for commercial aircraft: the Airbus experience, International Journal of Control, 59:1, 139-157, https://doi.org/10.1080/00207179408923072 p. 153