Does a positive electron affinity actually correspond to the release of energy?
It may help to take a look at the two processes you are comparing in terms of reactions. Ionisation follows the following reaction:

$$\ce{A -&gt;[][$\Delta E_\mathrm{i}$] A+ + e-}\tag{1}$$

This process involves tearing charges apart, separating a positive and a negative one. Even if the original atom was somewhat unstable, it will be unfavourable to remove that electron simply due to electrostatic interaction. You write in the question that you were able to deduce this already.

The electron affinity on the other hand describes the process in this following reaction:

$$\ce{A + e- -&gt;[][EA] A-}\tag{2}$$

Here, a charged particle is approaching a neutral particle and we have the same number of charged particles (and the same charges) on both sides of the equation. This means that we don’t have a strong, obvious physical effect that will make all electron affinities sign-equal for very simple reasons. Instead, we need to consider the details more.

The process often involves a release of energy because most atomic configurations have space for extra electrons at little or no extra cost. Except for the noble gases and the second group, atoms don’t have completely populated sub-shells so there should always be a position available in a subshell at an acceptable exothermic energy level. While adding an electron will increase electron-electron repulsion and increase all energy levels by a certain amount, this increase is typically less than the first ionisation energy which corresponds to the energy level of the highest occupied atomic orbital. Thus in electron affinity, we in principle gain the entire ionisation enthalpy minus the destabilising contribution of the new electron. (Caution! This is a very simplified picture and not suitable for generalisation or even strictly for explanation. It gets the basic point across though.)

Therefore, most electron affinities are exothermic with the notable exceptions of group 2, group 18 and nitrogen.