Conceptual question on entropy and its relation to information
The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article.

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

To find the density that maximizes the entropy we have to use the calculus of variations, and that involves determining several coefficients called Lagrange multipliers. This is where the moments come in: they're the constraints that determine the multipliers. We set the $n$'th absolute moment for n=1..N to constants and we get the corresponding maximum entropy distribution. For example, the maximum entropy distribution whose first two moments are $\mu$ and $\sigma^2$ is the Gaussian distribution $N(\mu, \sigma^2)$. For details of the derivation please refer to a textbook.

I haven't read the paper but I'm guessing the rationale is that if you're trying to fit a model, you want to capture as much information of the signal as possible. Equivalently, you want to leave as little residual information as possible on the table; i.e., in the error. For details see 3.4 Minimum error entropy algorithm in Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives.