Is it acceptable for authors of papers in a special issue of a journal to review each others' papers?
In general I would say it would be very poor judgement to use only authors from the special issue as reviewers. But, if the theme is such that they would constitute the best choice for reviewers it might be difficult to avoid. The best way, in my opinion, to try to circumvent the problem is to use one reviewer from the group and one external. this way it is possible for an editor to identify issues.

One problem with the scenario you paint is that it seems you suggest the editor is more trustworthy than the authors, or conversely that authors would be worse than the editor. Although I am sure some special issues may suffer from problems such as those you describe, most scientists, editors and journals want high quality work, not sloppy work and so the process is to some extent self-correcting.

In terms of publishing, the term conflict of interest is usually not considered other than in situations where, for example, an author has received help from a person who later is requested to be a reviewer or where the author feels that a certain person will not provide a fair assessment of the work. Usually such conflicts will be flagged by the people involved since the editor is in no way positioned to know about all such potential problems. It is thus not difficult to trick the system but uncritical reviews are usually quite easy to see through.