Can I publish research in a peer-reviewed journal if parts of it have previously been published in a self published book?
I didn't find any formal policies addressing this specific case, but my intuition is that it would probably be all right.  It could be analogous to publishing sections of a thesis, which is generally accepted.

If you have a journal in mind for the paper, you could contact the editors and ask their position on the issue.  If and when you eventually submit the paper, you should include a note to the editor explaining the situation, and probably mention this in the text of the paper as well.  If you disclose everything, there won't be any ethical issue; the editor can make the decision, and I expect there would be some good journals that won't have a problem with it.

Factors that might be relevant:


Was the book offered for sale publicly, or distributed privately among colleagues?
How many copies were produced/sold?
Did you retain the copyright on the book?