Is there any research on Turing machines with transition relation homomorphic to given algebraic structure?
There are results about algebraic properties of languages (as opposed to the machines accepting the languages). For example, regular languages can be characterised using monoids. The Myhill-Nerode theorem relates an algebraic structure with the transition relation of a specific class of finite automata. You could look up those and subsequent results. 

Rather than homomorphisms, bisimulation and simulation relations are the notion of choice for relating the structure of transition systems. Checking language equivalence between pushdown automata is undecidable but deciding equivalence of languages accepted by deterministic PDAs was open until 2001 when Gerard Sénizergues showed it was decidable. His proof uses algebraic ideas. Colin Stirling later gave a simplified proof (writeups and slides are available on his homepage). 


  Sénizergues, Géraud, The equivalence problem for deterministic pushdown automata is decidable, Degano, Pierpaolo (ed.) et al., Automata, languages and programming. 24th international colloquium, ICALP ’97, Bologna, Italy, July 7--11, 1997. Proceedings. Berlin: Springer-Verlag (ISBN 978-3-540-63165-1/pbk; 978-3-540-69194-5/ebook). Lecture Notes in Computer Science 1256, 671-681 (1997). ZBL1401.68168.