Game theory with rational and irrational players
Yes, a whole book has been written on Behavioral Game Theory. 

More specifically, standard solution concept such as Nash equilibrium requires that players best respond to a correct belief about other players' moves. The following are examples that relax one of these cognitive restrictions:


Quantal response equilibrium allows for the possibility that players do not always choose a best response. Instead, they "quantal respond" (or best respond with mistakes) to a correct belief about others' quantal response. 
Level-k or Cognitive Hierarchy Theory relaxes the correctness of beliefs assumption, and allows players to best respond to a set of mutually inconsistent beliefs. 
Psychological games allow players to derive utility directly from beliefs (and belief-dependent emotions). Relevant solution concept include sequential reciprocity equilibrium and Rabin's fairness equilibrium. 


For a complete review of models of behavioral game theory, see Chapter 13 of Dhami (2016).

For a specific example of a game with mixed players, see this paper on robust protection of fisheries. A more comprehensive treatment of mixed games is available in Rani Spiegler's Bounded Rationality and Industrial Organization per Michael Greinecker's comment on the original question.