How to change someone's beliefs? I tried to change my brother's beliefs. He gets motivated by some bad ideas. Following my attempt to change his beliefs, his beliefs have become even stronger.
It's a little unclear to me from what you've said that your efforts are truly affecting your brother's ideas. It's also unclear whether those ideas are objectively bad, or whether you should be trying to change them. In general, we have to be careful to avoid recommending specific actions for specific people here.

That being said, the general phenomenon you seem to describe is reactance. Excerpts from Wikipedia:


  Reactance is a motivational reaction to offers, persons, rules, or regulations that threaten or eliminate specific behavioral freedoms [(Brehm, 1966; Brehm &amp; Brehm, 1981)]. Reactance occurs when a person feels that someone or something is taking away his or her choices or limiting the range of alternatives.
  
  Reactance can occur when someone is heavily pressured to accept a certain view or attitude. Reactance can cause the person to adopt or strengthen a view or attitude that is contrary to what was intended, and also increases resistance to persuasion. People using reverse psychology are playing on at least an informal awareness of reactance, attempting to influence someone to choose the opposite of what they request...
  
  According to William R. Miller [(2000),] "Research demonstrates that a counselor can drive resistance (denial) levels up and down dramatically according to his or her personal counseling style". Use of a "respectful, reflective approach" described in motivational interviewing and applied as motivation enhancement therapy, rather than by argumentation, the accusation of "being in denial", and direct confrontations, lead to the motivation to change and avoid the resistance and denial, or reactance, elicited by strong direct confrontation [(Miller &amp; Rollnick, 1991)]...
  
  Silvia's 2005 study Deflecting reactance: The role of similarity in increasing compliance and reducing resistance concluded that one way to increase the activity of a threatened freedom is to censor it, or provide a threatening message toward the activity. In turn a "boomerang effect" occurs, in which people choose forbidden alternatives. This study also shows that social influence has better results when it does not threaten one's core freedoms. Two concepts revealed in this study are that a communicator may be able to increase the positive force towards compliance by increasing their credibility, and that increasing the positive communication force and decreasing the negative communication force simultaneously should increase compliance [(Silvia, 2005)].


You may want to follow some of the hyperlinks above. They're all useful concepts for this issue, and may help you understand what you're doing and what your alternatives are. As a general principle, counselors and therapists avoid trying to change others' beliefs and attitudes explicitly and directly. Instead, they focus on understanding their clients, and in the process, they prompt their clients to reexamine themselves and find their own reasons to change. It's subtle yet impressive social jujutsu.

References


- Brehm, J. W. (1966). A theory of psychological reactance. Academic Press.
- Brehm, S. S., &amp; Brehm, J. W. (1981). Psychological reactance: A theory of freedom and control. Academic Press.
- Miller, W. R. (2000) Motivational enhancement therapy: Description of counseling approach. In J. J. Boren, L. S. Onken, &amp; K. M. Carroll (Eds.), Approaches to Drug Abuse Counseling (pp. 89–93). National Institute on Drug Abuse.
- Miller, W.R. &amp; Rollnick, S. (1991). Motivational interviewing: Preparing people to change addictive behavior. New York: Guilford Press.
- Silvia, P. J. (2005). Deflecting reactance: The role of similarity in increasing compliance and reducing resistance. Basic and Applied Social Psychology, 27, 277–284. Retrieved from http://www.niu.edu/user/tj0bjs1/psyc624/Silvia%20%282005%29.pdf.