What are the benefits of organizing a conference?
If everyone refused to organize conferences, we would have no conferences.

Yes, it's that obvious.  The same goes for editing journals, refereeing papers, and most other service activities.

The main result of a research conference is that it helps attendees do better research, to find and understand new ideas, and to disseminate their own research, etc.  By organizing the conference, you are the facilitator of all that.  Furthermore, you get to influence the direction of the conference -- what it will focus on, who the invited speakers will be, how it will be run, and so forth.

Additionally, I would argue that academics are paid to organize conferences in the same way that we are paid to referee papers.

Finally, there are fringe benefits to you personally, including the prestige mentioned by @user3209815 -- you can list it on your CV under service activities.  Frankly, this is not going to make or break your career, and you probably shouldn't be doing it early in your career, since (as you point out) it takes time away from things that may be more essential.