Anime with house with girls with different hobbies?
That sounds like Sister Princess. It came out in the early 2000s and started with the main character, Wataru, on a boat going to an island with a big mansion. The mansion had Wataru's twelve younger sisters. One sister, Chikage, was a magician who had tarot cards and a crystal ball, and I believe also a broom, as you describe in the question. Another girl, I believe named RinRin, was a sports nut. There was also a girl who was a detective (Yotsuba), a girl who was a French noblewoman (Aria), a girl who was trendy (Sakuya), and several others.