Did the Buddha ever talk about his family?
The Buddha praised princess Yashodara's virtue by dispensing the Channakinnara Jataka when he visited her at Kapilavatthu. Yashodara has never been a wife to another man since the life when Bodhisatva received the confirmation from Deepankara Buddha to become a Buddha in the future. The Bodhisatva was always her husband.

So the Bodhisatva and his family have spent plenty of time together as family in many a previous life. If that didn't take away their suffering, what's the point of doing more of the same?

Lets take an analogy. If your family is suffering from a mortal illness and if you alone have the capability to go find the cure, wouldn't you leave them behind to get the cure instead of lingering there to give them false comfort?