Is it right time to buy 4K video camera?
You will pay substantially more for a good 4k camera right now than you would for a good 1080p camera. Quality =/= resolution, there are phones that can shoot in 4k but the videos will look a lot worse than a 1080p video from a dedicated 1080p camcorder/dslr, the image quality depends on a lot of factors.
A bigger sensor for example has the ability to capture a lot more light and have much more accurate colors than a tiny sensor that only captures very few photons per pixel.
The video compression used by the camera is also a big factor, even with h264 there can be huge quality differences between cameras depending on the encoding settings of the camera.

The question you should also ask yourself, what do you get from a 4k resolution when you mostly shoot things that are meant to be memories that you don't already get with 1080p.

My advise would be to not think about 4k and look for a nice 1080p camera. Even in commercial video there is not that much to gain with 4k video right now and probably not even in the future, we are reaching resolutions that don't necessarily improve the image quality for every content. It really depends on what you shoot and most importantly how big your screen is and how far you sit away from it. Right now there are quite a few living room configurations where its already hard for a lot of people to tell the difference between 1080p and 720p.