Should I be worried if I missed the studs while mounting my TV?
Your TV is being supported by two pieces of paper. That should be all you need to know.

The white plaster material in the middle of drywall resists compression. The interaction between the shaft of the bolts and the inner plaster is keeping the load from shearing straight down the wall and will support a significant load.

However, on the front and back of the drywall are two sheets of paper that resists tension, similar to the purpose of rebar in concrete. And the interaction between the threads of the bolts and these two sheets of paper are all that are preventing your TV from being pulled out of the wall. Over time, the top bolts will pull out from vibration, temperature and humidity changes, and especially if you adjust the angle of the TV.

Once those top bolts start to pull through, the paper will tear, the plaster will shear out of the hole, and the TV will only be supported by the lower bolts under a highly leveraged load. Those lower bolts will immediately fail, and you'll have a broken TV on the floor at best, or a broken person that was hit by the TV at worst.

So, as everyone else has already said, remount the TV to proper structure.