What is this electrical junction box? (re: adding 3-way switch)
That looks like a doorbell transformer. Those smaller wires should lead to your doorbell chime unit (assuming you have a doorbell). The junction box is there simply to provide power to the doorbell transformer.

From the picture it looks like you have 12/3 and 14/2 in that box, but it is difficult to tell. How confident are you the larger wires are larger than #12?

Figure out what breaker is supplying power to that circuit. As long as it is sized appropriately for the smallest conductors on the circuit, it will be fine. For example, if that is indeed a mix of #12 and #10 copper, your breaker must be 20A or smaller.

Also as a side note, NM cables entering that metal junction box need to be secured with a cable clamp. Any unused knockout holes need to be plugged. It looks like you've got a couple of code violations there at the top of your picture.