Silicon–Boron Covalent Bond
Using your diagram, the boron-silicon bond with only the silicon electron has only half of the possible electron content.  A &quot;hole&quot; as you said. The hole is not confined to the one $\ce{B-Si}$ bond, but can be in any of the 4 positions. It's not like the boron sits there and the silicon pops its electron into the potential bond; the bond is there and it's only half filled.
Now this isn't a molecule, but you can focus on the boron part of the crystal and say that all four adjacent silicons are bonded to the boron, but you have only 7 electrons where the &quot;molecular&quot; orbitals could contain 8.
Electrons don't really &quot;belong&quot; to one atom or another; they comprise some sort of sea of electrons, and the sea above this boron is a little lower than elsewhere. I think of the sea of electrons in a copper-boron alloy as being very fluid, and having a level surface everywhere, but the sea in a p-type semiconductor is more like mud (quicksand?) and can have some dips that could catch and puddle some water (~electrons). The sea is still controlled mostly by the silicon, just modified by boron (submerged holes).