HM-10 only works when holding button
I just had to connect the RX/TX jumpers. There was no real issue.

I though connecting the power jumpers was enough to just see HM-10 on my phone, since I didn't care about any data. Turns out if the RX/TX jumpers are not connect, the module behaves weird.