Malfunctioning atmega32u4 split keyboard -- help diagnose
"Intermittently" is the hardest type of error to troubleshoot in any technology, be it hardware or software. The Best advice I can offer is to try to replace the word intermittently with some specific conditions. You said that reconnecting the TRRS connection clears the issue for a while. Presumably the issue resurfaces.

There are two things that reseating a connection can do. One is hardware/electrically based. For example, opening or closing a circuit and discharging it. The other is software based and could for example trigger a cache/buffer clear. I would personally, begin at this connection and see if I can isolate hardware culprit. Did you double check the component locations and make sure you didn't accidentally swap out 2 similarly looking components?

I would put that connection on a scope and record the signal so you can see any electrical changes when the issue occurs.  If you do see something, you could have a component that is acting out of the specified range prescribed for it. All component hardware has a range of performance factors.  While a resistor might say it's 10k ohms it might have a variance of 1000-2000 ohms  while another from a different manufacturer could be +- 5000. A component variance too far outside the design of a circuit can cause unwanted behavior. It could similar, for example, to what you might see in a circuit with batteries that are on the edge of being "dead". This is why I asked about double checking the assembly instructions. It would help if you had the whitepapers handy for the electronics(Both from the Kit and also the component's manufacturer) if you end up going down this road. 

If the issue is software, it may be harder to determine, due to the nature of the device. Depending on what you used to flash the firmware and whether you can actively debug and look at the values of the variables being monitored in real time with an IDE, you may be able to watch the function that interprets key presses and see what signal it is actually receiving and what it is choosing to do with them. You could for instance narrow down your list of culprits  if you can say that the issue only appears or appears more frequently when a specific key is used and the more often it is triggered the quicker your problem manifests. The software route can also aid in hardware troubleshooting especially if you don't have an oscilloscope available.  

I wish I could offer you something  more specific but intermittent issues are the single hardest thing to diagnose and repair and need to be reclassified into something you can reproduce at will first if visual inspections fail to locate the issue.