Extruder prints fine up until further down the print
I used a different heat break with a PFTE lining inside. After that it ran smoothly. 

Probably what was happening was that my filament was getting too hot and started to stick on the full metal heat break.