Cycles Combined bake black spots
The problem was my high poly mesh being in the same scene. Even though it was hidden it botched the baking. After moving the high poly  to a different scene the problem disappeared.