Which agency issues private pilot certificates?
Here in the USA the FAA is the agency that issues all levels of Pilots Certificates. However the certificate is not simply applied for, it requires training and the passage of a written, oral, and practical exam. You can find the current requirements here.

For reference the NTSB investigates accidents (all transportation accidents for that matter) 

And ICAO is a UN organization that deals with coordinating aviation related maters all over the globe.