What is the downside of not pursuing postdoctoral fellowship position?
The purpose of a postdoctoral fellowship is simply to gain the skills detailed in this answer to a similar question, notably:


Get involved with obtaining funding 
Build an international reputation
Start collaborating with external parties
Learn how to manage projects and a lab
Start to devise a strategic research plan
Improve your publication record


If you have all those things from your work as a graduate student, then you should have no problem finding employment and furthering your academic career immediately after you earn your PhD. Most students don't, which is why they try to find a postdoc position to help them  gain these skills before attempting to run their own lab.