What am I missing for my SSRS Service account local server permissions?
As part of the troubleshooting I was doing to try and address the issue I ran Process Monitor while I was trying to start the service. 

I tracked as event that also had a result of "Access Denied" which was the service trying to read files inside the directory where reporting services was installed an running from. In my case it was: "C:\Program Files\Microsoft SQL Server\MSRS10_50.MSSQLSERVER\Reporting Services"

I check the security of the folder and the service account I was using had no rights to the folder. That is why giving it local admin rights fixed it because that group did have access. 

I gave my service account Modify access to the folder and its contents. After that I was able to start the service.