How to get Panelizer to only affect a specific node
I have figured out my error. You need to go to Admin->Configuration->Content Authoring->Panelizer and make sure you don't have the Provide Default Panel option selected for the content type you are using Panelizer on.