Will removing a foreign key constraint affect performance?
At time of writing (PostgreSQL 9.4 and below), PostgreSQL's optimiser doesn't use foreign key constraints to prove anything or allow for extra optimisations. The only impact removing one can have on performance is a performance improvement due to the removal of foreign key checks.

It's a data integrity enforcement tool, not a performance tool.