Does the fact a table is big can impact the overall performance of a PostgreSQL server?
No, this shouldn't affect the speed of queries for other tables.
One way it could indirectly affect the speed is, if you regularly run queries against that big table and retrieve many rows from there (either directly or indirectly e.g. because of a Seq Scan). This could then cause data from other (smaller) tables to be evicted from the cache.
Then a query using one of the other tables needs to retrieve the data again from the hard disk which of course is slower than from cache.