What exactly does PostgreSQL (or other databases) do internally when you "prepare" a query rather than just running it directly?
Actually, the difference between a parameterized statement and a prepared statement is not that big in PostgreSQL: the former is internally an unnamed prepared statement, and the main difference is that it is automatically deallocated when the next unnamed prepared statement comes along.
When you prepare a statement, PostgreSQL remembers that statement in your current session (nothing is shared between sessions).
For the first five executions, PostgreSQL will generate a custom query plan that takes the parameter values into account. After that, PostgreSQL will generate a generic plan that is independent of the query parameters, and if that generic plan is estimated to be no more expensive than the previous custom plans, it will be used from that point on.
For insert statement like you describe, PostgreSQL will always switch to the generic plan. The main advantage there is that the query doesn't have to be planned any more. With simple statements, planning time can exceed execution time, so the savings can be considerable.
PostgreSQL only caches query plans for prepared statements and statements in PL/pgSQL functions.
The other advantage of prepared (or parameterized) statements is that they avoid the danger of SQL injection.
If you have to insert a lot of rows, using COPY would be even faster.