Why might a parallel pg_restore command take hours longer to complete its non-parallel equivalent?
It's probably just database contention. 

Doing a parallel dump, you postgres can write to many different files at once, so the load can be spread around. 

Doing a parallel restore, postgres has to manage all that [heavy-weight] traffic coming in from jobs+1 connections, all filling up data pages in the buffer cache and writing them to disk, all at once.  To create indexes, you need to read data pages back in and scan them; doing this for lots of tables at once puts a lot of pressure on the buffer cache ...   more work for postgres.  

Also, you've specified four jobs - does your database [server] have enough CPUs / cores to support that?  If not then the extra jobs will still get spun up, but will suffer contention at the operating system level, as they get switched in an out of memory as the [poor] machine tries to run them all "at the same time". 
On smaller servers, it may be that you get [some] benefit from running with two jobs, but more than that and the benefits are rapidly outweighed by the cost.