Restore MSDB data to a new server
I did that kind of migration and I never want to do it again. But if you really want to do it, it is feasable. Here's my recommandations:


Make sure you have the exact same version on both instance.

select SERVERPROPERTY('ProductVersion')
Migration will be way easier if the root folder is in the same path. If you can't have that, it's much more complicated and I strongly suggest taking another path. BTW I did that with SQL 2000. Now I think you can copy over the ROOT folder in one shot. Some files like FullText will be failing if the catalogs are not migrated and replication snapshot folder and etc... Take note of all the datafiles path with:

select * from master..sysaltfiles
If you can't have the user databases at the same location, no big deal. You'll attach them later.
Start the service and pray.


Now back to restoring MSDB and MODEL.
In your particular problem. Your last restore was done without recovery. The MSDB is inconsistant and can't be recovered. I suggest you restart SQL Server in Single User Mode (using -m startup parameter) then DROP MSDB and restore it again.