Can I just restore the .diff file over full backup to speed up restore?
Yes, it is possible.
You should restore full backup first WITH NORECOVERY  option, and after that, you can restore only DIFF backup.
Just try without FULL backup;
RESTORE DATABASE [MyDb] FROM  DISK = N'C:\db\diff-final.bak' WITH  FILE = 1,  RECOVERY,  NOUNLOAD,  STATS = 5


RESTORE WITH NORECOVERY Leaves the database in the restoring state. This allows you to restore additional backups in the current recovery
path. To recover the database, you will have to perform a restore
operation by using the RESTORE WITH RECOVERY option (see the preceding
option).

Check out the document.