Can .mp4 files infect my Mac with malware/adware/spyware?
Exploits through any other type of file than a .dmg disk file are rare on macOS. To harm a Mac with any other file type than an application, said file (be it .mp4, .mp3, .pdf, .png, .jpg, etc.) would have to exploit a vulnerability in the operating system or media player. If you are using a standard video player (like the latest version of QuickTime) you'll be almost immune to any malware-laced .mp4 files that probably don't exist anyway.

No hacker in their right mind would lace a video file with malware: it's just too hard to get it to work across multiple video players and operating systems, and it's so much easier to get malware on a user's machine using .dmg files. So I highly doubt a .mp4 file would contain malware that would affect your Mac, which would almost certainly be immune to it anyway with the latest operating system and media player. 

So without even answering your follow-up questions yet, you are safe. If you want to feel even safer (and truly ensure that your Mac is unaffected), you can download an antivirus app like Avast or BitDefender as I explain in this answer.

To answer your follow-up questions: if this "video" file was really an application file, your Mac would notify you that it's an application from an unknown developer, force you to go to System Preferences to allow installations of unknown applications, and insist that you type in an administrator's password before allowing this application to run any code. 

So even if this was an application file (.dmg) masquerading as an audio or video file, you'd still be safe. Read more about how you're still safe even if you downloaded a malicious application file (so long as you don't manually install it) in this answer.

tl;dr: you're not going to get  malware from downloading a video or audio file.



Edit 1: A notable exception to this "a file can't harm your Mac unless it's an application" rule is the Word .docx macro. Word documents (and documents for Excel and the rest of Open Office, thanks WGroleau) can try to convince you to run them as a macro, essentially turning them into mini-applications. These can be malware, so never run a downloaded Word document as a macro.  Read more about Word macro malware.

Edit 2: Email attachments can look like .mp3 files or .jpg files, but actually be .dmg files disguised as such. Whereas when you download a file from the web you'll get a prompt that lets you know what file type you're about to download, you may have no such warning when opening an email attachment. Thus, don't open a suspicious email attachment even if it claims to be a .mp4/.mp3/.jpg/.png/.pdf format thinking it's necessarily safe.

Edit 3: If you don't want to install an antivirus app, you can use the website virustotal.com to upload a file and have multiple antivirus engines scan it at once, in the cloud.

Woah, so many edits here.