File field upload in modal/dialog
For sure this is due to some problem with your sever setting, contact your web hosting company and talk with them regarding the issue.

It could be due to ModSecurity that scans incoming http requests for any malicious scripts. There may be an issue with the script causing mod_security to stop the script. ModSecurity is an is an open sourceweb application firewall Apache module, which help in providing an external security layer that increases security, and help in preventing attacks.

In the event that mod_security is causing the error, you will need to contact tech support to have them look at the logs to see what is exactly causing mod_security to flag the uploaded file. You may need to Disable mod_security via cPanel Modsec Manager on your specific domain if there is no work around. Otherwise, tech support will need do this for you.

The other reason could be the size you are allow to upload that can be change in your server setting. It is also important to check apache max_execution_time.

If you have control over your server then you can try to change your php file

post_max_size = 128M upload_max_filesize = 32M max_execution_time = 5000
max_input_time = 5000
memory_limit = 1000M

Source

If not solved then Other Reason May be :
Check the .htaccess or settings.php for conflicting Code
Edit the .htaccess file to see if there are any overrides in the code. Some server configurations allow .htaccess settings to override your file upload size. Comment any code that controls the files upload size and save the changes.

The settings.php file also can hold overrides for the file upload size. Edit the settings.php directly on the server or you can change the settings through the Drupal dashboard by going to Administer > Site configuration > File uploads and updating the settings from there.