Why/How I could remove index.php in the URL
I suggest you look at the .htaccess file supplied with Drupal, beside index.php. In the standard one, it has rules for both redirecting http: to https: (line ~104) and for rewriting URIs to index.php (line ~136)

Some servers are configured with these rules in the main config file because relying on .htaccess results in slower response from Apache (it reads the main files on startup but .htaccess per request).

So:
- Remove any additional rules you have added (i.e. revert to "out of the box") and check for expected behaviour;
- Uncomment the protossl conditions in .htaccess and check again;
- Optionally, move some of these into the .conf file for faster response.