Increasing PHP memory limit (Possible Drupal override?) Problem

I've recently come across a problem with my dev version of Drupal where I realised the php memory limit isn't high enough.

I've changed the main php.ini files but they seem to be ignored and I can only increase the memory limit by putting the following line into sites/default/settings.php:

ini_set('memory_limit', '500M');


Could there be something Drupal related which is overriding the main settings (that I'm then overriding in settings.php) as this is happening in multiple environments?

Set-Up


Windows 7, XAMPP:


Altered php.ini in C:/XAMPP/php/php.ini

memory_limit=500M



Ubuntu(10.04), Apache, PHP, MySQL


Altered php.ini in /etc/php5/apache2/php.ini

memory_limit=500M



Ubuntu(12.04), nginx, PHP-FastCGI, MySQL


Altered php.ini in /etc/php5/cli/php.ini
(checked this was correct ini file using php --ini)

memory_limit=500M


I have a php.ini in my drupal root, but it has

memory_limit = -1


which I'm pretty sure is no limit.
Drupal does not edit any php config settings. The memory_limit and what php.ini it is using depends on how your server is setup, there could be something else that is overriding your php.ini you are trying to set. You should use phpinfo() to find out what php.ini it is using and make sure the last occurrence of memory_limit is set to what ever you want.