What are the downsides of using PHP Filter code in blocks, nodes, views-args, etc?
Some reasons:


Code in your database can not be version controlled and is also harder to find in general later on.
Eval()'d code is much slower than something hardcoded in a file.
If there is an error somewhere in that code, you will get a very unhelpful error message (error in eval()'d code at line 3) and you might even have go through your database manually to find and fix the error. If it is inside a block that is displayed on all pages and results in a fatal error all the time for example.
The above is also true when upgrading from Drupal 6 to 7 and whatever API's you used were changed. So you will have to port your code while migrating. If the code is in a module, you can port it in advance, test it, and only deploy it on the new site. Inside a node or block, it will only work with either Drupal 6 or 7.
Writing and maintaing that code is also harder, because you are working in a textfield inside your browser. Having it in a module allows you to use an editor/IDE with syntax highlighting, autocomplete and so on.
There is always the possibility of a misconfiguration that gives people access to a text format/block/whatever with php execution enabled. If php.module (in D7, D6 is not so strict, for example for block access rules) isn't even enabled, that risk is much lower already.
If your CMS allows execution of PHP then an attacker who finds a security vulnerability of XSS or privilege escalation can now use your server for enormously malicious things (as part of a DDOS, sending spam, hosting malware, hacking into other sites/databases on the server, hacking into other servers on the network that might be behind firewalls). In addition to making small vulnerabilities more painful, this makes the site a more likely target of attack if its known that it can be used to execute php.


There might be more reasons, but that should be enough :)