How to set custom DNS settings?
Did you follow this steps:

Mac OS X

DNS settings are specified in the Network window. 

Example: Changing DNS server settings on Mac OS 10.5

From the Apple menu, click System Preferences, then click Network. 
If the lock icon in the lower left-hand corner of the window is locked, click the icon to make changes, and when prompted to authenticate, enter your password.
Select the connection for which you want to configure Google Public DNS. For example:
    To change the settings for an Ethernet connection, select Built-In Ethernet, and click Advanced.
    To change the settings for a wireless connection, select WiFi, and click Advanced.
Select the DNS tab. (take note of the existing to be able to go back)
Click + to replace any listed addresses with, or add, the Google IP addresses at the top of the list:
    For IPv4: 8.8.8.8 and/or 8.8.4.4.
    For IPv6: 2001:4860:4860::8888 and/or 2001:4860:4860::8844
Click Apply and OK.
Test that your setup is working correctly; see Testing your new settings below.
Repeat the procedure for additional network connections you want to change.


To test that the Google DNS resolver is working:

From your browser, type in a hostname (such as http://www.google.com/). If it resolves correctly, bookmark the page, and try accessing the page from the bookmark. If both of these tests work, everything is working correctly. If not, go to step 2.
From your browser, type in a fixed IP address. You can use http://18.62.0.96/ (which points to the website http://www.eecs.mit.edu/) as the URL.* If this works correctly, bookmark the page, and try accessing the page from the bookmark. If these tests work (but step 1 fails), then there is a problem with your DNS configuration; check the steps above to make sure you have configured everything correctly. If these tests do not work, go to step 3.
Roll back the DNS changes you made and run the tests again. If the tests still do not work, then there is a problem with your network settings; contact your ISP or network administrator for assistance.


Possible limitations:

Routers

Every router uses a different user interface for configuring DNS server settings; we provide only a generic procedure below. For more information, please consult your router documentation.

Note: Some ISPs hard-code their DNS servers into the equipment they provide; if you are using such a device, you will not be able to configure it to use Google Public DNS. Instead, you can configure each of the computers connected to the router, as described above.