how much overhead an error in RDBMS has? I have a class in my project. On the class I need to do two inner select, to know is there any duplication or not, but I think if I get duplication error and then manage that is better than to select.
Error overhead Vs. Two inner select, which one?
If the unique field is properly indexed, a UNIQUE violation should be fairly quick.

Honestly though, I think doing a "two inner select" (I think I know what you mean here) is a better solution.  Even with trapping for an exception (at the application level), that's just not a very clean way to handle programatic logic.  Exception handling should be reserved for those rare occasions where you have an "exception to the rule."

And (without knowing which language you're coding in) if you're just trapping for a "SQLException", it's important to remember that those can be thrown for a variety of reasons. Your application might produce unexpected results if you incorrectly assume why an exception happened.

Bottom-line, I would say that the best approach is to do the SELECT to check for duplication.