Is verbal commitment to work with a potential advisor binding in a moral sense?
A few points:

First, morality is a bad way to evaluate this decision. Certainly you want to be an ethical person, but realize that taking on a graduate student position is an employer/employee relationship. It is generally expected that you will make decisions that are in your best interest, even if that means switching advisors or switching programs. Your potential advisors are certainly going to make the decisions that are in their best interests. Your advisor/advisee relationship should only last as long as it's in both of your best interests to do so.

Second, there are definite consequences to reneging on an agreement, written or verbal. Academics is a small community. You are embarking on a professional career, and your reputation will follow you. You want to be seen as someone who follows through on commitments. When you do need to back off of an agreement you want to do so gracefully and avoid "burning bridges". That said, your actions in your current position are not going to follow you for the rest of your life. Your potential PI has not sacrificed for you, they will be able to find another grad student, and you leaving them at this time is not going to ruin their career. This is very different than, for example, working with an advisor for years, building expectations and relationships, and then vanishing.

However, if your decision is firm you should definitely inform them as soon as possible. The pool of good applicants grows smaller the longer you wait. 

Third, and perhaps most importantly, there are community expectations and norms concerning this. The most relevant thing here is the April 15th Resolution. Many graduate schools are signatories to this document (including most or all of the highly desirable "role-model" institutions). Essentially, the resolution says:


Prospective students should not be obligated to commit to a graduate school until April 15th.
Students are only considered committed once they have formally accepted an offer of financial support. This means formal acceptance of a scholarship or stipend supported position such as a research or teaching assistantship. A verbal commitment to attend, without a specific formal offer of financial support, is not considered formal acceptance.
Students may withdraw their acceptance until April 15th by submitting a written resignation to their prospective department. Appointments not recinded by the department or resigned by the student by April 15th are expected to be honored by both parties.


Lastly, the goal of the graduate admissions process is to make the best matches between students and schools. You are doing yourself and your potential advisor a disservice if you go into a program without full conviction. Your potential advisor certainly will not like the news that you've decided to go someplace else or study a different topic, but they will be far happier with a student who really wants to be with them and study their topic. You will be far happier if you are a place you really want to be and studying something that truly engages you. Certainly they will understand your actions and will (probably) not hold a grudge if you secure a better position than what they can offer.