How do I ensure consistent MySQL backups? How can we ensure MySQL backups are consistent when they are taken as full backups as well as incremental backups using binlogs? 

According to me, I record the COUNT(*) from random tables after issuing flush logs on the database.  I then take binlogs generated until it flushes the binary file and ensuring the table is not altered while doing SELECT COUNT(*).  I then restore the binary logs in different database server and cross check COUNT(*) to ensure the random tables are the same or not against production database and restored database.

Any other thoughts?  Keep in mind that hits from application to tables can be paused only for a few minutes.
Are you using innodb? If so, Percona's XtraBackup is a great choice for automating backups while keeping the server operational and responsive. It also supports incremental backups to shorten the backup duration.

Another option is to have another database that is a replication slave and you backup this slave. You can shut down replication, backup, and resume, never impacting the production data.

I personally use a combination of the two solutions. We have a backup sync and we use xtrabackup to backup this server, thus allowing us consistant backups on demand without impacting production load.