Avg load goes up after upgrading to Mavericks
Finally found the culprit. I'm using zsh and the plugin vi-mode from Oh-my-zsh interacts poorly with git tools. Disabling the plugin made my zsh usable again in terms of responsiveness and reducing CPU load. 

There are conflicting reports whether Apple's git needs to be upgraded and you can follow that process at https://github.com/robbyrussell/oh-my-zsh/issues/2189

Still many thanks for all the people reply to the answer.