Doubts about PhD advisor and unsure how to continue
It's your project, not your advisor's.  Your lack of progress/motivation and supposed lack of time have nothing to do with him.
Nevertheless, you must sit down with your advisor and have a serious talk about both of your expectations. Be honest about your lack of motivation, be honest about your disappointment, be honest about your intellectual loneliness.  Ask for suggestions. Listen.
You have other senior collaborators that you can work with, so if you need senior collaborators, work with them. Ask your advisor to fly them out to visit you, or to fly you out to visit them.  Schedule regular Skype/Gchat meetings. Similarly, get to know other faculty in your department, who might serve as more experienced mentors/advisors, if not collaborators.
You're surrounded by colleagues who are excited enough about their research to want to draw you in, and you turn them away because you feel stupid?  What a wasted opportunity!  Of course you feel stupid.  Everybody feels stupid listening to other people talk about their ongoing research, because by definition, they're talking about stuff that nobody understands (yet).  You're simultaneously complaining about intellectual loneliness and turning away potential collaborators.  So what if they're not working on your project?
Of course you're constantly forgetting important things you once knew well.  It is totally unrealistic to think that you can keep everything you've ever learned/read fresh in your mind. Don't even try. Write things down. Maintain a library of papers, or at least a close personal friendship with Google Scholar. Let yourself forget details, confident in the knowledge that they'll come back amazingly quickly when you read them again.
Since your feeling is shared by other students in your group, it sounds like your advisor is a bit overexcited and has bitten off more than he can chew. All the more reason to sit down and talk with him directly, and to develop mentoring relationships with other faculty in your department and elsewhere. Also, start looking into other potential advisors, even if it means moving to another university.