Will I look bad if I'm never in my office as a graduate student?
Look bad, not necessarily. Will it be good for you? Maybe not.

Assuming that you don't have a really significant reason not to be on the lab, you should be. Stuff only happens with people that show up. If you are not in the lab, you won't talk to people as much. I literally can't count how many good ideas (and papers) came out of random coffee talk, that wouldn't happen on an otherwise "official" meeting.

full disclosure: That's something I didn't do during my phd and I regret it. I had my reasons, probably not really good enough, but I was almost never at my office and that hindered my development. I could have done more if I was there.