Recent publication versus old, crappy letters of recommendation?
If you have done valuable work on open source projects, a recommendation from one of the lead developers would certainly count highly in my book, if you are applying to a related field (or software related in general). That you have never even met face to face should make no difference.

You say you did well at some classes you liked, presumably some of the people who taught you (preferably later courses) will write you "face to face based" letters, I wouldn't forego such totally.

For the record, I had some students around here (OK, still have) that do extremely well in non-directly study related stuff, and do so-so or directly terrible in their regular classes. I would recommend some of them highly, even if their grades aren't stellar. But probably with some warning about "careful, if work isn't interesting to them they might just drop it".