Does out-of-bag sampling make Random Forests inherently less robust than other classifiers?
The out-of-bag-error is calculated from the samples that are not used anyway for the particular tree. The original set of records gets bootstrapped. So, a new set is generated that does not contain all samples anyway. The out-of-bag set can then be used to monitor the performance of the model. When the out-of-bag-error goes up (given it has a significant size) it means that the current tree is over-fitting the training sample. So, out-of-bag sampling can be used to prevent over-fitting and therefore makes the model more robust rather than less robust. 

So, in fact all samples are used to train the random forest model. Though each tree only uses a subset of the dataset at a time. Don't confuse the RF model with the individual trees! 

Look here for a more detailed description.