Do Penicillin based antibiotics affect birth control?
Evidence on the association between antibiotic use and combination oral contraceptive (COC) failure remains controversial, with recent studies reporting no evidence to support decreased effectiveness of birth control with the use of antibiotics except rifampin and rifabutin.
The reasoning is antibiotics might interfere with the effectiveness of COCs by decreasing steroid hormone’s plasma concentrations by hepatic microsomal enzyme induction or inhibition, interference with enterohepatic circulation of
COC metabolites, interference with absorption from the GI tract, competition between two drugs for the same metabolizing enzyme, alterations in plasma protein binding, induction of an opposite physiologic effect, or increased urinary or fecal excretion of the contraceptive.
As you stated, the strongest evidence is for rifabutin and rifampin: a significant decrease in the hormonal levels was noted in women taking rifampin even after a single dose.  Dirithromycin slightly decreased plasma ethinyl estradiol levels, with questionable clinical importance. A recent study in the Netherlands did find a relationship between the use of antibiotics and breakthrough pregnancy in a population-based prescription database; also, individual patients do show large decreases in the plasma concentrations of ethinyl estradiol when they take certain antibiotics, notably tetracycline and penicillin derivatives.
Because of earlier studies, anecdotal reports, and the above (it's not possible to identify whose levels will drop with antibiotic use), a cautious approach is advised. Physicians and pharmacists (80-90% of both) still lean towards believing that broad-spectrum antibiotics decrease the effectiveness of COCs, and continue to advise the use of back-up contraception.
The tide is shifting, however, as new studies come out. Here are a few excerpts of recent studies:

Available scientific and pharmacokinetic data do not support the hypothesis that antibiotics (with the exception of rifampin) lower the contraceptive efficacy of oral contraceptives. (J Am Acad Dermatol 2002;46:917-23.)

Since dermatologists treat acne with antibiotics, and a significant percentage of these patients are young women of child-bearing age, they certainly have a vested interest in knowing if this is true.

Rifampicin and griseofulvin induce hepatic enzymes and do appear to have a genuine interaction with the COCP, leading to reduced efficacy. The situation with the broad-spectrum antibiotics is less clear. There are relatively few prospective studies of the pharmacokinetics of concurrent COCP and antibiotic use and few, if any, demonstrate a convincing basis for any reduced contraceptive efficacy.
We did not find an association between concomitant antibiotic use and the risk of breakthrough pregnancy among COC users. However, due to limited power and potential carryover effects, findings from this study cannot rule out an elevated risk of COC failure among antibiotic users. (2011)

So, the answer is (except with your noted exceptions) the penicillin-class antibiotics probably do not interfere with effectiveness of COCs. However, one unwanted pregnancy is one pregnancy too many, and the use of a back-up method is not a huge deal (just ~14 days).
Until more and larger studies demonstrate a lack of interaction between most antibiotics and COCs, this will likely be the advice people continue to receive. When you look at the risk to benefit ratio of back-up, it's probably the wisest course for people prescribed a short course of antibiotics. It is more significantly problematic for people prescribed continuous antibiotic use for acne, Crohn's and other conditions/illnesses.
Oral contraceptive efficacy and antibiotic interaction: A myth debunked
Interaction between broad-spectrum antibiotics and the combined oral contraceptive pill: A literature review
Antibiotics and oral contraceptive failure — a case-crossover study
Are antibiotics related to oral combination contraceptive failures in the Netherlands? A case-crossover study
Drug interactions between oral contraceptives and antibiotics.
Survey of pharmacists and physicians on drug interactions between combined oral contraceptives and broad-spectrum antibiotics.