Mycobacteria grow and divide asymmetrically, creating variability in growth pole age, growth properties, and antibiotic susceptibilities. Here, we investigate the importance of growth pole age and other growth properties in determining the spectrum of responses of Mycobacterium smegmatis to challenge with rifampicin. We used a combination of live-cell microscopy and modeling to prospectively identify subpopulations with altered rifampicin susceptibility. We found two subpopulations that had increased susceptibility. At the initiation of treatment, susceptible cells were either small and at early stages of the cell cycle, or large and in later stages of their cell cycle. In contrast to this temporal window of susceptibility, tolerance was associated with factors inherited at division: long birth length and mature growth poles. Thus, rifampicin response is complex and due to a combination of differences established from both asymmetric division and the timing of treatment relative to cell birth.