If all B cells are present at birth, why should the primary response to an infection take longer than the secondary response?
These so called virgin B-cells (becase they never had contact with their antigen) which are present in your body only have random B-cell receptors (when secreted, these are the antibodies), mostly of a relatively low binding affinity. These B-cells undergo division, re-arrangement and somatic hypermutation of the B-cell receptor and clonal selection. Only cells producing high-affinity antibodies survive this selection process and build later the pool for the memory cells and the plasma cells. It is like in the image below (from here):



Using this approach we have a big pool of low affinity antibodies (which somewhat fit). These are then used as the basis for the elaboration of a population of high affinity (perfectly fitting) antibodies, which target only the pathogen. And the antibodies are available relatively fast.