Is it harmful for someone to consume things full of bacteria if they don't get physically sick from the bacteria at all?
It really depends on what you consume, how much of it you consume, and the state of your immune system. Let me give you some examples. Yogourt is full of bacteria, and yet we can eat it without issue. The bacteria likely don't survive digestion, but if they do, they will quite happily live in our intestines contributing to the already existing population of good bacteria.

Streptococcus is a genus of bacteria that contains some species that live all over us. They are on our skin, in our upper respiratory tract and intestine. If the species, S. pneumoniae makes it into our lungs, this can cause a bacterial pneumonia. There are likely few bacteria making it into our lungs on a regular basis, and normally our immune system can fight this off without concern. If we some how become immune depressed or compromised (for a number of medical reasons) then that's when we become symptomatic and ill.

On a more extreme side, if we have a bacteremia it is called a bacteremia. There are benign cases of this every time your brush your teeth and bacteria are able to enter tiny cuts in your gums from the brushing action. A healthy person fights this off. If a bacteremia is left unchecked, because of a lack of immune response or trauma, this can develop into a sepetcemia, a toxic infection of the blood. This can be very life threatening and is quite serious.

In general, we always have bacteria all over our hands which we likely transmit to our food or otherwise ingest. This is normally not a problem, but there is an appreciable risk of getting some kinds of viral or bacterial infection (eg, strep throat or a cold). Washing with soap (and to a lesser degree, using hand sanitizers) minimizes this risk.