What exactly is bootstrapping in reinforcement learning?
Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value". 

In most TD update rules, you will see something like this SARSA(0) update:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(R_{t+1} + \gamma Q(s',a') - Q(s,a))$$

The value $R_{t+1} + \gamma Q(s',a')$ is an estimate for the true value of $Q(s,a)$, and also called the TD target. It is a bootstrap method because we are in part using a Q value to update another Q value. There is a small amount of real observed data in the form of $R_{t+1}$, the immediate reward for the step, and also in the state transition $s \rightarrow s'$.

Contrast with Monte Carlo where the equivalent update rule might be:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(G_{t} - Q(s,a))$$

Where $G_{t}$ was the total discounted reward  at time $t$, assuming in this update, that it started in state $s$, taking action $a$, then followed the current policy until the end of the episode. Technically, $G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$ where $T$ is the time step for the terminal reward and state. Notably, this target value does not use any existing estimates (from other Q values) at all, it only uses a set of observations (i.e., rewards) from the environment. As such, it is guaranteed to be unbiased estimate of the true value of $Q(s,a)$, as it is technically a sample of $Q(s,a)$.

The main disadvantage of bootstrapping is that it is biased towards whatever your starting values of $Q(s',a')$ (or $V(s')$) are. Those are are most likely wrong, and the update system can be unstable as a whole because of too much self-reference and not enough real data - this is a problem with off-policy learning (e.g. Q-learning) using neural networks.

Without bootstrapping, using longer trajectories, there is often high variance instead, which, in practice, means you need more samples before the estimates converge. So, despite the problems with bootstrapping, if it can be made to work, it may learn significantly faster, and is often preferred over Monte Carlo approaches.

You can compromise between Monte Carlo sample based methods and single-step TD methods that bootstrap by using a mix of results from different length trajectories. This is called TD($\lambda$) learning, and there are a variety of specific methods such as SARSA($\lambda$) or Q($\lambda$).