Loss of taste and smell during a SARS-CoV-2 infection
This is not completely clear to say the least, but there are some hints. Please keep in mind that there was not much time for extensive research, since this disease is still quite new. What seems clear (at least at the moment) is that most like the nerve cells (in the olfactory bulb as well as in the taste bud) are not directly affected, since they do not express the ACE2 receptor on their surface which is needed by the virus to enter a cell.
In the nose, there are two studies, which link the loss of smelling to massive damage of the epithelial cells surrounding the nerve cells. These cells have a lot of ACE2 receptors and are thus susceptible for an infection with SARS-CoV-2. An infection leads to the death of these cells, which disturbs the balance of ions for the nerve cells leading to loss of smell. Specifically this seems to affect sustentacular cells. Since the epithelial cells recover over time, smelling comes back. This also fits the description of affected people, who describe the loss of smell as a sudden event, while the recovery happens gradually.
For the tasting it is less clear, but it also seems that supporting cells (which carry ACE2 receptors) are affected. For sensing of sharp (for example Chili), which is not chemical sensing but via pain sensing, it seems that at least a part of the cells are also susceptible to the virus as they express ACE2.
Taken together this seems to indicate that important cells in the vicinity of the nerve cells are hit by the virus, causing anosmia and ageusia. It is also important to note that not all people who lose their sense of smell also lose the sense of taste.
For furter reading have a look at the references, the first one is a nice article in the scientific american, the second goes deep into the topic with a lot of nice images.
I will have a deeper look into the topic later and probably update the answer.
References:

Mysteries of COVID Smell Loss Finally Yield Some Answers
COVID-19 and the Chemical Senses: Supporting Players Take Center
Stage
Massive transient damage of the olfactory epithelium associated with
infection of sustentacular cells by SARS-CoV-2 in golden Syrian
hamsters
SARS-CoV-2 Receptor ACE2 Is Enriched in a Subpopulation of Mouse
Tongue Epithelial Cells in Nongustatory Papillae but Not in Taste
Buds or Embryonic Oral Epithelium
Real-time tracking of self-reported symptoms to predict potential
COVID-19