Silver and milk (or milk products)
You're right: the silver is reacting with sulfur compounds in the food to form a tarnish of silver sulfide. This is most commonly observed, in my experience, using silver teaspoons with boiled eggs, which are pretty rich in sulfur.
There are a number of reactions that can take place depending on the sulfur-containing species - the abstract from this paper from the Journal of Chemical Education (unfortunately pay-walled) has a nice example:

Silver and silver-plated objects react with sulfur and sulfur compounds to produce silver sulfide $\ce{Ag2S}$ or tarnish. Contact with materials that contain sulfur compounds, such as hard boiled eggs, mayonnaise, mustard, and rubber bands can cause tarnish. In the air, a silver object can tarnish owing to the reaction of silver with hydrogen sulfide ($\ce{H2S}$). This is a gas found in the air as a result of some industrial processes and the decomposition of dead plants and animals. The reaction of silver with hydrogen sulfide to form tarnish is as follows:
$\ce{2 Ag(s) + H2S(g) -&gt; Ag2S(s) + H2(g)}$

In Sterling silver specifically, you'll often also get copper sulfide formed.
The paper also mentions that you can remove the tarnish abrasively, or through reacting the tarnish with aluminium (via solution, electrochemically):

$\ce{3 Ag2S(s) + 2 Al(s) -&gt; 6 Ag(s) + Al2S3(s)}$

But a more common approach used by many commercial products is to dissolve off the tarnish, usually in sodium thiosulfate or thiourea. This means you end up with slightly less silver on your item, but is otherwise pretty effective and very mild (it's commonly used for museum pieces).