Deep Cooking: Predicting Relative Food Ingredient Amounts from Images
Jiatong Li Ricardo Guerrero Vladimir Pavlovic















jiatong.li@rutgers.edu

  Rutgers University
  Piscataway
  NJ
  USA



r.guerrero@samsung.com

  Samsung AI Center
  Cambridge
  UK



vladimir@cs.rutgers.edu

  Rutgers University
  Piscataway
  NJ
  USA







  In this paper, we study the novel problem of not only predicting ingredients from a food image, but also predicting the relative amounts of the detected ingredients. We propose two prediction-based models using deep learning that output sparse and dense predictions, coupled with important semi-automatic multi-database integrative data pre-processing, to solve the problem. Experiments on a dataset of recipes collected from the Internet show the models generate encouraging experimental results.








<ccs2012>
<concept>
<concept_id>10010147.10010257.10010293.10010294</concept_id>
<concept_desc>Computing methodologies Neural networks</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Computing methodologies Neural networks













Deep Cooking: Predicting Relative Food Ingredient Amounts from Images
    Vladimir Pavlovic
    December 30, 2023
=====================================================================





§ INTRODUCTION

Increased awareness of the impact of food consumption on health and lifestyle today has given rise to novel data-driven food analysis systems, e.g., <cit.>, whose goals are to alleviate the challenge of tracking the daily food intake. Many of those systems use data modalities such as images to seamlessly extract information related to the food item that was consumed, often the identity of the meal or its ingredients, or even its caloric value.  While these systems frequently claim to predict the energy intake, they base these predictions on standard energy tables of standardized ingredients (e.g., USDA[https://ndb.nal.usda.gov/ndb/search/list]). The estimation of the food amount, a highly challenging and often ambiguous task, is delegated to the users themselves. Even the systems that aim to predict a fine-grained ingredient-based representation of the food item, e.g., <cit.> and <cit.>, do not consider the problem of predicting the ingredients' amounts or relative contributions of the food items in each dish. However, these amounts are paramount for estimating the correct energy value of the meal. A small amount of high-fat food might not be a major health risk factor, while an unhealthy ingredient with a dominant amount may lead to potential health problems.

In this paper, we study the novel problem of predicting the relative amount of each ingredient in a food item from images. To our knowledge, this is the first work goes in to the detail of analyzing the amounts of each ingredients. The problem is both interesting and challenging. It's interesting because we can analyze the nutrients of a food in detail and how each ingredient in the food contributes to health. Some ingredients can be replaced with their low-calorie counterparts and some can be replaced with vegan substitutions according to users' dietary choices. We can also modify the amounts to create healthier foods. It's challenging because the shape and color of even the same ingredient can exhibit large visual differences due to diverse ways of cutting and cooking <cit.>.  Analyzing the amounts of ingredients may also suffer from occlusion. We attack the problem by deep learning models which output the amount of each ingredient.

The contributions of this paper are:

    
  * We propose a novel and challenging problem: analyzing the relative amounts of each ingredient from a food image.
    
  * We propose prediction-based models using deep learning to solve the problem.




§ RELATED WORK

As food plays an essential part in our life, there has been a lot of research focusing on food classification, cross-modal retrieval, ingredient analysis and volume estimation.






Food Classification. For classification, initially hand-crafted features and traditional classifiers are used to classify food images. <cit.> uses random forests and SVMs. <cit.> uses bag of visual words approach, SVM as the classifier to classify restaurant-specific food and calculate the calories of a meal from a given restaurant.
The traditional methods are outperformed by methods using deep learning features or directly using deep learning.  <cit.> uses SVM classifier with CNN features. <cit.> also uses deep learning features for classification. <cit.>, <cit.> and <cit.> train deep-learning models for food recognition.

Cross-Modal Retrieval is given an image, retrieving its recipe from a collection of test recipes or the other direction: retrieving the corresponding image given a recipe. <cit.> finds a joint embedding of recipes and images for the image-recipe retrieval task. <cit.> expands the dataset Recipe1M<cit.> from 800k images to 13M. However, they expand with images from web search using recipe titles as queries. It is likely that the web images no longer share the same ingredients or instructions as the original ones. They also annotate amount or nutrition information for 50k recipes, while we annotate 250k recipes with our method and 80k of them have images. <cit.> uses attention for finding the embedding for cross-modal retrieval. <cit.> proposes ACME which learns cross-modal embeddings by imposing modality alignment using an adversarial learning strategy and imposing cross-modal translation consistency. 

Ingredient Analysis is predicting the existence of ingredients given a food image. <cit.> uses multi-task deep learning and a graph modeling ingredient co-occurrences. <cit.> predicts ingredients as sets and generates cooking instructions by attending to both image and its inferred ingredients simultaneously.
Unfortunately, previous work discard quantity information, losing a lot of important information.

Volume Estimation. Estimating food amounts itself, especially estimating absolute amounts, is not a novel problem. Some methods focus on multi-view 3d reconstruction while others focus on single-view volume estimation.
Multi-view reconstruction can date back to 1994 <cit.>. Even a more recent work <cit.> using two views uses only traditional computer vision techniques. <cit.> uses a simple method, mapping contour, but the performance might come from an easy dataset. Although <cit.> uses deep learning in their work, it's only for object detection and they use a simple method to calculate volume.

As for single view methods, <cit.> tries to recognize the contents of a meal from a single image then predicts the calories for home-cooked foods. Their method is based on deep learning for classification and depth estimation to calculate volume and calories. <cit.> uses GANs but it requires densely annotated datasets. <cit.> reviews three of their existing systems and proposes two novel systems, either using size-known reference objects or foods, including rice grains, or using special mobile devices, like built-in inertial sensors and stereo cameras.

Estimating relative amounts, on the other hand, does not require reference objects or special devices. Our methods estimating relative ingredient amounts can be applied to food images on the internet, where camera information and size-known reference objects are not available.

Furthermore, previous work calculate the amounts of foods as a whole while in our work, we go into the detail and estimate the amount of each ingredient using deep learning models.



§ METHODS




















Let the amounts of ingredients of a recipe be 𝐯_y=(v_y_1,…,v_y_i,…,v_y_I) where I is the total number of ingredients and v_y_i is the amount of the i-th ingredient in grams, v_y_i≥ 0. v_y_i=0 when the i-th ingredient is not present in the recipe. Suppose a is a constant and a>0, a𝐯_y also represents the same recipe. Therefore we assume that 𝐯_y is normalized such that ∑_i=1^Iv_y_i=C where C is a constant. v_y_i can then be interpreted as the proportion of the i-th ingredient, or there are v_y_i grams of the i-th ingredient every C grams of total ingredients. 

Let the amount prediction of an image 𝐱 corresponding to the recipe be 𝐯_x=(v_x_1,…,v_x_I) and v_x is normalized such that ∑_i=1^Iv_x_i=C. 

We propose two methods, one outputs dense amount predictions and users can threshold according to their applications; the other outputs sparse amount predictions, just like real recipes which usually use a few ingredients. 


 §.§ Dense Method

When C=1, both v_x, v_y can be viewed as probability distributions. If the last layer of a neural network is activated by softmax, the prediction is naturally normalized to C=1. Therefore we use softmax to activate the last fully connected layer of a neural network which predicts v_x given 𝐱. The loss function to minimize is the cross entropy of v_x and v_y, L=-∑_i=1^Iv_y_ilog v_x_i.


 §.§ Sparse Method

ReLU activation brings negative values to 0, which brings some sparsity in the output and L1 loss also enforces sparsity. Here, the last fully connected layer of the neural network is activated by ReLU and the loss function is the L1 distance between v_x,v_y,  L=∑_i=1^I|v_y_i-v_x_i|.




§ EXPERIMENTS



 §.§ Dataset

We use the data in Recipe1M <cit.> dataset. As the fraction slash is not an ASCII character on recipe websites, it is missing in the dataset after preprocessing. We re-scrape recipe ingredients from websites that are nicely structured, with the quantity and ingredient names parsed.

Unit Extraction We define 3 types of units:

    
  * Basic units, including volume, weight and counting units (e.g. box). We define 52 of them and list them in Appendix <ref>.
    
  * Size modified units. Usually a counting unit after large/ big/ medium/ small (e.g. small scoop of ice cream). We treat "large/ big" as a multiplier 1.2, "medium" as 1.0 and "small" as 0.8.
    
  * Numbers and basic units combined in a parenthesis. (e.g. 1 (15 1/4 ounce) box of cake mix)

We only keep recipes that for all ingredients in the recipe, the words between quantity and ingredient name belong to the units we defined. If an ingredient in a recipe does not have quantity field (e.g. some salt), we ignore the ingredient in the recipe. This step only removes 6% of re-scraped recipes.

Canonical Ingredient Construction. Recipe1M contains about 16k unique ingredients and the top 4k ingredients account for an average coverage of 95%, which is calculated by 1/R∑_k=1^Ri_ck/i_k, where R is the number of recipes, i_k is the number of ingredients in the k-th recipe and i_ck is the number of ingredients both in the k-th recipe and the top 4k ingredients. The 4k ingredients are further reduced to 1.4k by first merging the ingredients with the same name after a stemming operation and semi-automatically fusing other ingredients. Ingredients are fused if they:

    
  * Are close together in Word2vec <cit.> embedding space, which is trained on the titles, ingredients and instructions of Recipe1M.
    
  * Share two or three words.
    
  * Are mapped to the same item in Nutritionix[https://www.nutritionix.com/]

A human annotator accepts the proposed merger. We only keep recipes with at least 80% coverage. After the two steps, our data contains 460k recipes in total.

Unit Conversion. The amount numbers are meaningless without converting them to the same unit. In the dataset, one ingredient can be represented by volume, weight and counting units. For example, 2 onions, 1 cup chopped onion, 1 lb onions. We convert the units to grams using Nutritionix and USDA Food Database. The details of mapping can be found in appendix <ref>. Some unit conversions like "1 packet of sugar" are not defined by neither of websites and we only keep recipes where 80% of the ingredients and their corresponding units are converted. Next we vectorize each recipe into two vectors, v_y for the amount value, described in section <ref> and v_r=(v_r_1,…,v_r_I) for the amount range, if the amounts are not exactly given. v_r_i corresponds to the amount range of the i-th ingredient. For "24-25 ounce cheese_ravioli", the value is 24.5 ounce, or 686 grams, the range is 28 grams (1 ounce).  Our data contains 250k recipes after unit conversion and 80k of them have images.



 §.§ Compared Methods

Retrieval-Based Method: One method of predicting the ingredients given a food image is cross-modal recipe retrieval which outputs the ingredients and the corresponding amounts of the retrieved recipe. We use the model in <cit.>. For a fair comparison, the model is trained only with ingredients and images. Titles and instructions are not included as the prediction-based models do not use titles or instructions during training. No amount information is used during training the retrieval model. Following <cit.>, the top 1 recipe among 1000 randomly selected recipes is retrieved. We try two settings, one includes the ground truth recipe in the 1000 recipes and the other without. The two settings only differ by 1 recipe.

Prediction-Based Methods: We use a Resnet50 <cit.> pre-trained on UPMC <cit.> and replace the last layer with ingredient amount prediction. Softmax: The dense method in Section <ref>. As recipe ingredients are sparse, the dense output is thresholded to top 10 predictions and renormalized. L1: The sparse method in Section <ref>. Both models are fine tuned with Adam optimizer with learning rate 10^-4. The batch size is 64.



 §.§ Evaluation and Results

We report 4 evaluation metrics, the first two evaluate ingredient detection and the last two evaluate amounts and calories. If the predicted amount is non-zero for an ingredient, the ingredient is viewed as detected. The ingredient is viewed as not exist if the predicted amount is 0.

    
  * Recall of ground truth ingredients # common ingredients between ground truth and predicted over # ground truth.
    
  * IOU # common over # the union of ground truth and predicted.
    
  * L1 Error First normalize both the ground truth range vector and the predicted vector to C=1000 (every kilogram of total ingredients). The ground truth range vector is normalized according to the ground truth amount vector. If the predicted amount falls outside the range, the error in the dimension is the difference between the amounts. The error is zero otherwise. The L1 norm of the error is reported.
    
  * Relative Calorie Error (RCE) First normalize both the ground truth vector and the predicted vector. Suppose the energy of the i-th ingredient is c_i kCal/g, the ground truth calorie can be estimated as C_y=∑_i=1^Ic_iv_y_i and the predicted calorie is C_x=∑_i=1^Ic_iv_x_i, the error |C_y-C_x|/C_y is reported.
 
The results are shown in Table <ref>. The 80k recipes with images are randomly split into training, validation and test sets, with 48k, 16k, 16k each. The results are on test set. Numbers are "mean (standard deviation over test set)". The up arrows indicate the higher the better and the down arrows indicate the lower the better.


The results show that there is no significant difference between the performance of prediction based methods and retrieval with the ground truth recipe included while they all outperform retrieval without ground truth in terms of recall and L1 loss. Compared with retrieval with ground truth, prediction-based methods have lower standard deviations in IOU, L1 and RCE, or more robust performances in general. This is reasonable as retrieval based methods can be affected by the collection of recipes available for the retrieval system. 

Relative calorie error tend to have a larger standard deviation because low calorie recipes are sensitive to calorie differences. Figure <ref> shows the histogram of relative calorie errors in log scale and the recipe with the largest relative calorie error 4725.32. Most recipes have small relative calorie errors while there are some outliers. The error of about 88% of the test recipes is less than 1. The calorie of the recipe with the largest RCE after normalizing the sum of ingredients is about 1.11 kCal and the calorie of the retrieved recipe is about 5255.65 kCal. The system retrieves the salad dressing recipe probably because both recipes are liquid.


When thresholded to top 10 predictions and renormalized, the Softmax method performs well in terms of amounts and calories. Furthermore, the threshold can be adjusted according to users' needs, showing the method is able to produce encouraging results at the same time being flexible.

We also demonstrate some easy, average and difficult examples for the models in terms of evaluation metrics. We notice that even the methods perform poorly quantitatively in some difficult test cases, they still produce reasonable ingredient and amount combinations. The result are shown in Table <ref> and Figure <ref>.





In the good and average example, all three methods produce reasonable results. The retrieval system gets foods from the same high-level category: baked goods and prediction-based models capture main ingredients like flour, sugar and butter well. The amount of salt is also accurately predicted. In the bad example, the ground truth uses puffed rice cereals and the similar ingredient rice is also successfully retrieved, although the taste is incorrect: the ground truth is sweet while the retrieved is savory. Softmax successfully predicts the existence of magarine and the overall recipe produced is reasonable as the ground truth image looks like baked goods. L1 successfully predicts a similar ingredient butter with nearly the same amount. Even when the quantitative evaluations are bad, qualitatively the models still give robust predictions.



§ CONCLUSION

We studied a novel problem: given a food image, predict the relative amount of each ingredient needed to prepare the observed food item. Our experiments show that deep model-based prediction methods produce reasonable ingredient and amount predictions; even in the presence of challenging test examples, the methods are still able to yield robust qualitative results.

The problem opens up interesting avenues for future work. First, prediction-based methods leverage amount information during training while the retrieval-based method does not. Nevertheless, there is no significant difference in performance of prediction-based methods and retrieval with ground truth. Further research is needed for improving the performance on this problem. Second, we can leverage amount information in retrieval based methods. Finally, we can adjust the ingredients and the corresponding amounts according to users' dietary needs to generate novel food images.

















ACM-Reference-Format




§ LIST OF BASIC UNITS



    
  * Volume Units: pint, liter, gallon, teaspoon, tablespoon, cup, dash, quart, fluid_ounce, ml, pinch
    
  * Weight Units: kg, lb, g, ounce
    
  * Counting Units: packet, handful, links, sheet, big, package, bunch, clove, leaf, jar, medium, strip, envelope, stick, large, drop, piece, small, container, bottle, head, scoop, of, stalk, glass, sprig, bag, inch, loaf, can, cm, ears, no_unit, dozen, box, slice, squares

"no_unit" means the ingredient name directly follows the quantity, which is an expression for counting (e.g. 2 apples)



§ MAPPING INGREDIENTS TO NUTRITIONIX AND USDA


We pass the list of our ingredients to Nutritionix natural language tagging API[trackapi.nutritionix.com/v2/natural/tags] and get json formatted responses. The mapping of the ingredient is the "ITEM" field in the response and unit conversion is the "ALT_MESAURES" field. We then filter out ingredients with no mappings in Nutritionix and ingredients with multiple mappings (e.g."butter flavored shortening" is mapped to both "butter" and "shortening"). For the rest of the results, we first check if the mapping is a sigular or plural form of the ingredient and get a list of ingredients that are not mapped exactly to themselves. For these results, we refine them manually:

    
  * No result: First we try to query the ingredients in USDA Food database to get the corresponding NDB ID as mapping and the conversion tables. If there is no result in USDA, we look for a synonym for the ingredient and query with Nutritionix to get the mapping and conversion table.
    
  * Multiple mappings: First we try to select one of the mappings to match the ingredient. If none of the mappings match the ingredient, we query the ingredient in USDA.
    
  * Mismatch: First we check if the mapping is correct. If it is incorrect (e.g. pea shoots→pea), we query the ingredient in USDA.

We then pass the mappings to Nutritionix nutrients API[trackapi.nutritionix.com/v2/natural/nutrients] to get the calories per gram with the "serving_weight_grams" field and "nf_calories" field. If the mappings are from USDA, we query the USDA Food database to get calories per gram.



