Single-View Food Portion Estimation: Learning Image-to-Energy Mappings Using Generative Adversarial Networks
Shaobo Fang Zeman Shao Runyu Mao Chichen Fu Deborah A. Kerr Carol J. Boushey Edward J. Delp Fengqing Zhu




𝐱
ŁL




 




Shaobo Fang^⋆, Zeman Shao^⋆, Runyu Mao^⋆, Chichen Fu^⋆, 
This work was partially sponsored by the National Science Foundation under grant 1657262,  a Healthway Health Promotion Research Grant and from the Department of Health, Western Australia, and by the endowment of the Charles William Harrison Distinguished Professorship at Purdue University. Address all correspondence to Fengqing Zhu, zhu0@ecn.purdue.edu or see www.tadaproject.org.
Deborah A. Kerr^†, Carol J. Boushey^, Edward J. Delp^⋆ and Fengqing Zhu^⋆

^⋆School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, USA

^†School of Public Health, Curtin University, Perth, Western Australia

^Cancer Epidemiology Program, University of Hawaii Cancer Center, Honolulu, Hawaii, USA


Single-View Food Portion Estimation: Learning Image-to-Energy Mappings Using Generative Adversarial Networks
    [
    
============================================================================================================






Due to the growing concern of chronic diseases and other health problems related to diet, there is a need to develop accurate methods to estimate an individual's food and energy intake. Measuring accurate dietary intake is an open research problem. In particular, accurate food portion estimation is challenging since the process of food preparation and consumption impose large variations on food shapes and appearances. 
In this paper, we present a food portion estimation method to estimate food energy (kilocalories) from food images using Generative Adversarial Networks (GAN).
We introduce the concept of an “energy distribution” for each food image.


To train the GAN, we design a food image dataset based on ground truth food labels and segmentation masks for each food image as well as energy information associated with the food image.
Our goal is to learn the mapping of the food image to the food energy.


We can then estimate food energy based on the energy distribution.
We show that an average energy estimation error rate of 10.89% can be obtained by learning the image-to-energy mapping. 





Dietary Assessment, Food Portion Estimation, Generative Models, Generative Adversarial Networks, Image-to-Energy Mapping






§ INTRODUCTION


Dietary assessment, the process of determining what someone eats during the course of a day, provides valuable insights for mounting intervention programs for prevention of many chronic diseases.






Traditional dietary assessment techniques, such as dietary record, requires individuals to keep detailed written reports for 3-7 days of all foods or drink consumed <cit.> and is a time consuming and tedious process.
Smartphones provide a unique mechanism for collecting dietary information and monitoring personal health.
Several mobile dietary assessment systems, that use food images acquired during eating occasions, have been described such as the TADA system <cit.>, FoodLog <cit.>, FoodCam <cit.>, DietCam <cit.>, and Im2Calories <cit.> to automatically determine the food types and energy consumed  using image analysis and computer vision techniques. 

Estimating food portion size/energy (kilocalories) is a challenging task since the process of food preparation and consumption impose large variations on food shapes and appearances.  

There are several image based techniques for food portion size estimation that either require a user to take multiple images/videos or modify the mobile device such as the use of multiple images <cit.>, video <cit.>, 3D range finding <cit.> and RGB-D images <cit.>.

In this paper we focus on food portion estimation from a single food image since this reduces a user's burden in the number and types of images that need to be acquired <cit.>.

Estimating food portion size or food volume from a single image is an ill-posed inverse problem.
Most of the 3D information has been lost during the projection process from 3D world coordinates onto 2D camera sensor plane.
Various approaches have been developed to estimate food portion size and energy information from a single-view food image.
In <cit.>, a 3D model is manually fitted to a 2D food image to estimate the portion size. This approach is not feasible for automatic food portion analysis.

In <cit.>, food image areas are used for portion size estimation based on user's thumbnail as a size reference.  
In <cit.>, the pixels in each corresponding food segment are counted to determine the portion sizes. In <cit.>, food image is divided into sub-regions and food portion estimation is done via pre-determined serving size classification.  
We have previously developed a 3D geometric-model based technique for portion estimation <cit.> which incorporates the 3D structure of the eating scene and use geometric models for food objects. 

We showed that more accurate food portion estimates could be obtained using geometric models for food objects whose 3D shape can be well-defined compared to a high resolution RGB-D images <cit.>.
Geometric-model based techniques require accurate food labels and segmentation masks. Errors from these steps can propagate into food portion estimation.


More recently, several groups have developed food portion estimation methods using deep learning <cit.> techniques, in particular, Convolutional Neural Networks (CNN) <cit.>. 

In <cit.>, a food portion estimation method is proposed based on the prediction of depth maps <cit.> of the eating scene.
However, we have shown that the depth based technique is not guaranteed to produce accurate estimation of food portion <cit.>.
In addition, energy/nutrient estimation accuracy was not reported in  <cit.>.
In <cit.>, a multi-task CNN <cit.> architecture was used for simultaneous tasks of energy estimation, food identification, ingredient estimation and cooking direction estimation.
Food calorie estimation is treated as a single value regression task <cit.> and only one unit in the last fully-connected layer (FC) in the VGG-16 <cit.> is used for calorie estimation.

 




Although CNN techniques have achieved impressive results for many computer vision tasks, they depend heavily on well-constructed training datasets and proper selection of the CNN architecture.

We propose in this paper to use generative models to estimate the food energy distribution from a single food image.
We construct a food energy distribution image that has a one-to-one pixel correspondence with the food image.
Each pixel in the energy distribution image represents the relative spatial amount (or weight) of food energy at the corresponding pixel location.
Therefore, a food energy distribution image provides insight not only on where the food items are located in the scene, but also reflects the weights of energy in different food regions (for example, regions of the image containing broccoli should have smaller weights due to lower energy (kilocalories) compared to regions of the image containing steak).   
The energy distribution image is one way that we can visualize these relationship. 


More specifically, the generative model is trained on paired images <cit.> mapping a food image to its corresponding energy distribution image.
Our goal is to learn the mapping of the food image to the food energy distribution image so that we can construct an energy distribution image for any eating occasion and then use this energy distribution to estimate portion size.
The weights in food energy distribution image for the training data are assigned based on ground truth energy using a linear transform described in Section <ref>. 




We use a Generative Adversarial Networks (GAN) architecture <cit.> as GAN has shown impressive success in training generative models <cit.> in recent years.



Currently, no publicly available food image dataset meets all of the requirements for training our generative model that learns the “image-to-energy mapping."
We constructed our own dataset based on ground truth food labels, segmentation masks and energy information for training the generative model.


The contribution of this paper is twofold. First, we show that the proposed method can obtain accurate estimates of food energy from a single food image. Second, we introduce a  method for modeling the characteristics of energy distribution in an eating scene. 


      


§ LEARNING IMAGE-TO-ENERGY MAPPINGS

Here we will initially discuss the requirements of the training dataset and then we will describe in more details how we construct the energy distribution image from the training data.
Image pairs consisting of the food image and corresponding energy distribution image are required to train the GAN.




There are a several publicly available food image datasets such as the PFID <cit.>, UEC-Food 100/256 <cit.> and Food-101 <cit.>.  
However, none of these dataset contains sufficient information required for training a generative model that we can use to learn the “image-to-energy mapping".

We created our own paired image dataset for training the GAN with ground truth food labels, segmentation masks and energy/nutrient information from a food image dataset we have collected from dietary studies. This is described in more details in Section <ref>.
We use the conditional GAN architecture <cit.> for training our generative model.



 §.§ The Image-to-Energy Dataset


The generative model is designed to best capture the characteristics of the energy distribution associated with food items in an eating scene.

For food types that have different energy distribution (such as broccoli versus steak), the differences should be reflected in the energy distribution image.   

For constructing the image-to-energy training dataset, we use food images collected from a free-living dietary TADA study <cit.>.

We manually generated the ground truth food label and segmentation mask associated with each food item in the user food image dataset.    

The ground truth energy information (in kilocalories) for each food item was provided by registered dietitians. 
For these food images we have a fiducial marker with known dimension that is located in each eating scene to provide references for worlds coordinates, camera pose, and color calibration.
The fiducial marker is a 5 × 4 color checkerboard pattern as shown in Figure <ref>.




The food energy distribution image we construct from the above ground truth information needs to reflect the differences in spatial energy distribution for food regions in the scene.
 

For example, for French fries stacked in pyramid shape, the center region of French fries should have more relative energy weight compared to the edge regions in the energy distribution image.




















To construct the energy distribution image we first detect the location of the fiducial marker using <cit.>.
We then obtain the 3 × 3 homography matrix 𝐇 using the Direct Linear Transform (DLT) <cit.> to rectify the image and remove projective distortion.   
Assume 𝐈 is the original food image, the rectified image 𝐈̂ can then be obtained by: 𝐈̂ = 𝐇^-1𝐈.
The segmentation mask S_k associated with food k can then be projected from the original pixel coordinates to the rectified image coordinates as Ŝ_k = 𝐇^-1S_k.
At each pixel location (î, ĵ) ∈Ŝ_k, we assign a  scale factor ŵ_î,ĵ reflecting the distance of the pixel location (î,ĵ) to the centroid of the segmentation mask Ŝ_k.
The scale factor ŵ_î,ĵ is defined as:

    ŵ_î,ĵ  = 1/√((î -î_c)^2 + (ĵ-ĵ_c)^2) + ϕ_Ŝ_k^0.5,    ∀ (î,ĵ) ∈Ŝ_k,

where (î_c, ĵ_c) is the centroid of Ŝ_k and the regularization term, ϕ_Ŝ_k, is defined as ϕ_Ŝ_k = (∑_∀ (î,ĵ) ∈Ŝ_k1).
If the pixel location (î,ĵ) is outside of the segmentation mask Ŝ_k, then ŵ_î,ĵ = 0, ∀ (î,ĵ) ∉Ŝ_k. 

With the scale factor ŵ_î,ĵ assigned to each pixel location in Ŝ_k, we can project the weighted segmentation masks Ŝ_k back to the original pixel coordinates as S̅_k = 𝐇Ŝ_k, and learn the parameter ρ_k such that:

    c_k = ρ_k ∑_∀ (i̅,j̅) ∈S̅_kw̅_i̅,j̅,
  
where c_k is the ground truth energy associated with food k, ρ_k is the energy mapping coefficient for S̅_k and w̅_i̅,j̅ is the energy weight factor at each pixel that makes up the ground truth energy distribution image. 

We then update the energy weight factors in S̅_k as:

    w̅_i̅,j̅ = ρ_k ·w̅_i̅,j̅,   ∀ (i̅,j̅) ∈S̅_k.

We repeat the process following Equation <ref> and <ref> for all k ∈{1, …, M } where M is the number of food items in the eating scene image.
We can then construct a ground truth energy distribution image 𝒲̅ of the same size as 𝐈̅: 𝐈̅ = 𝐇𝐈̂, by overlaying all segments S̅_k, k ∈{1, …, M } onto 𝒲̅. Thus, we obtain the paired images of an eating scene: the image 𝐈̅ and the energy distribution image 𝒲̅ with one-to-one pixel correspondence as shown in Figure <ref> and <ref>. 










 §.§ Learning The Image-to-Energy Mappings








In this paper we use a Conditional GAN (cGAN) <cit.> that learns a generative model under conditional setting based on an input image.
A cGAN is a natural fit for our “image-to-energy mapping" task since we want to predict the energy distribution image based on a food image.     

More specifically, the  cGAN attempts to learn the mapping from a random noise vector 𝐳 to a target image 𝐲 conditioned on the observed image 𝐱: G(𝐱, 𝐳) →𝐲.
The objective function of a conditional GAN is expressed as:

    ℒ_cGAN(G, D)    = 𝔼_𝐱, 𝐲∼ p_data(𝐱, 𝐲)[log D(𝐱, 𝐲)] + 
       𝔼_𝐱∼ p_data(𝐱), 𝐳∼ p_z(𝐳) [log(1 - D(𝐱, G(𝐱, 𝐳))].

An additional conditional loss ℒ_conditional(G) is added <cit.> that further improves the generative model's mapping G(𝐱, 𝐳) →𝐲:

    ℒ_conditional(G) = 𝔼_𝐱, 𝐲∼ p_data(𝐱, 𝐲), 𝐳∼ p_z(𝐳)[𝒟(𝐲, G(𝐱, 𝐳))],

where 𝒟(𝐲, G(𝐱, 𝐳)) measure the distance between 𝐲 and G(𝐱, 𝐳).
Commonly used criteria for 𝒟(·) are the L_2 distance <cit.>:

    𝒟(𝐲, G(𝐱, 𝐳))  = 1/n∑_i=1^n(𝐲_i - G(𝐱_i, 𝐳_i))^2,

the L_1 distance <cit.>: 

    𝒟(𝐲, G(𝐱, 𝐳))  = 1/n∑_i=1^n|𝐲_i - G(𝐱_i, 𝐳_i)|,

and a smooth version of the L_1 distance:

    𝒟(𝐲, G(𝐱, 𝐳)) = 1/n∑_i=1^n(𝐲_i - G(𝐱_i, 𝐳_i))^2/2   if |𝐲_i - G(𝐱_i, 𝐳_i)| < 1
      
    
                |𝐲_i - G(𝐱_i, 𝐳_i)|    otherwise.

The final objective for both the cGAN and the conditional terms is defined as <cit.>:

    G^* = min_G max_D ℒ_cGAN(G, D) + λℒ_conditional(G).

The generative model G^* obtained from Equation <ref> is then used to predict the energy distribution image 𝒲̃ (Figure <ref>) based on the food image (Figure <ref>). 





 


§ EXPERIMENTAL RESULTS


We have 202 food images that have been manually annotated with ground truth segmentation masks and labels as training samples.
All the food images are collected from a free-living (in the wild) TADA dietary study <cit.>.


Registered dietitians provided the ground truth energy information for each food item in the images. 



We constructed a dataset of paired images based on the 202 food images.
Data augmentation techniques such as rotating, cropping and flipping were used to further expand our training dataset so that a total of 1875 paired images were used to train the cGAN.
We used 220 paired images for testing.




We believe the training dataset size is sufficient for our task of predicting the energy distribution image because the cGAN is a mapping of a higher dimensional food image to a lower dimensional energy distribution image. 
In addition, since all food images are captured by users sitting naturally at a table, there is no drastic changes in viewing angles (for example, from wide angle to close up).




In other image-to-image mapping tasks, a training dataset size of 400 has been used <cit.> for architectural labels (simple features) to photo translation (complex features) <cit.>.






In testing, once the cGAN estimates the energy distribution image 𝒲̃, we can then determine the energy for a food image (portion size estimation) as: estimated energy = ∑_∀ (i, j) ∈𝐈̅ (w̃_i,j).













We compared the estimated energy image 𝒲̃ (Figure <ref>) to the ground truth energy image 𝒲̅ (Figure <ref>), and define the error between 𝒲̅ and 𝒲̃ as:

    Energy Estimation Error Rate = ∑_∀ (i, j) ∈𝐈̅ (w̃_i,j - w̅_i,j)/∑_∀ (i, j) ∈𝐈̅ (w̅_i,j)




To compare different cGAN models, we used the encoder-decoder architecture <cit.> and the U-Net architecture <cit.>.
We compared the energy estimation error rates at different epochs for both architectures.
We observed that the U-Net architecture (Figure <ref>) is more accurate in energy estimation and more stable compared to the encoder-decoder architecture (Figure <ref>).
This is due to the fact that the U-Net can copy information from the “encoder” layers directly to the “decoder" layers to provide precise locations <cit.>, an idea similar to ResNet <cit.>.

 
We also compared the energy estimation error rates under different conditional loss settings: ℒ_conditional(G) using U-Net.
We used the batch size of 16 with λ = 100 in Equation <ref>,  the Adam <cit.> solver with initial learning rate α = 0.0002, and momentum parameters β_1 = 0.5, β_2 = 0.999 as in <cit.>. 
Based on our experiments, distance measure 𝒟(·) using the L_1 or L_2 norms is better than using smoothed L_1 norm.







At epoch 200, the energy estimation error rates are 10.89% (using L_1 criterion) and 12.67% (using L_2 criterion), respectively.
Using geometric-models <cit.> techniques, the energy estimation error was 35.58%  <cit.>.



































§ CONCLUSION

In this paper, we presented a food portion estimation technique based on generative models.
We showed that the energy estimation error was reduced to  10.89%.
Compared to earlier geometric model-based technique that relies on accurate food segmentation and classification, the energy estimation task can be performed in parallel with segmentation and classification.
We are interested in incorporating the results from different parts of our system (classification, segmentation and energy estimation) to further improve the overall accuracy of our dietary assessment system.

We are also investigating the effect of different eating scene characteristics on the energy estimation error. For example, food items that are occluded often yield high energy estimation error. 









IEEEbib


