Amount of RAM used 
@johnfhima 
A TensorFlow Lite model is represented in a special efficient portable format known as [FlatBuffers](https://google.github.io/flatbuffers/) (identified by the .tflite file extension). This provides several advantages over TensorFlow's protocol buffer model format such as reduced size (small code footprint) and faster inference (data is directly accessed without an extra parsing/unpacking step) that enables TensorFlow Lite to execute efficiently on devices with limited compute and memory resources and please refer to this [doc](https://www.tensorflow.org/lite/guide#1_generate_a_tensorflow_lite_model) for more details.
Thank you ! 
 Hi,
Tflite GPU delegates by default should use fp16. If on nvidia was used fp32 and in tflite metal was used fp16 it means 2x reduction of data size if everything else the same.
Tflite gpu delegates use special algorithms for memory reuse, it can save significant amount of memory for intermediate tensors that can explain smaller memory footprint.
Tflite gpu delegates by default trying to show the best inference performance and do not trying to save memory if this is bad for inference latency.

You can get some approximate info how much memory was used for constant/intermediate tensors using this methods:
GetIntermediateTensorsSize()
GetConstantTensorsSize()
Of InferenceContext
Sample can be found here:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/gpu/metal/benchmarking/main.mm
 Hi, 
Thanks for your help. I am currently experienced another trouble. As I told you before I am making the inference of my model using tflite and the metal delegate in swift.
When I try to perform a loop of inference, it seems that the memory never stop to increase until reaching an out of memory (if there is too many input).
Why does this behaviour happens? 
If I have enough memory to perform one inference is there a way to free the memory used during this inference before going to the next one ? 

Thanks in advance for your help

******************************************************
Here is the code I use for the inference:




	    var fundusqnet: ModelDataHandler? = ModelDataHandler(modelFileInfo: FundusQNet.modelInfo)
            names_array = []
            quality_array = []
            var i = 0
            for url in self.urlArray{
                print(url)
                if let data = try? Data(contentsOf: url){
                    let image = UIImage(data: data)!
                    var quality_ = self.quality(image: image, fundusqnet: fundusqnet)
                    self.quality_array.append(Float(Int(quality_[0]*10))/10)
                    self.names_array.append(url.lastPathComponent)
                    }
                }
 


func quality(image :UIImage, fundusqnet: ModelDataHandler?) -> [Float]{
        let targetSize = CGSize(width: 224, height: 224)
        let renderer = UIGraphicsImageRenderer(size: targetSize)
        let scaledImage = renderer.image { _ in
            image.draw(in: CGRect(origin: .zero, size: targetSize))
        }
        print(scaledImage.size)
        let pixelBuffer = scaledImage.pixelBuffer()
        let results = (fundusqnet?.runModel(onFrame: pixelBuffer!))
        return results!        
    }
    
    
    
    
And here is my modelDataHandler class:
******************************************
// Copyright 2019 The TensorFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//    http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import CoreImage
import TensorFlowLite
import UIKit


typealias FileInfo = (name: String, extension: String)

enum FundusQNet {
  static let modelInfo: FileInfo2 = (name: "fundusqnet", extension: "tflite") //512optimized 512Artery
}


/// This class handles all data preprocessing and makes calls to run inference on a given frame
/// by invoking the `Interpreter`. It then formats the inferences obtained and returns the top N
/// results for a successful inference.
class ModelDataHandler {
  
  // MARK: - Public Properties
  
  /// The current thread count used by the TensorFlow Lite Interpreter.
  let threadCount: Int
  
  let resultCount = 1
  
  // MARK: - Model Parameters
  
  let batchSize = 1
  let inputChannels = 3
  let inputWidth = 224
  let inputHeight = 224
  
  // MARK: - Private Properties
  
  
  /// TensorFlow Lite `Interpreter` object for performing inference on a given model.
  private var interpreter: Interpreter
  
  /// Information about the alpha component in RGBA data.
  private let alphaComponent = (baseOffset: 4, moduloRemainder: 3)
  
  // MARK: - Initialization
  
  /// A failable initializer for `ModelDataHandler`. A new instance is created if the model and
  /// labels files are successfully loaded from the app's main bundle. Default `threadCount` is 1.
  init?(modelFileInfo: FileInfo2, threadCount: Int = 1) {
    let modelFilename = modelFileInfo.name
    
    // Construct the path to the model file.
    guard let modelPath = Bundle.main.path(
      forResource: modelFilename,
      ofType: modelFileInfo.extension
      ) else {
        print("Failed to load the model file with name: \(modelFilename).")
        return nil
    }
    
    // Specify the options for the `Interpreter`.
    self.threadCount = threadCount
    var options = Interpreter.Options()
    options.threadCount = threadCount
    do {
      // Create the `Interpreter`.
        let delegate = MetalDelegate()
        if delegate != nil {
              //interpreter = try Interpreter(modelPath: modelPath,
                                           // delegates: [delegate])
            interpreter = try Interpreter(modelPath: modelPath, delegates: [delegate])
            } else {
              interpreter = try Interpreter(modelPath: modelPath)
            }
        try interpreter.allocateTensors()
    } catch let error {
      print("Failed to create the interpreter with error: \(error.localizedDescription)")
      return nil
    }
   
  }
  
  // MARK: - Public Methods
  
  /// Performs image preprocessing, invokes the `Interpreter`, and process the inference results.
  func runModel(onFrame pixelBuffer: CVPixelBuffer) -> ([Float])? {
    let sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)
    assert(sourcePixelFormat == kCVPixelFormatType_32ARGB ||
      sourcePixelFormat == kCVPixelFormatType_32BGRA ||
      sourcePixelFormat == kCVPixelFormatType_32RGBA)
    
    
    let imageChannels = 4
    assert(imageChannels >= inputChannels)
    
    // Crops the image to the biggest square in the center and scales it down to model dimensions.
    let scaledSize = CGSize(width: inputWidth, height: inputHeight)
    guard let thumbnailPixelBuffer = pixelBuffer.centerThumbnail(ofSize: scaledSize) else {
      return nil
    }
    
    let outputTensor: Tensor
    do {
      // Remove the alpha component from the image buffer to get the RGB data.
      guard let rgbData = rgbDataFromBuffer(
        thumbnailPixelBuffer,
        byteCount: batchSize * inputWidth * inputHeight * inputChannels
        ) else {
          print("Failed to convert the image buffer to RGB data.")
          return nil
      }
      
      // Copy the RGB data to the input `Tensor`.
      try interpreter.copy(rgbData, toInputAt: 0)
      
      // Run inference by invoking the `Interpreter`.
      try interpreter.invoke()
      
      // Get the output `Tensor` to process the inference results.
      outputTensor = try interpreter.output(at: 0)
    } catch let error {
      print("Failed to invoke the interpreter with error: \(error.localizedDescription)")
      return nil
    }
              
    var results = [Float32](unsafeData: outputTensor.data) ?? []
    
    
    // Return the inference time and inference results.
      return results
  }
  
  // MARK: - Private Methods
  

  

  
  /// Returns the RGB data representation of the given image buffer with the specified `byteCount`.
  ///
  /// - Parameters
  ///   - buffer: The pixel buffer to convert to RGB data.
  ///   - byteCount: The expected byte count for the RGB data calculated using the values that the
  ///       model was trained on: `batchSize * imageWidth * imageHeight * componentsCount`.
  ///   - isModelQuantized: Whether the model is quantized (i.e. fixed point values rather than
  ///       floating point values).
  /// - Returns: The RGB data representation of the image buffer or `nil` if the buffer could not be
  ///     converted.
  private func rgbDataFromBuffer(
    _ buffer: CVPixelBuffer,
    byteCount: Int
    ) -> Data? {
    CVPixelBufferLockBaseAddress(buffer, .readOnly)
    defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) }
    guard let mutableRawPointer = CVPixelBufferGetBaseAddress(buffer) else {
      return nil
    }
    let count = CVPixelBufferGetDataSize(buffer)
    let bufferData = Data(bytesNoCopy: mutableRawPointer, count: count, deallocator: .none)
    var rgbBytes = [Float](repeating: 0, count: byteCount)
    var index = 0
    for component in bufferData.enumerated() {
      let offset = component.offset
      let isAlphaComponent = (offset % alphaComponent.baseOffset) == alphaComponent.moduloRemainder
      guard !isAlphaComponent else { continue }
      rgbBytes[index] = Float(component.element) // 255.0
      index += 1
    }
    
    return rgbBytes.withUnsafeBufferPointer(Data.init)

  }
}

// MARK: - Extensions

extension Data {
  /// Creates a new buffer by copying the buffer pointer of the given array.
  ///
  /// - Warning: The given array's element type `T` must be trivial in that it can be copied bit
  ///     for bit with no indirection or reference-counting operations; otherwise, reinterpreting
  ///     data from the resulting buffer has undefined behavior.
  /// - Parameter array: An array with elements of type `T`.
  init<T>(copyingBufferOf array: [T]) {
    self = array.withUnsafeBufferPointer(Data.init)
  }
}

extension Array {
  /// Creates a new array from the bytes of the given unsafe data.
  ///
  /// - Warning: The array's `Element` type must be trivial in that it can be copied bit for bit
  ///     with no indirection or reference-counting operations; otherwise, copying the raw bytes in
  ///     the `unsafeData`'s buffer to a new array returns an unsafe copy.
  /// - Note: Returns `nil` if `unsafeData.count` is not a multiple of
  ///     `MemoryLayout<Element>.stride`.
  /// - Parameter unsafeData: The data containing the bytes to turn into an array.
  init?(unsafeData: Data) {
    
    guard unsafeData.count % MemoryLayout<Element>.stride == 0 else { return nil }
    #if swift(>=5.0)
    self = unsafeData.withUnsafeBytes { .init($0.bindMemory(to: Element.self)) }
    #else
    self = unsafeData.withUnsafeBytes {
      .init(UnsafeBufferPointer<Element>(
        start: $0,
        count: unsafeData.count / MemoryLayout<Element>.stride
      ))
    }
    #endif  // swift(>=5.0)
  }
}
 @johnfhima, As the primary issue has been addressed, could you please create a new ticket for this  [issue](https://github.com/tensorflow/tensorflow/issues/59771#issuecomment-1447992826).

> Hi, Thanks for your help. I am currently experienced another trouble. As I told you before I am making the inference of my model using tflite and the metal delegate in swift. When I try to perform a loop of inference, it seems that the memory never stop to increase until reaching an out of memory (if there is too many input). Why does this behaviour happens? If I have enough memory to perform one inference is there a way to free the memory used during this inference before going to the next one ?
>...


Thank you! 
 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further.
Hi,
I have built a custom model using tensorflow/keras.
While looking at the amount of RAM (nvidia-smi) used to process a single input it gives me 8 gb of RAM (it is expected, as the model is purpose really heavy, and the input resolution also).

Nevertheless, after converting the model to tflite (even without any optimization), and running it on my iPadProM1 with metal delegate support for GPU acceleration, the peak of memory I observed is around 2-3 GB of RAM.
The model even succeeds to run for a reasonable amount of time on an iPhone 12.

I was wondering what I am missing. For my specific project, it is really important for us to keep the performance of the model as close as possible to the original one, even if time/RAM will be more expensive.


Why is tflite lowering the RAM of my model while running on an edge device? 


Here is the parameter used to convert the model:

converter = tf.lite.TFLiteConverter.from_keras_model(my_model)
converter.optimizations = []
tflite_model = converter.convert()
with open('lunetnoopt.tflite', 'wb') as f:
    f.write(tflite_model)



