# Location Model Memory Growth

Hi There!

Here is a simple sample code of a location based attention. It takes into account the `last_alignment` to compute the next alignment.

``````import os

from torch import nn

import torch

class LocationAttention(nn.Module):
""" Simple mock of a location sensative alignment similar to this paper:
https://arxiv.org/pdf/1506.07503.pdf
"""

def __init__(self, hidden_size):
super().__init__()
self.project_alignment = nn.Linear(1, hidden_size)
self.score = nn.Linear(hidden_size, 1)
self.softmax = nn.Softmax(dim=1)

def forward(self, last_alignment):
last_alignment = last_alignment.unsqueeze(2)
last_alignment = self.project_alignment(last_alignment)
score = self.score(last_alignment)
score = score.squeeze(2)
last_alignment = self.softmax(score)
# last_alignment = last_alignment.detach()
return last_alignment

# Parameters
batch_size = 64
num_tokens = 150
num_iter = 3000
hidden_size = 128

# Instantiate
attention = LocationAttention(hidden_size=hidden_size)
attention.cuda()

# Run
last_alignment = torch.FloatTensor(batch_size, num_tokens).zero_().cuda()
for i in range(num_iter):  # WARNING: Uses 11 gig of memory in 1600 iterations
last_alignment = attention(last_alignment)
if i % 100 == 0:
print('Frame: %d' % (i,))
os.system('nvidia-smi')
``````

However, regardless of its simplicity and size, within less than 1600 iterations, the memory grows to 11 gigabytes.

My investigation has led me to find that adding `detach` stems the problem; therefore, the memory consumed is due to gradient information. It is not clear to me how an alignment of size `[64, 128]` grows to 11 gigabytes of memory. An LSTM, a much more complicated unit, does not use this much memory after so many iterations.

Any ideas?
{"ft_lang_label":"__label__en","ft_lang_prob":0.75557536,"math_prob":0.9051309,"size":1691,"snap":"2022-05-2022-21","text_gpt3_token_len":402,"char_repetition_ratio":0.18731476,"word_repetition_ratio":0.0,"special_character_ratio":0.2661147,"punctuation_ratio":0.18213059,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.96888447,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-05-29T09:30:32Z\",\"WARC-Record-ID\":\"<urn:uuid:569c53ac-f788-4efc-b18d-56a11ed6b82c>\",\"Content-Length\":\"13884\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:0e37ae29-4827-4b18-b152-fe7bae6b408d>\",\"WARC-Concurrent-To\":\"<urn:uuid:1e941b72-856d-4b56-bc04-46d12ec5a54f>\",\"WARC-IP-Address\":\"159.203.145.104\",\"WARC-Target-URI\":\"https://discuss.pytorch.org/t/location-model-memory-growth/17304\",\"WARC-Payload-Digest\":\"sha1:MWLC2RBULSZBWH7DYJ74GGDHQBAJ4M6T\",\"WARC-Block-Digest\":\"sha1:TU77JALVBSPKMMEG7I2KKHYV2GGSNV7U\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-21/CC-MAIN-2022-21_segments_1652663048462.97_warc_CC-MAIN-20220529072915-20220529102915-00344.warc.gz\"}"}