# Preallocating Array in a Memory Efficient Way

Hi,

I need to work with a very big Array

`z=zeros(Int64,2^K,2^K,K,T)`

What is the proper way of pre-allocating this Array in such a way that it uses the least memory possible when K is large?

How big is `K`? Will most of the elements be `0`? If so, using a sparse data structure will be helpful. Also, are you sure you need to store the array? If you can get the same effect from looping over a small portion of it, that should be much more efficient.

2 Likes

K is around 15! Yes, most elements will be 0. However, most of this are 3-dimensional arrays and I could not get sparse structures to work in this case. A MWE of my problem can be found here Performance Tips for Combinatorial Problem

Is there any structure to it? Blocks, for instance?

Yes, you can think of `z=zeros(Int64,2^K,2^K,K,T)` as 2^K2^K matrices of size KT

If you don’t need all of the matrices at the same time, store them as an array of arrays, with possibly “nothing” entries.

1 Like
{"ft_lang_label":"__label__en","ft_lang_prob":0.92631084,"math_prob":0.86248803,"size":1123,"snap":"2022-27-2022-33","text_gpt3_token_len":296,"char_repetition_ratio":0.09294012,"word_repetition_ratio":0.30243903,"special_character_ratio":0.25467497,"punctuation_ratio":0.13090909,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9790311,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-08-16T10:00:25Z\",\"WARC-Record-ID\":\"<urn:uuid:5b3ea6c0-3a91-417c-9dc2-8229609639a0>\",\"Content-Length\":\"30177\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:4ce69b29-f478-43e7-9100-991f54c7562e>\",\"WARC-Concurrent-To\":\"<urn:uuid:108f1511-64ab-44cd-a6be-b949c63b67a9>\",\"WARC-IP-Address\":\"64.71.144.205\",\"WARC-Target-URI\":\"https://discourse.julialang.org/t/preallocating-array-in-a-memory-efficient-way/32206\",\"WARC-Payload-Digest\":\"sha1:D3Z37KDVRUG2FHQQCQJ6NJB3DHHNIL72\",\"WARC-Block-Digest\":\"sha1:2MF2ZSKKD7KVL54C6ATOQRFYYM327KZQ\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-33/CC-MAIN-2022-33_segments_1659882572286.44_warc_CC-MAIN-20220816090541-20220816120541-00783.warc.gz\"}"}