[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

# Heap-expansion problem

```When I run programs that allocate a lot of memory,
I sometimes get an unreasonbly large heap expansion factor,
like this:

USER(1): ;; commands to load and start program

gc: E=81% N=3262536 O+=176 pfu=130+1689 pfg=0+119
gc: XN-E=35% N=13638880 O+=0 pfu=0+56 pfg=0+1896
gc: E=58% N=12750088 O+=104 pfu=0+1084 pfg=28+8061
gc: E=12% N=44423520 O+=0 pfu=50+2158 pfg=171+19088
Error: An explicit gc call caused a need for 42729472 more bytes of heap.
This would exceed the image size limit set at initial build.
[condition type: storage-condition]

My guess is that the memory requirement grew so fast that there wasn't
time for objects to be tenured.  Though I have no idea how the new space
got from the initial 8M to more than 44M in one expansion.

What's the best way to deal with this problem?
(The program does finish if I explicitly call gc periodically.)

Also, is there a way to get at the numbers printed by (room)?
If so, then I can call gc explicitly at places where the fewest
objects are likely to survive and when free space is low.

```
{"ft_lang_label":"__label__en","ft_lang_prob":0.8421856,"math_prob":0.83763766,"size":1136,"snap":"2021-21-2021-25","text_gpt3_token_len":333,"char_repetition_ratio":0.088339224,"word_repetition_ratio":0.0,"special_character_ratio":0.32658452,"punctuation_ratio":0.09016393,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.98511946,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-06-21T19:22:28Z\",\"WARC-Record-ID\":\"<urn:uuid:d25dd4e8-6963-447e-b2d0-e330533bef85>\",\"Content-Length\":\"3584\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:9015e128-bc79-400f-ab69-572b06653b23>\",\"WARC-Concurrent-To\":\"<urn:uuid:a0462c9a-dc9a-4c51-a08a-d6d4ae69fdcd>\",\"WARC-IP-Address\":\"160.16.144.200\",\"WARC-Target-URI\":\"https://ml.cddddr.org/allegro-cl/msg00415.html\",\"WARC-Payload-Digest\":\"sha1:YYWNK3LRG5BHWJLYUXEBWOI44DXOTXDZ\",\"WARC-Block-Digest\":\"sha1:IETWH6ULP4IIYVCSA33YJUCYEEBD5HLO\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-25/CC-MAIN-2021-25_segments_1623488289268.76_warc_CC-MAIN-20210621181810-20210621211810-00052.warc.gz\"}"}