## Background

ApsaraDB for Redis provides complex data structure types such as lists, hash tables, and sorted sets. When you use the service, improper key design may result in large keys. Due to Redis single-thread model, the operation to obtain or delete large keys may affect the service. In a cluster, large keys are prone to run out of memory of a certain node. Therefore, you can use a search tool to find large keys.

To scan for large keys of ApsaraDB for Redis, run the `SCAN` command for the master-replica edition or `ISCAN` command for the cluster edition. To obtain the number of nodes, run the `INFO` command. The command rules are as follows:

``ISCAN idx cursor [MATCH pattern] [COUNT count]``

In this command, idx specifies the node ID that starts from 0. For an eight-node cluster instance of 16 GB to 64 GB, idx ranges from 0 to 7. A 128 GB or 256 GB cluster instance contains 16 nodes.

## Procedure

``wget "https://pypi.python.org/packages/68/44/5efe9e98ad83ef5b742ce62a15bea609ed5a0d1caf35b79257ddb324031a/redis-2.10.5.tar.gz#md5=3b26c2b9703b4b56b30a1ad508e31083"``
2. Decompress the package and install the Python client.
`````` tar -xvf redis-2.10.5.tar.gz
cd redis-2.10.5
sudo python setup.py install``````
3. Create the following scanning script:
`````` import sys
import redis
def check_big_key(r, k):
bigKey = False
length = 0
try:
type = r.type(k)
if type == "string":
length = r.strlen(k)
elif type == "hash":
length = r.hlen(k)
elif type == "list":
length = r.llen(k)
elif type == "set":
length = r.scard(k)
elif type == "zset":
length = r.zcard(k)
except:
return
if length > 10240:
bigKey = True
if bigKey :
print db,k,type,length
for k in r.scan_iter(count=1000):
check_big_key(r, k)
def find_big_key_sharding(db_host, db_port, db_password, db_num, nodecount):
cursor = 0
for node in range(0, nodecount) :
while True:
iscan = r.execute_command("iscan",str(node), str(cursor), "count", "1000")
for k in iscan:
check_big_key(r, k)
cursor = iscan
print cursor, db, node, len(iscan)
if cursor == "0":
break;
if __name__ == '__main__':
if len(sys.argv) ! = 4:
print 'Usage: python ', sys.argv, ' host port password '
exit(1)
db_host = sys.argv
db_port = sys.argv
nodecount = r.info()['nodecount']
keyspace_info = r.info("keyspace")
for db in keyspace_info:
print 'check ', db, ' ', keyspace_info[db]
if nodecount > 1:
4. Run the `python find_bigkey <host> 6379 <password>` command to search for large keys.
{"ft_lang_label":"__label__en","ft_lang_prob":0.6789463,"math_prob":0.8378018,"size":3537,"snap":"2020-24-2020-29","text_gpt3_token_len":976,"char_repetition_ratio":0.124257006,"word_repetition_ratio":0.008048289,"special_character_ratio":0.29488266,"punctuation_ratio":0.1972028,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9525066,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-07-06T18:03:44Z\",\"WARC-Record-ID\":\"<urn:uuid:ea6cfd0a-eef8-4458-a341-6b5ecca03d12>\",\"Content-Length\":\"19128\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:b24222f1-0221-4777-8878-3aa104b6ffe7>\",\"WARC-Concurrent-To\":\"<urn:uuid:6097f4b1-65c0-4d1e-8015-9cdf971e42f2>\",\"WARC-IP-Address\":\"47.88.73.19\",\"WARC-Target-URI\":\"https://www.alibabacloud.com/help/faq-detail/56949.html\",\"WARC-Payload-Digest\":\"sha1:HB7CEZFSSRSARMWPTJ4NZCVZQIPPCUJR\",\"WARC-Block-Digest\":\"sha1:P57KP53ZJFYDGADBWNN3PILDNA7XY2QT\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-29/CC-MAIN-2020-29_segments_1593655881763.20_warc_CC-MAIN-20200706160424-20200706190424-00097.warc.gz\"}"}