# 利用Redis实现SQL伸缩的方法简介

•     创建一个绑定到传入实体的哈希键(hash key)
•     使用HINCRBY使计数器值增加
•     HSET所有的LWW数据(比如 "最后一次见到的")

•     使用ZRANGE获取所有的key
•     为每一个挂起的key发起一个作业到RabbitMQ

•     在下面我们想要只弹出一个设置的数量的例子中我们将使用一组排序(举例来说我们需要那100个旧集合)。
•     假使我们为了处理一个键值来结束多道排序的作业，这个人会得到no-oped由于另一个已经存在的处理和清空哈希的过程。
•     该系统能够在许多Redis节点上不断扩展下去仅仅是通过在每个节点上安置把一个'悬置'主键来实现。

```def incr_and_check_limit(user_id, limit):
key = '{user_id}:{epoch}'.format(user_id, int(time() / 60))

pipe = redis.pipeline()
pipe.incr(key)
pipe.expire(key, 60)
current_rate, _ = pipe.execute()

return int(current_rate) > limit
```

```def incr_and_check_limit_memcache(user_id, limit):
key = '{user_id}:{epoch}'.format(user_id, int(time() / 60))

return False

current_rate = cache.incr(key)

return current_rate > limit
```

```from contextlib import contextmanagerr = Redis()@contextmanagerdef lock(key, nowait=True):
while not r.setnx(key, '1'):
if nowait:
raise Locked('try again soon!')
sleep(0.01)

# limit lock time to 10 seconds
r.expire(key, 10)

# do something crazy
yield

# explicitly unlock
r.delete(key)
```

{
"<type enum>:<epoch>:<shard number>": {
"<id>": <count>
}}

{
"1:1399958363:0": {
"1": 53,
"2": 72,
}}

"1:1399958363:0:1": 53

1.     我们可以将所有的键设为一次性的（这也可能产生负面影响，但是目前为止是稳定的）
2.     大幅压缩键值，这是相当重要的处理

```def get_range(self, model, keys, start, end, rollup=None):
"""  To get a range of data for group ID=[1, 2, 3]:  Start and end are both inclusive.  >>> now = timezone.now()  >>> get_keys(tsdb.models.group, [1, 2, 3],  >>>     start=now - timedelta(days=1),  >>>     end=now)  """
normalize_to_epoch = self.normalize_to_epoch
normalize_to_rollup = self.normalize_to_rollup
make_key = self.make_key

if rollup is None:
rollup = self.get_optimal_rollup(start, end)

results = []
timestamp = end
with self.conn.map() as conn:
while timestamp >= start:
real_epoch = normalize_to_epoch(timestamp, rollup)
norm_epoch = normalize_to_rollup(timestamp, rollup)

for key in keys:
model_key = self.get_model_key(key)
hash_key = make_key(model, norm_epoch, model_key)
results.append((real_epoch, key, conn.hget(hash_key, model_key)))

timestamp = timestamp - timedelta(seconds=rollup)

results_by_key = defaultdict(dict)
for epoch, key, count in results:
results_by_key[key][epoch] = int(count or 0)

for key, points in results_by_key.iteritems():
results_by_key[key] = sorted(points.items())
return dict(results_by_key)
```

•     生成所必须的键。
•     使用工作区，提取所有连接操作的最小结果集（Nydus负责这些）。
•     给出结果，并且基于指定的时间间隔内和给定的键值将它们映射到当前的存储区内。
{"ft_lang_label":"__label__zh","ft_lang_prob":0.8198478,"math_prob":0.9570484,"size":3982,"snap":"2020-24-2020-29","text_gpt3_token_len":2381,"char_repetition_ratio":0.08647562,"word_repetition_ratio":0.02020202,"special_character_ratio":0.23304872,"punctuation_ratio":0.1949807,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9652318,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-06-03T04:09:05Z\",\"WARC-Record-ID\":\"<urn:uuid:4e1416ad-2bee-417a-bb9e-b38211775ba5>\",\"Content-Length\":\"20890\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:19d80591-441c-42db-9dd6-718801134f7b>\",\"WARC-Concurrent-To\":\"<urn:uuid:970ab4b8-9b03-492c-b71e-1e540e8772ad>\",\"WARC-IP-Address\":\"122.246.20.171\",\"WARC-Target-URI\":\"http://o.xker.com/page/e2015/06/202994.html\",\"WARC-Payload-Digest\":\"sha1:R7Z2ISW5HZAEPLR6SEXYDDOM2ZPZZDOR\",\"WARC-Block-Digest\":\"sha1:RXX66RKDNCAQQJPNXV6FZZSEYPHXCIFI\",\"WARC-Identified-Payload-Type\":\"application/xhtml+xml\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-24/CC-MAIN-2020-24_segments_1590347428990.62_warc_CC-MAIN-20200603015534-20200603045534-00292.warc.gz\"}"}