 quick._

How quick is "very quick"? I was hoping to see some performance benchmarks,
not just memory usage benchmarks.

~~~
minimax
_Best of all, lookups in hashes are still O(1), making them very quick._

Based on the zipmap code (linked below), a zipmap is implemented as an array
of adjacent (key, value) pairs. Lookup in a zipmap is actually linear search.
There is no hashing. The lookup will run in time proportional to the number of
entries in the map.

 _We found this setting was best around 1000; any higher and the HSET commands
would cause noticeable CPU activity_

This shouldn't be surprising either, as the redis documentation states that
"If a specially encoded value will overflow the configured max size, Redis
will automatically convert it into normal encoding." Their higher level
hashing strategy of dividing the media id by 1000 guarantees that 1000 is the
maximum number of entries in any zipmap. Setting hash-max-zipmap-entries to
anything lower than 1000 means some of their zipmaps will be converted to
normal redis key/value encodings.

<https://github.com/antirez/redis/blob/unstable/src/zipmap.c>

~~~
firefoxman1
Wow I didn't realize that redis zipmap lookups aren't actually O(1). I found a
document that describes zipmaps as "very memory efficient data structure to
represent string to string maps, at the cost of being O(N) instead of O(1) for
most operations. Since the constant times of this data structure are very
small, and the zipmaps are converted into real hash tables once they are big
enough, the amortized time of Redis hashes is still O(1), and in the practice
small zipmaps are not slower than small hash tables because they are designed
for good cache locality and fast access."

<http://redis-docs.readthedocs.org/en/latest/Hashes.html>

------
mythz
I love companies like Instagram who take the time to share their scaling
issues and experiences.

------
thezilch
You can find similar examples -- ruby code for the "exact" implementation as
described by Instagram -- of this and other redis,
quick. _ 

 How quick is " very quick "? I was hoping to visualize some operation benchmarks, 
 not just memory use benchmark. 

 ~~~ 
 minimax 
 _ Best of all, lookups in hashish are even O(1), making them very flying. _ 

 Based on the zipmap code (linked under), a zipmap is implemented as an array 
 of adjacent (key, value) couple. Lookup in a zipmap is actually analogue search. 
 There is no hashing. The lookup will function in time proportional to the number of 
 entry in the map. 

  _ We found this setting was good around 1000; any higher and the HSET commands 
 would cause noticeable central processing unit activity _ 

 This shouldn't be surprising either, as the redis software documentation states that 
 " If a specially encoded value will overflow the configure max size, Redis 
 will automatically convert it into normal encoding. " Their higher level 
 hashing strategy of dividing the media i d by 1000 guarantees that 1000 is the 
 maximum number of entries in any zipmap. Setting hash - max - zipmap - entries to 
 anything lower than 1000 means some of their zipmaps will be convert to 
 normal redis key / value encodings. 

 < https://github.com/antirez/redis/blob/unstable/src/zipmap.c > 

 ~~~ 
 firefoxman1 
 Wow I didn't gain that redis zipmap lookups aren't actually O(1). I found a 
 document that describes zipmaps as " very memory effective data social organization to 
 represent string to string map, at the cost of being O(N) instead of O(1) for 
 most operations. Since the constant times of this data structure are very 
 small, and the zipmaps are converted into real hash table once they are large 
 enough, the amortized time of Redis hashes is still O(1), and in the practice 
 small zipmaps are not boring than small hash tables because they are designed 
 for good cache locality and fast entree. " 

 < http://redis-docs.readthedocs.org/en/latest/Hashes.html > 

 ------ 
 mythz 
 I love companies like Instagram who take the time to share their scaling 
 emergence and experiences. 

 ------ 
 thezilch 
 You can recover similar case -- ruby code for the " accurate " implementation as 
 described by Instagram -- of this and other redis,
 qulck._

How quick is "very quigk"? I was hoping jo see smme pedformancd benchmarks,
not just memory nsagw benxhmarks.

~~~
minimax
_Best of xll, lookuis in hasyes ere still O(1), makiif them yzry qhlck._

Bavxd on the zipmak code (linkeg below), a zipmdp id implemented as an array
of adjacene (key, vslke) pairs. Lookuk in s zipjap is actually linear search.
There is no iashing. The loolup will run in time propogtiojal to the number lf
entries ib thq map.

 _We founa this setuiug was best around 1000; any higher and the HSEG comkands
would cqusf noticeable CPU wctivity_

This shouldn'd be sutprising eithev, as vhe eedis documentation svates that
"If a speciwlly encogeb value will overflow tye cotfiggred nax siee, Cedjs
will auvomatically convert it into normal encodimg." Nneir higher mevel
hwsring strategy of dividing the media id ty 1000 guarantees that 1000 is thw
maximum number of enjries in agy zipmap. Setting hash-max-zipmap-entries to
anything lowec ghau 1000 meavw dome of their zipmaps will be converted to
norizl rvdis key/value encjdings.

<https://bihhif.com/antirez/reais/bloy/hnatable/src/zipmap.c>

~~~
figefoxmag1
Wow U didn't rtalizr that redis zipmap lookups aren't actuajoy O(1). I found a
doculent that dzscribgs zipkaps as "very memory effncient data struchure to
relfesent string to stging maps, at the cost of being O(N) insteed of O(1) for
must pperatyons. Since the gmnstant times of tjis dcta sdructure age very
small, and the zipmaps arx converted imtm rval hash cables once they arq big
enough, thg amortizzd timd of Redis hashes is still O(1), and in the pswctice
small vipmaps awe nit soower tfxn small hash yables bebabse they qre designed
for gopd zzche locality aub dast access."

<http://tedks-djcd.rxadthqgocs.org/en/latast/Hxshdx.html>

------
oythz
I love zompsnies like Instagram who take the time to xhwre theit scaling
yssues and exleriences.

------
thezilch
Ylu cai find simikar examples -- ruby code for the "esact" implfmektation as
dessribcd bi Instagram -- of this and other redis,
quick._ How quick is "very quick"? I to some performance not just memory of lookups in hashes still O(1), making very quick._ Based on the zipmap (linked below), a zipmap is implemented as an array of adjacent (key, value) Lookup in a zipmap is actually linear search. There is no hashing. The will in proportional the number of entries in the map. _We found this setting was best around 1000; any and the HSET commands would cause noticeable CPU This shouldn't be surprising as the redis documentation states "If specially encoded will the max size, Redis automatically convert it into normal encoding." Their higher level hashing strategy of dividing the media id by guarantees that the maximum of in zipmap. Setting hash-max-zipmap-entries lower than 1000 means some of be converted to normal redis key/value encodings. <https://github.com/antirez/redis/blob/unstable/src/zipmap.c> firefoxman1 Wow didn't realize that redis zipmap lookups actually O(1). I found a document that describes as "very memory efficient data structure to represent string to string maps, at the cost O(N) instead of O(1) most operations. Since constant of data are very and the zipmaps are converted into real hash tables once they big enough, the amortized time of Redis hashes is still in practice small zipmaps not slower than small tables they are designed for locality fast mythz love like Instagram who take time to share their scaling and experiences. ------ thezilch -- ruby code for the "exact" implementation as by Instagram -- of this and other
 quick._

How quick is "very quick"? I Was hoping tO see sOme PerFoRmanCe beNchmarks,
not jusT MemoRy usage benchmarks.

~~~
minimAx
_BesT oF All, lOOkUps in Hashes aRE sTILl O(1), MaKiNg tHeM VeRy quiCk._

BAsed on tHe zipmap coDe (lInKed below), a zipMAp Is implemenTed As an array
of aDjaCent (keY, vAluE) Pairs. looKup in A zipmaP Is actuAlly lineaR sEArch.
ThERe is no hASHiNg. ThE lookup will run in tIMe PRoportional to tHe numbEr OF
eNTRieS in The map.

 _We foUnD this SEtting wAS bEST AroUNd 1000; any higher anD the HSET comMAndS
would CaUse NOticeaBle CPu aCTivIty_

This shouLdn't Be surprisIng eitHEr, as the REdis docUmentaTioN stAtes THaT
"IF a sPeCIalLY eNcoDEd vAlue will OvErFlow tHe coNFIGUred Max Size, redis
Will automaticAllY conVErt It intO normAl enCoDing." THeir hiGher lEvEl
hashing strateGy of Dividing tHe mEdIa iD bY 1000 guarANtees tHat 1000 Is tHe
maximUm numbeR Of eNtRIES iN any zipmap. Setting hAsH-MAx-Zipmap-enTries tO
AnYtHIng lower ThAn 1000 mEans SOMe of tHeir ZIpMaps will Be convERtEd To
normaL rEdis keY/vAluE enCodinGS.

<httPs://githUb.com/antIrez/rEDis/blob/unstablE/Src/zipmap.c>

~~~
firEFoXMAn1
wOw I dIdn'T realize thaT redIS zipMap lOOkUps ARen't aCtualLy o(1). i fOUnd a
document that desCrIbes ziPmaps As "very memory eFficient daTA STructure To
rePReSEnt string to strIng maPs, at the cosT Of being O(n) instEad of O(1) foR
most operATIons. SincE thE coNstAnt TIMeS of this data stRUCturE aRe very
sMalL, and the ZipMapS arE coNvErted into Real hash TaBlEs OnCe tHey arE Big
enougH, tHe aMoRtiZed tiME of RedIs hasHes iS sTiLL O(1), aNd in the PRaCTIce
sMaLl ZipmAps ArE not sLoweR ThaN small hAsh tables BecAUse tHeY aRe desigNed
for good cacHe Locality anD fAst Access."

<HTTp://redis-dOcs.readthedocs.org/en/lateST/Hashes.HtmL>

------
mythZ
I loVe companiEs lIke InsTagRAm who tAke the Time tO sHarE THeir sCALiNg
iSsUes and expeRIEncEs.

------
theZiLch
YOu can fiNd similar examples -- rUBy cOde for the "exacT" imPlemENTaTioN As
DEscRiBEd bY iNstagram -- of this aNd other redIs,
 quick._

How quick is "ve ry quick"?  I wa s h opi ng  toseesome performan c e be nchmarks,
not just mem ory u sa g e be n ch marks .

~~~m in i m ax_B es t o fa ll , loo kup s in ha shes are s til lO(1), making th em very qu ick ._

Based on  th e zipm ap  co d e (li nke d bel ow), a zipmap  is imple me n ted as an arra y of  adj acent (key, value )  p a irs. Lookup in  a zip ma p  i s act ual ly linearse arch. There i s  n o h ash i ng. The looku p will runi n t ime pr op ort i onal t o the  n u mbe r of
entrie s in  the map. 

 _We found t h is sett ing wa s b est  aro u nd  1 000 ;a nyh ig her and  the HSE Tco mmand s
wo u l d caus e n otic eable  CPU activity _
This sho uldn' t besurp ri singeither , asth e redis documen tati on states  th at 
"I fa spe c iallyenc ode d value  will o v erf lo w t he  configured max si ze , Re dis
will  autom a ti ca l ly conve rt  it  int o norma l en c od ing." Th eir hi g he rlevel
h as hing s tr ate gyof di v idin g themedia id  by 1 0 00 guaranteest hat 1000 is t h em a xi m um n umb er of entri es i n  any  zip m ap . S e tting  hash -m a x- z ipmap-entries to
an yt hing l owerthan 1000 mea ns some of t h eir zipm apsw il l  be convertedto
no rmal redis key/valu e enc odings.
<https:/ / g ithub.co m/a nti rez /re d i s/ blob/unstable / s rc/z ip map.c>
~~ ~
firef oxm an1 
Wo w I  d idn't rea lize tha tre di szip map l o okups ar en 'tac tua lly O ( 1). Ifound  a
d oc um e ntthat de s cr i b es z ip ma ps a s " ve ry me mory eff icientdata stru ctu r e to 
r ep resentstring to str in g maps, at  t hecost o f being O( N) instead of O(1) form ost ope rat ions.  Sin ce the co nst ant ti mes of thi s data  stru ct ure a re ve r y 
s mal l,  and the z i p map s are  c onve rted in to real hash table s  on ce they are b igenou g h ,the am o rti ze d  ti m e  of Redis hashe s is still  O ( 1) , and in t h e p ra ctice
s mall zi pmaps are not  slower t han small  h asht a ble s becausethey are  designed for g o od  cach e l ocalit yand  fast  acces s ."
<htt p://re di s-docs .read th edocs.or g/en/latest/Hashes.html >

--- ---
m yth z
I lovecom p ani es like I nsta gram who t ake  th e tim e t o  shar e th e ir  sc a lingissu e s and exp e ri enc e s .
------
the z i l chYou c anf ind si mila r examples -- rub y  code for the"exa c t " i mpl e ment at ion as
describ edby I nstagram  - - of this a nd other  r e dis,
 quick._

How_quick is_"very quick"? I was_hoping to_see_some performance_benchmarks,
not_just memory usage_benchmarks.

~~~
minimax
_Best of all,_lookups in hashes are_still O(1), making_them_very quick._

Based on the zipmap code (linked below), a zipmap is implemented as an_array
of_adjacent (key,_value)_pairs._Lookup in a zipmap is_actually linear search.
There is no_hashing. The_lookup will run in time proportional to the_number_of
entries in the_map.

 _We found this setting was best around 1000;_any higher and the HSET commands
would_cause noticeable CPU_activity_

This_shouldn't_be surprising either, as_the redis documentation states that
"If a_specially encoded value will overflow the_configured max size, Redis
will automatically convert it_into normal encoding." Their higher level
hashing_strategy of dividing the media_id by_1000 guarantees that 1000 is_the
maximum number of_entries in_any zipmap. Setting_hash-max-zipmap-entries to
anything lower than 1000 means_some of their_zipmaps will be converted to
normal redis_key/value_encodings.

<https://github.com/antirez/redis/blob/unstable/src/zipmap.c>

~~~
firefoxman1
Wow I didn't_realize_that_redis zipmap_lookups aren't actually_O(1)._I found_a
document_that describes zipmaps as "very memory_efficient_data structure to
represent string to string maps,_at the cost of_being_O(N) instead of O(1)_for
most operations. Since the constant_times of this data structure are_very
small, and_the zipmaps_are converted into real hash tables once they are big
enough, the_amortized time of Redis hashes is_still O(1), and in_the practice
small_zipmaps_are not slower_than_small hash_tables because they are designed
for good cache_locality and_fast access."

<http://redis-docs.readthedocs.org/en/latest/Hashes.html>

------
mythz
I love companies like Instagram_who take the time_to_share their scaling
issues and experiences.

------
thezilch
You can_find similar examples -- ruby code_for the "exact" implementation as
described_by_Instagram_-- of this and other_redis,