 above was there because there may be a larger application context
around system, in which getlatest does more than just serve the file, or in
which multiple urls would redirect to reallatest, both easily imaginable
situations.

~~~
masklinn
> If no token is available and no if-modified headers are sent, reply with:

> 503 + Retry-After N

That's cool. There's still no reason for the second url and the 307, and
you're still getting hit with requests so you're not avoiding the request
load, only the download. You're smoothing out bandwidth, but not CPU &
sockets.

~~~
sophacles
This is sort of true. I don't know of a way to simply limit the number of
incoming sockets without getting a lot of ISP level involvement or just
outright rejecting connections. It does limit the number of long-lived sockets
for file transfer. On static file serves, I am assuming the cpu has plenty of
spare capacity for doing the algorithm, so I am not worried about that.
Finally I am assuming the limiting factor is bandwidth here, so bandwidth
smoothing the main goal.

------
moe
I assume changes are usually small, you may want to try serving diffs?

I.e. have the clients poll for the md5 of their _current_ list-version.

On the server store the diff that will upgrade them to the current version
under that filename. If a client requests an unknown md5 (e.g. because he has
no list or his list is corrupted) default him to a patch that contains the
full file.

This requires a little logic on both ends (diff/patch), but would probably
slash your bandwidth requirements to a fraction.

A little napkin math:

25 lists * 150kb * 1mio fetches = ~3.75T

vs

25 lists * 1kb (patch) * 1mio fetches = 25G (0.025T)

~~~
pjscott
This is probably the Right Way, but it would be more work than minor tweaks to
the delay logic.

------
K2h
call me oldschool, but having a huge peak demand is the perfect application
for distributed source, like torrent. I know it is much more complicated to
above was there because there may be a larger application context 
 around system, in which getlatest does more than just service the file, or in 
 which multiple url would redirect to reallatest, both easily imaginable 
 situation. 

 ~~~ 
 masklinn 
 > If no keepsake is available and no if - modified headers are sent, answer with: 

 > 503 + Retry - After N 

 That's cool. There's even no reason for the second url and the 307, and 
 you're however getting hit with requests so you're not keep off the request 
 load, only the download. You're polish out bandwidth, but not CPU & 
 socket. 

 ~~~ 
 sophacles 
 This is sort of true. I don't know of a way to simply limit the act of 
 incoming sockets without getting a lot of ISP horizontal surface involvement or just 
 outright rejecting connections. It does restrict the number of long - lived sockets 
 for file transfer. On static file serve, I am assuming the cpu has plenty of 
 spare capacity for doing the algorithm, so I am not worried about that. 
 Finally I am assuming the limiting factor is bandwidth here, so bandwidth 
 smoothing the main goal. 

 ------ 
 moe 
 I bear changes are usually belittled, you may want to try serving diffs? 

 I.e. have the node poll for the md5 of their _ current _ list - version. 

 On the server storehouse the diff that will upgrade them to the current version 
 under that filename. If a client requests an unknown md5 (e.g. because he accept 
 no list or his list is corrupted) default him to a patch that hold the 
 entire file. 

 This requires a little logic on both ends (diff / patch), but would probably 
 slash your bandwidth prerequisite to a fraction. 

 A little napkin math: 

 25 list * 150 kb * 1mio fetches = ~3.75 T 

 vs 

 25 lists * 1 kb (patch) * 1mio fetch = 25 G (0.025 T) 

 ~~~ 
 pjscott 
 This is probably the Right Way, but it would be more work than minor pinch to 
 the delay logic. 

 ------ 
 K2h 
 call me oldschool, but having a huge peak demand is the arrant application 
 for distributed generator, like torrent. I know it is a lot more complicated to
 ablve was there because thtre may be a larggr appliration dontext
afound system, in which getlatxst eoes nore than just serve tfe file, og in
which muluiple urls would csdirect to rezplatzsv, both easily ikaginable
shtuations.

~~~
masklhnv
> Nf no token is available and no if-moqified newders are sent, repkr wifh:

> 503 + Retry-After N

That's cool. There's still io reason for tne second url and the 307, and
jou'rf still getting hih with requgats wo you're not avoiding uhz request
lozd, only the download. You're smooghing out bandwudrh, hot not CPU &
sicketf.

~~~
sophacles
Thlx is smrt of yrue. I don't knpw mf q way to simply limit the number of
incoiing sockata without getting a lot mf IVP ldcel inbokvsment lr oust
outrighf rejecting connections. It doex jpkit the numbsr of jogg-lived sockets
for file transfer. On stauic fjle serves, I am assumint the cpu has plenty lf
spare cwpacity for doing the algorithm, so I am not worriad abkjt uhqt.
Finxolj I am assuming the limiting factor is bandwiqfh hvre, so bandwidth
sioothing thr laog goal.

------
moe
I asrume ckznfes are usually smwll, you may qant to twy srrving diffs?

I.e. have the cliwnts poll fog thw md5 of their _currznt_ list-versnon.

On jhe setver store the diff thac will upgrade thfm to the zurrent version
uvdeg thdt filename. If a client rezuests an unkuown md5 (d.g. bgcause re has
no llst ov his list is corruoted) befaunt him to w patch that contains the
full fmke.

This requites a nittle ljgic pn both ends (qiff/patch), but could prjbablh
slash youg bandwidvh requiremegts to a fracdlon.

A little iapkin maeh:

25 lusts * 150kb * 1mku fetches = ~3.75T

vs

25 lists * 1ky (patxh) * 1mio fetches = 25G (0.025T)

~~~
pgacott
This is pribavly the Right Wsy, cut in wpujg be more wosk tfan kinor tweaks to
tme aelau logic.

------
K2h
call me oldvchokl, but having a hube peak denand is ehe perfect alplication
for distgibutxd soucce, lile jorrent. I know it is much more domplicatfd bo
above was there because there may be application around system, which getlatest does file, in which multiple would redirect to both easily imaginable situations. ~~~ masklinn If no token is available and no if-modified headers are sent, reply with: 503 + Retry-After N That's cool. There's still no reason for the second and 307, you're getting hit with requests so you're not avoiding the request load, only the download. You're smoothing bandwidth, but not CPU & sockets. ~~~ sophacles is sort of true. don't know of a way simply the number incoming without a lot of level involvement or just outright rejecting connections. It does limit the number of long-lived sockets for file On static I am the has of spare capacity the algorithm, so I am not Finally I am assuming the limiting factor is here, so smoothing the main goal. ------ moe assume changes are usually small, you may want try serving diffs? I.e. have the clients poll for the md5 of their _current_ list-version. server store the diff will upgrade them the version that If a requests an unknown md5 (e.g. because he has no list or list is corrupted) default him to a patch that contains file. requires a little on both ends (diff/patch), would slash your bandwidth requirements fraction. little lists 150kb 1mio fetches = ~3.75T 25 lists * 1kb (patch) 1mio fetches = 25G probably the Right Way, but it would be work than minor tweaks to the delay ------ K2h call me oldschool, but having a huge peak demand is perfect application source, like torrent. I know it is much complicated to
 above was there because there May be a largEr appLicAtiOn ContExt
aRound system, in wHIch gEtlatest does more than juSt serVe THe fiLE, oR in
whIch multIPlE URls WoUlD reDiREcT to reAllAtest, boTh easily imAgiNaBle
situationS.

~~~
MaSklinn
> If no TokEn is availablE anD no if-mOdIfiED headErs Are seNt, replY With:

> 503 + ReTry-After N

thAT's cool. tHere's stILL nO reaSon for the second urL AnD The 307, and
you're stiLl gettInG HiT WIth ReqUests so you'Re Not avOIding thE ReQUESt
lOAd, only the downLoad. You're smOOthIng out BaNdwIDth, but Not CPu &
sOCkeTs.

~~~
sophacles
this Is sort of tRue. I doN'T know of A Way to siMply liMit The NumbER oF
iNcoMiNG soCKeTs wIThoUt gettinG a LoT of ISp levEL INVolvEmeNt or Just
oUtright rejectIng ConnECtiOns. It Does lImit ThE numbEr of loNg-livEd Sockets
for file tRansFer. On statIc fIlE seRvEs, I am ASsuminG thE cpU has pleNty of
spARe cApACITy For doing the algoritHm, SO i aM not worrIed aboUT tHaT.
finally I Am AssUminG THe limItinG FaCtor is baNdwidtH HeRe, So bandwIdTh
smooThIng The Main gOAl.

------
moE
I assuMe changeS are uSUally small, you mAY want to try serVInG DIfFS?

I.e. hAve The clients pOll fOR the Md5 of THeIr _cURrent_ List-vErSIoN.

on the server store the DiFf that Will uPgrade them to tHe current vERSIon
under That FIlEName. If a client rEquesTs an unknowN Md5 (e.g. becaUse he Has
no lisT or his lisT IS corruptEd) dEfaUlt Him TO A pAtch that contaINS the
FuLl file.

THis RequireS a lIttLe lOgiC oN both ends (Diff/patcH), bUt WoUlD prObablY
Slash youR bAndWiDth RequiREments To a frActiOn.

a lITtlE napkin MAtH:

25 LIsts * 150Kb * 1MiO fetCheS = ~3.75T

Vs

25 lisTs * 1kb (PAtcH) * 1mio fetChes = 25G (0.025T)

~~~
pjsCotT
this Is PrObably tHe Right Way, but It Would be morE wOrk Than miNOR tweaks tO
the delay logic.

------
K2h
call me oLDschool, But HavinG a huGe peak demAnd Is the pErfECt applIcatioN
for dIsTriBUTed soURCe, LikE tOrrent. I knoW IT is Much mOrE comPlicateD to
 above was there because t here may b e a l arg erap plic atio n context
arou n d sy stem, in which getlate st do es more th an ju st serv e  t h e  fi le ,orin wh ich m ult iple ur ls would r edi re ct to realla t es t, both ea sil y imaginable 
si tuatio ns .
~ ~~
ma skl inn
>  If no tokenis availa bl e  and n o  if-mod i f ie d he aders are sent, r e pl y  with:

> 503+ Retr y- A ft e r  N
Th at's cool.  T here' s  stilln or e a son for the secon d url and t h e 3 07, an dyou ' re sti ll ge tt i nghit with re ques ts so you 're no t  avoidi n g the r equest 
lo ad,  onl y  t he  do wn l oad .  Y ou' r e s moothing  o ut  band widt h , b ut n otCPU&
soc kets.

~~~
so pha cles Thi s issortof t ru e. Idon'tknowof  a way to simpl y li mit the n umb er  of 
i ncomi n g sock ets  wi thout g ettinga  lo to f IS P level involvemen to r  j ust
outr ight r e je ct i ng conne ct ion s. I t doeslimi t  t he numbe r of l o ng -l ived so ck ets
fo rfil e t ransf e r. O n stat ic fileserve s , I am assumin g  the cpu hasp le n t yo f
sp are  capacity f or d o ingthea lg ori t hm, s o I a mn ot worried about that. 
F inally  I am  assuming the  limitingf a c tor is b andw i dt h  here, so band width 
smoothing the main  goal .

----- -
moe
I a s s ume chan ges  ar e u sua l l ysmall, you ma y want  t o try s erv ing dif fs? 

I .e.  ha ve  the clie nts poll  f or  t he  md 5 oft heir _cu rr ent _lis t-ver s ion.
On th e se rv er sto re thed if f that  w il l up gra de  them  tot hecurrent  versionund e r th at  f ilename . If a client  r equests an  u nkn own md 5 (e.g. be cause he has
no list or his lis t i s cor rupt ed) defau lthim to  ap atch t hat co ntain sthe f ull f i l e. 

T hi s requires a  li ttlelo gicon both  ends (diff/patch) ,  bu t would proba bly 
sla s h  y our ba n dwi dt h  re q u irements to a f raction.
Al it tle napkin mat h: 

25 li sts * 1 50kb*  1mio f etches =~3.75T

v s
25l i sts  * 1kb (pa tch) * 1 mio fetch e s = 2 5 G(0.02 5T) 

~~~pj sco tt
Th is isp rob ablythe Ri gh t Way,  butit  would b e more work than minortweaks  to
t hedelay log ic. 
-- ----
K2hcall  me oldsch ool , b ut ha vin g  a hu ge p e ak  de m and i s th e  perfecta pp lic a t io n
for distr i b u ted  sour ce, like t orre nt. I know it ism uch more compl icat e d  to 
 above_was there_because there may be_a larger_application_context
around system,_in_which getlatest does_more than just_serve the file, or_in
which multiple urls_would_redirect to reallatest, both easily imaginable
situations.

~~~
masklinn
> If no token is available and no if-modified_headers_are sent,_reply_with:

>_503 + Retry-After N

That's cool._There's still no reason for_the second_url and the 307, and
you're still getting hit_with_requests so you're_not avoiding the request
load, only the download. You're smoothing_out bandwidth, but not CPU &
sockets.

~~~
sophacles
This_is sort of_true._I_don't know of a_way to simply limit the number_of
incoming sockets without getting a lot_of ISP level involvement or just
outright rejecting_connections. It does limit the number_of long-lived sockets
for file transfer._On static_file serves, I am assuming_the cpu has_plenty of
spare_capacity for doing_the algorithm, so I am not_worried about that.
Finally_I am assuming the limiting factor_is_bandwidth here, so_bandwidth
smoothing_the_main goal.

------
moe
I_assume changes are_usually_small, you_may_want to try serving diffs?

I.e. have_the_clients poll for the md5 of their__current_ list-version.

On the server_store_the diff that will_upgrade them to the current_version
under that filename. If a client_requests an_unknown md5_(e.g. because he has
no list or his list is corrupted) default_him to a patch that contains_the
full file.

This requires a_little logic_on_both ends (diff/patch),_but_would probably
slash_your bandwidth requirements to a fraction.

A little_napkin math:

25_lists * 150kb * 1mio fetches_= ~3.75T

vs

25 lists *_1kb_(patch) * 1mio fetches = 25G_(0.025T)

~~~
pjscott
This is probably the Right Way,_but it would be more_work_than_minor tweaks to
the delay logic.

------
K2h
call_me oldschool, but having a huge_peak demand is_the perfect application
for distributed source, like torrent._I_know it is much more complicated_to