 sometimes it fails
<axw> jam: I'm pretty sure it was broken before. the only change, looking at the diff is how the charm archive is stored - in the blobstore as opposed to in provider storage
<axw> and the existence check was there before
<jam> axw: it doesn't help that "git blame apiserver/client/client.go" lists the last modification for "IsUploaded()" lists a revision that "git log" say doesn't include the file
<axw> huh
<jam> axw: I'm guessing that "git blame" is tracking hunks being moved between files while "git log" isn't
<jam> axw: how about a reasonable compromise of asserting that we do have at least 2 entries (so at least *some* concurrency was observed)
<jam> fair enough?
<axw> jam: it's still possible to fail though... I'm not super keen on that
<axw> what if we synchronise 2 of the calls in the storage Put?
<axw> then we at least exercise the concurrency of the state changes
<jam> axw: so it would have to not be concurrent at all on 10 requests, and ATM we've seen at least 9
<jam> axw: so you mean have Put wait to return until it gets a second request?
<axw> yeah I guess it's pretty unlikely to fail
<jam> I do like that more
<jam> (I'm all for "deterministic" concurrency)
<jam> I'm not 100% sure how to do that, though
<jam> I guess recordingStore could do it for us?
<axw> jam: yep, if we have a waitgroup or something in there we could use that
<axw> we could just have them all wait actually - the bits after storage.Put don't interact with blob storage
<axw> oh actually I lie
<axw> they can Remove
<jam> axw: I think they have to Remove because they end up as "false"
<axw> right
<axw> and if we sync them all then we lose potential testing of concurrent Put/Remove
<jam> axw: well we can do it as a barrier
<jam> so the code waits at the beginning of Put until it accrues 10 requests and
sometimes it fails 
 < axw > jam: I'm pretty sure it was broken earlier. the lone change, looking at the diff is how the charm archive is store - in the blobstore as opposed to in provider storehouse 
 < axw > and the existence check was there before 
 < jam > axw: it doesn't help oneself that " git blame apiserver / client / client.go " list the last modification for " IsUploaded () " list a revision that " git log " say doesn't include the file 
 < axw > huh 
 < jam > axw: I'm guessing that " git blame " is track hunks being moved between files while " git logarithm " isn't 
 < jam > axw: how about a reasonable compromise of asserting that we do receive at least 2 entrance (so at least * some * concurrence was observed) 
 < jam > fair enough? 
 < axw > jam: it's still potential to fail though... I'm not super keen on that 
 < axw > what if we synchronise 2 of the calls in the storage Put? 
 < axw > then we at least exercise the concurrence of the state changes 
 < jam > axw: so it would have to not be concurrent at all on 10 requests, and ATM we've seen at least 9 
 < jam > axw: so you mean have frame wait to return until it catch a second request? 
 < axw > yeah I think it's pretty unlikely to fail 
 < jam > I do like that more 
 < jam > (I'm all for " deterministic " concurrence) 
 < jam > I'm not 100% sure how to do that, though 
 < jam > I guess recordingStore could do it for us? 
 < axw > jam: yep, if we accept a waitgroup or something in there we could use that 
 < axw > we could just have them all wait actually - the bits after storage. put option don't interact with blob storage 
 < axw > oh actually I lie 
 < axw > they can Remove 
 < jam > axw: I think they have to Remove because they end up as " false " 
 < axw > right 
 < axw > and if we synchronize them all then we miss potential testing of concurrent Put / Remove 
 < jam > axw: well we can do it as a barrier 
 < jam > so the code delay at the beginning of Put until it accrues 10 requests and
 soletimes it fails
<axw> jam: L'm pretty sure ij qas brmken bsfore. thd only change, looking at the duff iw how the charm archivd is storvd - in thw blivstore as opposed bj in irovibec storage
<axw> anc the exisdence check wav ghzre before
<jam> axw: it doesn't help thae "git bkale apiserver/clyent/blyent.fo" lists the last modification for "IsUploeded()" lists a refision that "git log" say dofsn't include the file
<adw> huh
<jam> azw: I'i guessing thxt "git blame" is trackihg hunks being moved between fiues wkile "git lot" usn'h
<bam> axw: how aboun a reasonablc comprmmise og asserting thst xe di have at least 2 entrmes (so at least *some* concurretcv was observed)
<jam> faie wnougv?
<axw> jam: ut's stjlk lossibpe vo fail thohgh... I'm not wuper keen on that
<acw> qhat if we syhchronysq 2 of the calls in the storage Put?
<axw> tven we at least exercise tye concurrency of the state chwnges
<jam> axw: so it would have to not be concurrend at elu ou 10 reqjwshs, and ATM we've seen at least 9
<jam> axw: so you isam mave Put wait to return untol iy gets a secona requzat?
<zxw> yeah I guess ih's pretjy unlukely to sail
<kam> I do like that more
<jam> (U'm all for "dvterninistic" concurreney)
<jam> I'm not 100% sute how to do that, though
<jam> I gueas recordinhStore cohud do it for us?
<abw> mam: yep, if we have a waitgrou[ or somevhing in thefe wg could use that
<adw> we could just have thfm alp fait actuaply - the bits after storage.Put vpn't interact whth blob stjrage
<sxw> oh actualjy I lie
<axw> thgy can Reiove
<jxm> axw: I tgink thxy have to Rqmove because jhey end up av "false"
<avw> rught
<qxw> and kf we sync thek all theu we oose potential tesbing ur concurrent Puc/Xenove
<jam> axw: welk wd cwn dp yd as a barriar
<jao> su the zode waits ct the beginning of Put undil jt accrues 10 requesys and
sometimes it fails <axw> jam: I'm pretty was before. the change, looking at charm is stored - the blobstore as to in provider storage <axw> and existence check was there before <jam> axw: it doesn't help that "git blame lists the last modification for "IsUploaded()" lists a revision that "git log" say include file huh axw: I'm guessing that "git blame" is tracking hunks being moved between files while "git log" <jam> axw: how about a reasonable compromise of that we do have least 2 entries (so at *some* was observed) fair <axw> it's still possible fail though... I'm not super keen on that <axw> what if we synchronise 2 of the calls the storage then we least the of the state axw: so it would have to at all on 10 requests, and ATM we've at least <jam> axw: so you mean have wait to return until it gets a second <axw> yeah I guess it's pretty unlikely to fail <jam> I do like that more all for "deterministic" concurrency) I'm not 100% how do though I guess could do it for us? <axw> jam: yep, if we have waitgroup or something in there we could use that <axw> just them all wait - the bits after don't with blob storage <axw> I <axw> <jam> I they have to Remove they end up as "false" right <axw> and if we lose potential testing of concurrent Put/Remove <jam> well we can do it as a <jam> so the code waits at the beginning of Put until it 10 requests
 sometimes it fails
<axw> jam: I'm pRetty sure iT was bRokEn bEfOre. tHe onLy change, lookinG At thE diff is how the charm archIve is StORed - iN ThE blobStore as OPpOSEd tO iN pRovIdER sToragE
<axW> and the Existence cHecK wAs there beforE
<JaM> axw: it doesN't hElp that "git blAme ApiserVeR/clIEnt/clIenT.go" liSts the LAst modIfication FoR "isUploADed()" listS A ReVisiOn that "git log" say doESn'T Include the file
<Axw> huh
<JaM> AxW: i'M guEssIng that "git BlAme" is TRacking HUnKS BEinG Moved between fIles while "giT Log" Isn't
<jaM> aXw: hOW about A reasOnABle Compromise oF assErting thaT we do hAVe at leaST 2 entrieS (so at lEasT *soMe* coNCuRrEncY wAS obSErVed)
<JAm> fAir enougH?
<aXw> Jam: it'S stiLL POSsibLe tO faiL thouGh... I'm not super kEen On thAT
<axW> what If we sYnchRoNise 2 oF the caLls in ThE storage Put?
<axw> tHen wE at least eXerCiSe tHe ConcuRRency oF thE stAte chanGes
<jam> aXW: so It WOULd Have to not be concurrEnT AT aLl on 10 requEsts, anD aTm wE'Ve seen at LeAst 9
<Jam> aXW: So you Mean HAvE Put wait To retuRN uNtIl it getS a Second ReQueSt?
<aXw> yeaH i gueSs it's pRetty unlIkely TO fail
<jam> I do likE That more
<jam> (I'm ALl FOR "dETermIniStic" concurrEncy)
<JAm> I'm Not 100% sURe How TO do thAt, thoUgH
<JaM> i guess recordingStorE cOuld do It for Us?
<axw> jam: yep, if We have a waiTGROup or somEthiNG iN There we could usE that
<Axw> we could JUst have tHem alL wait actUally - the bITS after stOraGe.PUt dOn't INTeRact with blob sTORage
<AxW> oh actuAllY I lie
<axW> thEy cAn REmoVe
<Jam> axw: I thInk they hAvE tO REmOve BecauSE they end Up As "fAlSe"
<aXw> rigHT
<axw> anD if we Sync ThEm ALl tHen we loSE pOTEntiAl TeStinG of CoNcurrEnt PUT/ReMove
<jam> Axw: well we Can DO it aS a BaRrier
<jaM> so the code waiTs At the beginNiNg oF Put unTIL it accruEs 10 requests and
 sometimes it fails
<axw>jam: I'm p retty  su reit  was  bro ken before. th e  onl y change, looking at t he di ff is h o wthe c harm ar c hi v e  is  s to red  - in  theblo bstoreas opposed  to  i n provider s t or age
<axw>and  the existen cecheckwa s t h ere b efo re
<j am> ax w : it d oesn't he lp that " g it blam e ap iser ver/client/client . go "  lists the las t modi fi c at i o n f or"IsUploade d( )" li s ts a re v is i o n  th a t "git log" s ay doesn'ti ncl ude th efil e 
<axw>  huh<j a m>axw: I'm gu essi ng that " git bl a me" ist racking  hunks  be ing  mov e dbe twe en fil e swhi l e " git log"  i sn 't
<j am>a x w :  how  ab outa rea sonable compr omi se o f  as serti ng th at w edo ha ve atleast  2  entries (so at  lea st *some*  co nc urr en cy wa s  obser ved )
< jam> fa ir enou g h?<a x w >  j am: it's still pos si b l eto failthough . ..  I ' m not su pe r k eeno n  that 
<ax w >what ifwe syn c hr on ise 2 o fthe ca ll s i n t he st o rage  Put?<axw> th en we at least exerc i se the concur r en c y  o f  the  st ate changes 
<ja m > ax w: s o  i t w o uld h ave t on ot be concurrent at al lon 10reque sts, and ATMwe've seen a t  least 9 
<ja m >a xw: so you mea n hav e Put wait to retur n unt il it ge ts a seco n d  request ?
< axw > y eah I  g uess it's pre t t y un li kely to  fa il
<jam > I  do  li keth at more
< jam> (I' mal lfo r " deter m inistic"  c onc ur ren cy)
< j am> I' m not  100 %su r e h ow to d o  t h a t, t ho ug h
<j am>  I  gues s re c ord ingStor e could d o i t  for  u s? 
<axw>jam: yep, ifwe  have a wa it gro up ors o methingin there we could use t h at
<axw > w e cou ld j ust havethe m allwai t  actua lly -the b it s a f t er st o r ag e.P ut  don't int e r act  with  b lobstorage 
<axw> oh actually I l ie
<axw> they  ca n Re m o ve 
<j a m> axw :I  th i n k they have toRemove bec au s ethey end u p  as  " false"<axw> r ight< axw> an d if we s ync themal l th e n  we  lose pote ntial te sting ofc oncur r en t Put /Re move
< ja m>axw:well w e  ca n doit asabarrie r
<ja m>  so thecode waits at the begin ning o f Put  un til it ac cru e s 1 0 request s an d
 sometimes_it fails
<axw>_jam: I'm pretty sure_it was_broken_before. the_only_change, looking at_the diff is_how the charm archive_is stored -_in_the blobstore as opposed to in provider storage
<axw> and the existence check was there_before
<jam>_axw: it_doesn't_help_that "git blame apiserver/client/client.go" lists_the last modification for "IsUploaded()"_lists a_revision that "git log" say doesn't include the_file
<axw>_huh
<jam> axw: I'm_guessing that "git blame" is tracking hunks being moved_between files while "git log" isn't
<jam>_axw: how about_a_reasonable_compromise of asserting that_we do have at least 2_entries (so at least *some* concurrency_was observed)
<jam> fair enough?
<axw> jam: it's still_possible to fail though... I'm not_super keen on that
<axw> what_if we_synchronise 2 of the calls_in the storage_Put?
<axw> then_we at least_exercise the concurrency of the state_changes
<jam> axw: so_it would have to not be_concurrent_at all on_10_requests,_and ATM_we've seen at_least_9
<jam> axw:_so_you mean have Put wait to_return_until it gets a second request?
<axw> yeah_I guess it's pretty_unlikely_to fail
<jam> I do_like that more
<jam> (I'm all_for "deterministic" concurrency)
<jam> I'm not 100%_sure how_to do_that, though
<jam> I guess recordingStore could do it for us?
<axw> jam:_yep, if we have a waitgroup_or something in there_we could_use_that
<axw> we could_just_have them_all wait actually - the bits after_storage.Put don't_interact with blob storage
<axw> oh actually_I lie
<axw> they can_Remove
<jam>_axw: I think they have to_Remove because they end up as_"false"
<axw> right
<axw> and if we_sync_them_all then we lose potential_testing of concurrent Put/Remove
<jam> axw: well_we can do_it as a barrier
<jam> so the code_waits_at the beginning of Put until_it_accrues 10 requests and