#juju-dev 2012-07-02 <Aram> morning. <TheMue> Morning <davecheney> howdy <Aram> hey. <fwereade> davecheney, Aram, TheMue, morning <davecheney> fwereade: morning <davecheney> fwereade: looking forward to your testing change getting merged, we have a lot of overlap <fwereade> davecheney, just saw your mail <fwereade> davecheney, I'm feeling very conflicted ovr it <fwereade> davecheney, part of me is saying "just break it into 6 separate CLs" <davecheney> fwereade: i don't think that will reduce the wall time of merging it <fwereade> davecheney, but a bigger part is saying "this is really all one change: it's actually going to make it harder and take longer" <fwereade> davecheney, cool, glad it seems that way to you too :) <davecheney> fwereade: if there are improvements to be made, they can be done after this change it merged <fwereade> davecheney, let's hope niemeyer sees it that way :) <davecheney> aye, there's the rub <TheMue> Hmm, once again morning, connection seems to be broken. <fwereade> TheMue, heyhey <TheMue> fwereade: How has the weekend been? <fwereade> TheMue, very nice thanks <fwereade> TheMue, went to a charmingly low-rent charity event on saturday -- bunch of different foods and presentations from various nationalities <davecheney> have a nice evening folks, i'll be online later <TheMue> fwereade: We've been outside a lot, at friends on Saturday and in our own garden and a park on Sunday. <fwereade> TheMue, lovely :) <TheMue> fwereade: Enjoyed it a lot. <fwereade> TheMue, I bet, sounds pleasingly idyllic <TheMue> fwereade: Yep. We've got a park here near our home town which has founded for larger exhibition more than 10 years ago. And since then we're several times there each year. <TheMue> fwereade: https://plus.google.com/photos/107694490695522997974/albums/5760290760673390033 shows some pics from yesterday <fwereade> TheMue, awesome! <fwereade> TheMue, Malta is a bit lacking in green spaces, they're probably what I miss most <TheMue> fwereade: Hehe, yes, can imagine. Your change hasn't been small. Where in England did you lived before? <fwereade> TheMue, london, which is pretty green really all things considered <fwereade> TheMue, I grew up in the countryside in gloucestershite though so that's really what I feel is a "correct" environment, if you know what I mean <fwereade> er, gloucestershi*r*e <TheMue> fwereade: *rofl* <fwereade> TheMue, it's a nice place, a little village in the costwolds called eastleach <TheMue> fwereade: I sadly hadn't the chance to visit Britain yet, but there are so many places I want to see. <TheMue> fwereade: From the crowded London to the Outer Hebrides. <fwereade> TheMue, never been on the outlying islands <fwereade> TheMue, been camping by the sea pretty far north in scotland <TheMue> fwereade: So far I've only been in Heathrow for transit. *lol* <fwereade> TheMue, haha <fwereade> TheMue, the lake district is gorgeous <TheMue> fwereade: We've seen so many fantastic locations in TV. I think we'll take a longer time and a Land Rover Defender and then cruise all over UK. <fwereade> TheMue, I can think of worse ways to spend a month :) <TheMue> fwereade: I would also take an other great Britain car, but that would be a bit expensive: An Aston Martin. <fwereade> TheMue, haha <TheMue> fwereade: I've got a part of a single malt cask on the Isle of Arran. We bought it a few years ago with some friends. <fwereade> TheMue, I remember talking about it in budapest :) <TheMue> fwereade: I can't hide that I've got a passion for the British culture, even that I've not been there. Funny, isn't it? <fwereade> TheMue, it has good bits and bad bits, but the good bits somehow seem to have generated a lot of good PR ;) <TheMue> fwereade: You definitely have better insides than me. OTOH I'm only a tourist. :D <TheMue> fwereade: I'm off for about an hour, routine visit at the dentist. <hazmat> morning folks, tool question, does lbox submit -adopt notice local modificattions you've had to make to someone else's branch? <hazmat> it does <Aram> I hate it when I fix bugs but I don't understand my own fix. <Aram> heh. <Aram> go it. <Aram> wow, this one was subtle. <hazmat> fwereade, is there any docs extant for various cli changes in gojuju ? <hazmat> actually even an auto generated go docs at a public site would be nice <fwereade> hazmat, heyhey <fwereade> hazmat, --help will tell you what exists, but it is not otherwise explicitly available anywhere; that is a sensible idea * hazmat works through compiling the tree <hazmat> oh.. install is now get <hazmat> bson in charm urls? <hazmat> has install always been get. its been a while i guess <hazmat> oh.. the bson is the incremental serialization work <hazmat> for mongo <hazmat> i thought most of that was in mstate <fwereade> hazmat, sorry, I am missing a little bit of context <hazmat> fwereade, ./cloudinit_test.go:210: undefined: Commentf ? do i need a more recent version of go? or am i missing a lib <fwereade> hazmat, that's in gocheck <hazmat> fwereade, shouldn't i have gotten an import error then? <fwereade> hazmat, probably out-of-date <fwereade> hazmat, `go get -u launchpad.net/juju-core/...`, I think (but make sure you don;t have local changes you want to keep if you do this) <fwereade> hazmat, also worth checking what version of go you have <hazmat> 1.0.1 <hazmat> fwereade, re go get juju-core what's the ... ? <Aram> literal ... <hazmat> sure.. just curious what it meant <fwereade> hazmat, and everything underneath it <hazmat> ah <hazmat> fwereade, thanks <fwereade> hazmat, (or imported by it, but that's go get's work not the ...'s, if you see what I mean) <fwereade> hazmat, not sure whether 1.0.1 will pick all the latest library versions, if you still have trouble consider updating that <fwereade> hazmat, offhand, do you have any idea of the approximate range of ratios between zookeeper time and wall clock time at the client end of a specific connection? <fwereade> hazmat, s/zookeeper time and/the apparent rates of progression of zookeeper time and of/ <hazmat> fwereade, not sure what you mean.. notifications from zk are delivered in order <hazmat> but the delay in delivery is subject to the quality of the network connection <fwereade> hazmat, they are delivered in order but 2 conns do not necessarily have the same idea of "now" <hazmat> fwereade, right <fwereade> hazmat, not just that, I'm pretty sure the docs state that two conns can be out of syn by order-of 10s of seconds <hazmat> fwereade, each conn is an independent view of the ordered stream <fwereade> hazmat, quite so; this indicates that a client may see two events that are 100ms apart in ZK time arrive only 50ms appart on wall clock time <fwereade> hazmat, or possibly 1ms apart <hazmat> fwereade, that sounds odd, but given a conn on a separate server of the zk quorum that's a little out of date perhaps.. the conn can request the server catch up via sync <hazmat> fwereade, in what context is that an issue? <fwereade> hazmat, yeah, I know about sync, it's not in gozk AFAICS <hazmat> are you trying to have multiple conns in a single server? <hazmat> er. process <fwereade> hazmat, testing of the presence nodes using two separate connections <hazmat> fwereade, the possibility is greatly diminished of long deltas in independent views of time if their connected to the same server and the same network quality <hazmat> fwereade, there are numerous tests in txzk that i've looped 10s of k that exercise multiple conns fwiw <hazmat> fwereade, is this a theoretical concern or something you've seen in practice? <fwereade> hazmat, ok; but I have directly observed an alternate connection, running in test X, to have a conception of ZK "now" that corresponds to a state that was current during a previous test <hazmat> fwereade, are the tests building state across tests? <fwereade> hazmat, this happens very unpredictably <fwereade> hazmat, no <fwereade> hazmat, the main connection nukes everything between test cases <hazmat> so it sounds like a test framework issue then with cleanup <hazmat> fwereade, or perhaps a bug in gozk delivering events on closed conns <hazmat> fwereade, does the conn get closed/open for teardown/setup? <fwereade> hazmat, no; I'll try doing that <hazmat> fwereade, if it doesn't then yes.. its quite possible your getting event delivery on old state <fwereade> hazmat, but based on what I've seen that will only add more uncertainty... <hazmat> keep in mind you've only got one execution thread <fwereade> hazmat, new conns get events from older state <hazmat> fwereade, was the old conn closed? <fwereade> hazmat, no, the old conn is still being used to perform new operations, which I expect the alternate conn to respond to <fwereade> hazmat, or vice versa in some tests <hazmat> fwereade, that sounds like an event dispatch issue with events <hazmat> i'd try instrumenting gozk <fwereade> hazmat, is it not behaviour consistent with the somewhat loose guarantees that ZK makes? <hazmat> and printing out with handle info <fwereade> hazmat, yeah, I will look into that, it's perfectly plausible <hazmat> fwereade, old events on new conns? no <hazmat> that's not part of the guarantee, the event only happens on state observation <hazmat> and temporarily if that state is new, it should never see an old event <fwereade> hazmat, potentially old view of history implies potentially old state changes, doesn't it? <hazmat> ie. seeing old event on new state, would be a violation of the guarantee zk makes <fwereade> hazmat, does a new conn guarantee up-to-date view of history? <hazmat> fwereade, temporarily its should be up to date <fwereade> hazmat, it explicitly does not AIUI <hazmat> fwereade, connected to the same server yes it does <hazmat> fwereade, the only exception is if your running a quorum of servers <hazmat> and connecting to a not quite up to date server with the new conn <fwereade> hazmat, hold on though: it guarantees that a single client will only see a single view of history, and that that view is independent of the server it connects to <hazmat> its not eventually consistent <fwereade> hazmat, therefore, surely, it is possible that two clients connected to the same server may have an alternate view of history <hazmat> fwereade, feel free to verify, but it sounds like a code bug not a zk bug <hazmat> fwereade, the state is in memory on the zk server and modified by each op <hazmat> fwereade, and flushed to disk, a new client, will see current state <hazmat> as i said there are exceptions, but not to a single server setup <fwereade> hazmat, Single System Image <fwereade> A client will see the same view of the service regardless of the server that it connects to. <hazmat> and even then the limiting factor is the overall speed of the quorum to propagate changes <hazmat> fwereade, history doesn't exist from a client perspective, there is only present state and future observation <fwereade> hazmat, yes: but for it to have the single system image, history surely *must* exist at the server level? <hazmat> fwereade, yes.. but your asking about the delta between multiple clients <hazmat> fwereade, it does but its not exposed <hazmat> fwereade, and its only the delta on disk, not in mem <fwereade> hazmat, will look into it further, but still unconvinced that the fact the server is standalone guarantees consistency of client connections <hazmat> fwereade, the watch notifications for the client are in mem and are queued up <hazmat> again the notification carries no state <hazmat> only the change info, observation is required to capture state <hazmat> fwereade, feel free to verify, but it sounds like a code bug not a zk bug <hazmat> fwereade, the zk lists are pretty helpful <fwereade> hazmat, sure, that's the plan -- but I'm not actually claiming a ZK bug, I'm just claiming that this surprising behaviour is not actually inconsistent with the guarantees made by the docs <hazmat> fwereade, that a new client sees non current state <hazmat> against a single server <hazmat> i don't think so <fwereade> hazmat, I agree that the explanation for this bit: <fwereade> Sometimes developers mistakenly assume one other guarantee that ZooKeeper does not in fact make. This is: <fwereade> Simultaneously Conistent Cross-Client Views <fwereade> hazmat, *does* always mention multiple servers <hazmat> fwereade, like i said.. on a single server.. that's not possible.. we have many tests in python to verify that <hazmat> but i'm just repeating myself at this point... <hazmat> fwereade, and the form of consistency i'm referencing is weaker than whats in there <fwereade> hazmat, ok, I misunderstood your statement that you had verified this precise behaviour, and I accept your diagnosis of the likely cause; that is why I'm looking into it ;) <fwereade> hazmat, the actual original question I asked though is different, and does potentially involve multiple servers <fwereade> hazmat, and client connections from separate machines <hazmat> fwereade, perhaps you should backup and explain what the goal is? <fwereade> hazmat, the goal is to understand what could be causing unpredictable test failures in which two separate zk connections on the client are respectively seeing snapshots of state that appear indicate that deleting a node on conn A does not guarantee that the next request for state made on conn B will see that the node has been deleted <hazmat> fwereade, has the delete node op completed before B makes a request? <hazmat> fwereade, and you don't want to restrict to single server? <fwereade> hazmat, the calls are performed synchronously; my understanding is that the call completing without error indicates that the operation has completed successfully <fwereade> hazmat, in the general case of the problem, assuming multiple zookeeper, my initial question about relative rates of time progression may be relevant, but just for now we can worry about single servers <hazmat> fwereade.. well its still likely async.. there are two results to check <hazmat> the api call, and the result call <hazmat> given that you've got a single server (*ignoring multi server for a moment) for your tests it would appear to be a bug in gozk <hazmat> the unpredictable nature of the failures reinforces that guess, namely that something isn't properly waiting on results <fwereade> hazmat, ok, so, by "still likely async", do you mean "a line of code immediately following `err := conn.Delete("/some/path"); c.Assert(err, IsNil)` is not guaranteed to see the change" <hazmat> fwereade, you need to go deeper <hazmat> fwereade, conn.Delete is what? <hazmat> a gozk binding to the libzk <hazmat> underneath the hood its doing what <fwereade> hazmat, that is what I intend to do <hazmat> i'd guess adelete <hazmat> which is async * hazmat takes a look <hazmat> interesting <fwereade> hazmat, zoo_delete which I presume is not adelete <Aram> Delete does zoo_delete which is synchronous. <Aram> gozk is sync. <hazmat> wow <hazmat> ok.. <hazmat> fwereade, so i'd suggest instrumenting delete and the subsequent client op with some prints <Aram> I suspect you simply check against the wrong version. <Aram> that's why you don't see the change. * Aram didn't read much of the backlog. <hazmat> so gozk is sync, and gojuju runs with a single thread ? <hazmat> Aram, interesting idea <hazmat> Aram, versions are only passed for modifications <hazmat> in this case its an observation that shows old state <fwereade> hazmat, sorry, lunch * hazmat moves onto openstack provider review <hazmat> fwereade, could pastebin the test in question? <fwereade> hazmat, the clearest situation is in line 10 of http://paste.ubuntu.com/1071241/ -- when it occurs, other connections are known to have been going around creating the node we're watching in the past <hazmat> fwereade, that looks like the same connection? <fwereade> hazmat, yes, it is an alternate connection is a previous test about which I am concerned <niemeyer> Gooooood morning! <Aram> hi niemeyer. <Aram> how was your trip? <hazmat> niemeyer, g'morning, how was i/o? <Aram> and how was SF? <niemeyer> hazmat: Superb <niemeyer> Aram: Superb too :) <niemeyer> Aram: Great to meet all the folks <TheMue> niemeyer: Heya <Aram> niemeyer: yeah, that must have been great. <hazmat> fwereade, what happens to the conn in setup/teardown? <hazmat> is there anyway to get the go test to be verbose about test cases being run? <fwereade> hazmat, in TearDownTest, recursively delete everything and (IIRC, checking) panic on error except nonodoe <niemeyer> http://arethegovideosupyet.com/ < This is great :) <hazmat> fwereade, so it is the same open connection for multiple tests? <fwereade> hazmat, -test.v -gocheck.vv should give you plenty <fwereade> hazmat, yes <TheMue> niemeyer: Yep, funny idea. Waiting for the next ones to be online. <niemeyer> How're things going in juju-dev land? <TheMue> niemeyer: First a hurdle but now moving forward. <niemeyer> fwereade: So, hazmat tells me we're not testing our code properly because we have a bug. What's up there? <fwereade> niemeyer, I am still trying t characterize it properly <fwereade> niemeyer, the stars appear to be aligned such that I can repro more often than not <niemeyer> fwereade: Ah, this is perfect <fwereade> niemeyer, but I am still trying to coax an "aha" out of the data <niemeyer> fwereade: http://paste.ubuntu.com/1071241/ is this the test? <fwereade> niemeyer, that is one of the many that *can* exhibit anomalous behaviour <fwereade> niemeyer, but the vast majority of the presence suite has been *slightly* flaky for a while, and I now appear to be close to pinning down the problem <niemeyer> fwereade: Ah, that's awesome <niemeyer> fwereade: Does it fail if you run just presence in isolation? <fwereade> niemeyer, very very very rarely <niemeyer> fwereade: Your more-often-than not is achieved with a few packages, or just with the whole suite? <fwereade> niemeyer, just state <niemeyer> Nice <niemeyer> fwereade: What's the liveness timing being used by the tests? <fwereade> niemeyer, 50ms, and there is certainly something tricksy there which I think I am close to accounting for <fwereade> niemeyer, ie sometimes we get mtimes more than 100ms apart when we do that <niemeyer> fwereade: This may well be the issue <niemeyer> fwereade: The GC may be stopping to collect stuff <fwereade> niemeyer, this is surely *part* of the issue <fwereade> niemeyer, I am trying to eliminate it and see whether I can goose theweirder one into existence <niemeyer> fwereade: Does it change the situation if you set GOMAXPROCS=4? <fwereade> niemeyer, hmm, quite possibly; but that's another dimension of phase space that I don't need right now I think <niemeyer> fwereade: Well, perhaps without the parallel collection that went into tip it wouldn't make much of a difference anyway <fwereade> niemeyer, well, we need timing tweaks, but the real important ones will come in real usage I think <fwereade> niemeyer, if we make them noticeably more generous than whatever I find to be rock-solid in test usage we will hopefully be ok <niemeyer> fwereade: Agreed <niemeyer> fwereade: Is there anything more unusual than the timing bits, that you'd like a second pair of eyes over? <fwereade> niemeyer, if I can't figure it out today I will cry uncle, but I feel like I'm converging on something <niemeyer> fwereade: Superb, you have me excited on the other side meanwhile ;-D <fwereade> niemeyer, cool :) <niemeyer> fwereade: Any luck there? <fwereade> niemeyer, broke for supper at an opportune point; it looks like there was a case we'd missed in the code, subtly distinct from the normal failures due to c pauses/whatever <fwereade> niemeyer, state seems to be solid, given a fix for that <fwereade> niemeyer, trying a few full runs <niemeyer> fwereade: Oh, so happy that you found it.. even if I don't know what the issue really is yet :-) <fwereade> niemeyer, there's another bit of the issue which is trivial and embarrassing and kinda contributed to some of the confusion <fwereade> niemeyer, not all the tests were nicely cleaning up pingers on failure <fwereade> niemeyer, so I plan to do a pass for that too <niemeyer> fwereade: Aha, that was my initial guess at the problem <niemeyer> fwereade: Not to be embarrassed, though.. it's kind of easy to miss tear downs in any testing <fwereade> niemeyer, the thing is, I knew about that -- the first failure often causes a cascade -- but there was always a particular mode of initial failure that didn't seem to match reality <niemeyer> fwereade: Well, I appreciate you going after root cause.. a lot of people just ignore the obvious hints and go for the trivial solutions <fwereade> niemeyer, has to be done really <fwereade> niemeyer, fwiw there's a very occasional store failure that I never remember to capture <fwereade> niemeyer, primarily because the sheer weight of mongo logs is intimidating <fwereade> niemeyer, I promise that next time I see it I will make a proper bug <fwereade> ;) <niemeyer> fwereade: If it's the one I'm thinking off, it's just timing <fwereade> niemeyer, sounds very plausible <niemeyer> fwereade: I feel bad for it.. I've been kind of postponing increasing the timing to see if it will force myself to get the test to run faster rather than going for the easy solution <niemeyer> fwereade: I can tell it's not working so far <fwereade> niemeyer, I can sympathise <niemeyer> brb <niemeyer> fwereade: Dude <niemeyer> fwereade: There? <niemeyer> davecheney: Morning! <davecheney> niemeyer: howdy! <niemeyer> davecheney: Good to see you around from the usual time zone :) <niemeyer> davecheney: Less overlap, but at least it's easy to actually talk :) <davecheney> indeed <davecheney> hows things ? <niemeyer> Pretty good. Just having a look at William's monster branch <niemeyer> Looks very nice <davecheney> niemeyer: it would be awesome if that got a green light <davecheney> i need his refactorings of zksuite etc for the local ec2 tests <niemeyer> davecheney: Thanks for reviewing it too, btw.. really appreciate having more eyes <davecheney> niemeyer: anytime <niemeyer> davecheney: It will surely get a green light <davecheney> it is big, and a lot of the changes are one line per file <niemeyer> davecheney: huge improvement overall <davecheney> indeed <niemeyer> davecheney: The concerns I have so far are lateral.. e.g., mstate needs to be included <davecheney> niemeyer: in related news, I have a fix for the location constraint issue in goamz <davecheney> but am unsure how to write tests for it <niemeyer> davecheney: Hmm <niemeyer> davecheney: Can you please open a CL with the fix? <niemeyer> davecheney: I can have a look and suggest something <davecheney> niemeyer: twosecs <davecheney> niemeyer: https://codereview.appspot.com/6344050 <niemeyer> davecheney: Checking <davecheney> niemeyer: i can add support for LocationConstraint parsing into s3 test if you like <niemeyer> davecheney: Okay, so <niemeyer> davecheney: The testing seems to be easy to do inside s3_test.go <niemeyer> davecheney: We mock the server, and can easily compare the result against something we own. <niemeyer> davecheney: We don't even need to parse it <niemeyer> davecheney: Check out.. hmmm.. <niemeyer> davecheney: Well, we actually don't have an example yet <niemeyer> davecheney: But the req we get out of WaitRequest is a normal http.Request <niemeyer> davecheney: With a Body and all <davecheney> niemeyer: yeah, I can address the TODO in the s3_test server <niemeyer> davecheney: A second detail: it looks like this info is well suited for a Name field <niemeyer> davecheney: Region.name <niemeyer> davecheney: Region.Name <davecheney> niemeyer: yes, there are a number of places where we want to convert from the Region type back to its canonical name <davecheney> niemeyer: https://codereview.appspot.com/6347059 << adds Region.Name <niemeyer> davecheney: The map is a nice touch, thanks <niemeyer> davecheney: LGTM <davecheney> niemeyer: that was a TODO from environs/ec2 <niemeyer> davecheney: Ah, I didn't recall.. still looks like a good idea then! ;-) <davecheney> niemeyer: i'll address the todo in juju after I commit the location constraint fix <davecheney> so that people get a hint to go update goamz <niemeyer> Super * niemeyer => dinner.. back soon <davecheney> niemeyer: would you mind commiting williams lp:~fwereade/juju-core/vast-zookeeper-tests-cleanup ? <niemeyer> davecheney: Hmm.. there are a few trivial details there to be sorted out.. I think I'd prefer to let him consider these details, including the mstate stuff, to see how to push it forward before getting it in #juju-dev 2012-07-03 <davecheney> niemeyer: sure thing <niemeyer> davecheney: ping <davecheney> niemeyer: ac <davecheney> niemeyer: ack <niemeyer> davecheney: Yo <niemeyer> davecheney: I was having a look at go-cmd-jujud-services <niemeyer> davecheney: A few points: <niemeyer> - You mention the actual meat of the change. Has anything changed within the moved files themselves? Just wondering if I should review that or not. <niemeyer> - I was expecting the package to be service/<service name>/*.go <niemeyer> More precisely, juju-core/service/<service name>/*.go <niemeyer> Moving them out of jujud but keeping all of the services within the same file would allow importing, but preserve the "mess" which is having all of those unrelated things in the same place <niemeyer> I think that's it.. otherwise the idea of moving sounds great <davecheney> niemeyer: thanks for the feedback, i'll fix up the CL and repropse <davecheney> niemeyer: i'm going to do the service CL in three parts, which might make it easier to review and merge <davecheney> 1, cmd/services <davecheney> 2, split provisioner and provisioning <davecheney> 3, move provisioner to the new package <niemeyer> davecheney: Sounds good <davecheney> niemeyer: does bzr have the concept of hg cp ? <niemeyer> davecheney: Btw, I suggest juju-core/services <niemeyer> davecheney: Hmm.. not sure <davecheney> niemeyer: yes, good idea, they dont' belong directly in cmd <davecheney> niemeyer: just trying to reduce the size of the CL that just bifurcate provisioning.go <niemeyer> davecheney: Cool.. as long as we don't move + change the code at the same time, it's easy to review either way I think <niemeyer> davecheney: If there are changes that are tricky to avoid, it's fine to just mention them in the description so we can have a more careful look rather than assuming it's just coded being moved <davecheney> niemeyer: https://codereview.appspot.com/6350064 << just moves Provisioner into it's own file (and tests) <davecheney> which will make the other change easier to digest <niemeyer> davecheney: If it's just moving, +1 for sure <davecheney> niemeyer: yeah, just splits so I can move later and the diff will just show the package header changed and a few imports <niemeyer> davecheney: Awesome, thanks, sent the LGTM there too <davecheney> niemeyer: this will have a small impact on williams test CL as well <davecheney> i'll work with him this afternoon to help unbreak it <davecheney> niemeyer: can I propose a branch on top of his banch ? ie, can I do the fixes for his branch and he can import them ? <niemeyer> davecheney: Absolutely <davecheney> niemeyer: right, sounds fair, and I want his coretesting and zksuite types <niemeyer> davecheney: Neat, branches are lovely in those cases <niemeyer> davecheney: What would otherwise be a blocker is just a branch + merge <davecheney> niemeyer: https://codereview.appspot.com/6344071 << much smaller <niemeyer> Phew' <niemeyer> davecheney: LGTM <niemeyer> Okay.. time to sleep now <niemeyer> Good night all <niemeyer> davecheney: and good afternoon for you <fwereade> davecheney, morning <davecheney> fwereade: howdy <davecheney> you've got a green light to drop that massive branch <davecheney> but i've spent the day making conflicts <davecheney> so sorry about that <fwereade> davecheney, ah, no worries, I expect I can manage :) <davecheney> fwereade: let me know if I can be of use, the faster I can get your changes, the better <fwereade> davecheney, you can just merge from that branch and work on top of it -- but then the requested changes to that will lead you to have to deal with probably as many conflicts <davecheney> fwereade: yeah, i've got a few branches in the air at the moment <davecheney> but i'm truing to keep my main 'provision on ec2' as close to trunk as possible <davecheney> error: cannot upload tools: cannot make S3 control bucket: The specified location-constraint is not valid <davecheney> s3 can get stuffed <davecheney> you _must_ specify a LocationConstraint if you are _not_ using us-east-1 or us-west-1 <davecheney> but you _mustnot_ specify one if you _are_ using us-east-1 <davecheney> which is bullshit <fwereade> davecheney, eww <davecheney> fwereade: going to be a bit of a shit to write a mock for <fwereade> davecheney, yeah, I bet <davecheney> fwereade: this is a nasty wart from us-east-1 being special because it's the first region <TheMue> Hi fwereade <fwereade> TheMue, heyhey <TheMue> fwereade: Just watching the I/O talks. <fwereade> TheMue, I must get to those <fwereade> TheMue, somewhat code-obsessed at the moment <TheMue> fwereade: Yeah, with your great change <fwereade> TheMue, I'm going through an obsessive set of running the presence tests <fwereade> TheMue, sadly they are not 100% reliable <fwereade> TheMue, *but* I'm pretty sure that every failure is now timing-related and can be attributed to impolite schedulers and/or GC pauses <TheMue> fwereade: Hard to fix stuff. <fwereade> TheMue, yeah, GOMAXPROCS seems to help <fwereade> TheMue, but that could just be my monkey-brain overlaying patterns where none exist <TheMue> fwereade: *rofl* <TheMue> fwereade: Could you please help me with todays firewall code? <TheMue> fwereade: It's regarding line 278ff. <fwereade> TheMue, sure, lookin for it... * fwereade looks somewhat apprehensive <TheMue> fwereade: Thx, and there especially line 281. <TheMue> fwereade: Here a machine, not the machine state, is retrieved from the provider. <TheMue> fwereade: Sadly I don't find yet what has this role in our code today. <TheMue> fwereade: Did it change or isn't it yet ported? <fwereade> TheMue, in go I think that's called an Instance <fwereade> TheMue, corresponds to ProviderMachine, right? <TheMue> fwereade: That's what I want to find out. <fwereade> TheMue, I'm 99% sure it corresponds directly to environs.Instance <TheMue> fwereade: OK, will take a deeper look. That may be the missing link. <TheMue> fwereade: Thank you. <fwereade> TheMue, a pleasure :0 <fwereade> TheMue, *dammit*, I have to go to the bank, wanted to finish this off before :/ <fwereade> TheMue, meeting in 50 mins, right? <davecheney> niemeyer: thanks for your review <davecheney> niemeyer: figuring out how to test a condition that can only be false if you're talking to the real us-east-1 is tricky <niemeyer> Yo! <niemeyer> davecheney: Hmm <niemeyer> davecheney: Does it fail if you keep the constraint while talking to us-east-1? <davecheney> niemeyer: yes * davecheney facepalms <davecheney> that was the first thing i tried <davecheney> us-east-1 is ... special <niemeyer> davecheney: Lovely <niemeyer> davecheney: Anyway, that still sounds easy to test in s3_test.go <davecheney> niemeyer: otherwise it would be straight forward <davecheney> niemeyer: i <davecheney> i'll take another crack tomorrow <niemeyer> davecheney: We have the payload at hanhd <niemeyer> davecheney: And also have the test region at hand <niemeyer> davecheney: We can tweak the region however we want, and see the effect on the payload <niemeyer> Isn't it meeting time? <niemeyer> fwereade, TheMue, davecheney: Meeting? <fwereade> niemeyer, SGTM <TheMue> Yes, but currently phone. <TheMue> In a few moments. <davecheney> niemeyer: sure, two secs <niemeyer> Hmm.. Aram isn't around <niemeyer> Starting a Hangout <davecheney> niemeyer: send the invite to david.cheney@canonical <niemeyer> Done <niemeyer> fwereade: ping <davecheney> nouns that are taken: service, agent, command, process <niemeyer> worker? <davecheney> possible nouns: daemon, <davecheney> ^ pretentious ? <davecheney> http://laughingmeme.org/2005/12/23/there-are-only-two-hard-things-in-computer-science-cache-invalidation-and-naming-things/ <fwereade> davecheney, and off-by-one-errors <niemeyer> fwereade, davecheney: Suggestion: Life, SetLife and WatchLife.. <TheMue> +1 <davecheney> good discussion tonight <davecheney> just for reference, i'll sleep on the renaming stuff we discussed and email tomorrow <TheMue> davecheney: ;) <davecheney> i am in agreement that we want to avoid the situation of a bug report that says 'something something agent did not blah' <davecheney> and we have to ask for clarification on if they mean the agent process, or the one of several agent 'services' we had running inside that binary <Aram> hello. <Aram> grr, dreadfully sorry I couldn't make it to the meeting. <Aram> had internet problems all morning. <Aram> meh. <davecheney> i blame the leap second <TheMue> Hmm, could it be that todays "provider".get_opened_ports() has no counterpart? <hazmat> TheMue, what would the counterpart be? get closed ports? <TheMue> hazmat: Hi, eh, wrong wording. I mean the ported part, which I expected in environs.Environ. <hazmat> ah <hazmat> yeah.. status will need that, else its not really used outside of the provider <hazmat> er. provisioning agent <fwereade> TheMue, seems that way, it's not in the interface <TheMue> hazmat: Yes, I'm porting firewall.py for the provisioning agent. And this type needs it to retrieve the current open ports to compare it with the state. * niemeyer waves <Aram> hi niemeyer. <niemeyer> Aram: Yo <Aram> niemeyer: sorry for not making it to the meeting :(. <Aram> my internet didn't work. <Aram> which is a first, after an year of staying here. <niemeyer> Aram: It's ok, we can catch up later.. just a bit unfortunate because we talked about the proposal in the ML and covered a few angles, but we can revive that again <Aram> yes, I really want to hear opinions about that. <niemeyer> Aram: Well, me too ;) <Aram> niemeyer: more lbox trouble... <Aram> http://paste.ubuntu.com/1073238/ <fwereade> niemeyer, hey, I have something that looks very similar: http://paste.ubuntu.com/1073263/ <niemeyer> Hmm <fwereade> everyone, I have to go a bit early today... I will certainly be back on in the evening to submit that change, by hook or by crook ;) <niemeyer> fwereade: Cheers <Aram> niemeyer: maybe it's a leap second bug ;) <Aram> fwereade: have fun. <niemeyer> Aram: Possibly :) <fwereade> Aram, haha :) <fwereade> cheers guys, see you later <niemeyer> Aram: Can you please run with -debug? <niemeyer> Aram: It looks like codereview might be blowing up <Aram> niemeyer: sure thing <niemeyer> Aram: Seems that reaching that point is returning without err <niemeyer> Aram: oh, wait, maybe I misread <Aram> niemeyer: http://paste.ubuntu.com/1073272/ <niemeyer> Aram: The code on goetveld definitely looks wrong <niemeyer> Aram: Hah, it's the same problem as before :( <niemeyer> Aram: Just in a different package <Aram> heh. <niemeyer> Aram: The change in redirect semantics sucks <Aram> niemeyer: yeah, I don't think this should have happened, document the old behavior and stick with it until at least go 1.1. <niemeyer> Aram: I'll see what I can do to workaround the issue.. the change in semantics is actually a problem in this case, because I want the response before the redirect, and there's apparently no way to do this now <Aram> lunch <niemeyer> Aram: lbox is coming <Aram> niemeyer: fantastic <niemeyer> Aram: I'll rebuild against golang-stable.. filed a bug against golang and Brad F. will revert his change next week <niemeyer> Aram: There's simply no way to grab a response before a redirect with his change <niemeyer> Aram: Which is needed in this case, because the login endpoint redirects, and we want its Cookie before we move on <Aram> niemeyer: I see. <Aram> niemeyer: so you bakported the exp/ stuff we're using? <niemeyer> Aram: We're not using anything experimental <niemeyer> Aram: It was using golang-tip just for historical purposes (IOW, before Go 1 was out, we depended on tip) <niemeyer> Aram: It turns out that it's handy to depend on tip, though.. <niemeyer> Aram: I may well roll it back after next week <niemeyer> Aram: I wouldn't have caught the bug on time otherwise <niemeyer> Aram: Silly me.. apparently it does need exp/ from tip <Aram> yeah, I remembered I couldn't compile it at some point. <Aram> because of this. <niemeyer> Aram: You were totally right <niemeyer> Aram: I'll just embed it for the moment <niemeyer> Aram: lbox should be good <Aram> niemeyer: thanks. <niemeyer> Aram: ping <Aram> pong <Aram> seen your email <niemeyer> Aram: Cool, i was actually going to suggest something, but I'm just reviewing the pre-req.. will work out well <Aram> niemeyer: updated that CL, also seen your bug report, will fix it tomorrow, will retreat for now, it's kind of late here :). <niemeyer> Aram: I can imagine.. almost finishing the last review <niemeyer> Aram: Thanks a lot for your help on that stuff <Aram> my pleasure. <niemeyer> davecheney: Morning! * niemeyer heads off for dinner <davecheney> ziing <fwereade_> davecheney, heyhey, I just merged that monster <fwereade_> davecheney, I hop it didn't mess with your Machiner plans too badly <davecheney> fwereade_: nah, it was a few minutes work to rebranch and do it again <davecheney> fwereade_: the benefits far outweigh the cost <fwereade_> davecheney, btw, if you ever spot the presence tests misbehaving, scream and should and send me the log output please :) <fwereade_> s/should/shout/ <davecheney> m_3: you around ? <davecheney> niemeyer: I had to repropose the machiner -> service/machiner branch here https://codereview.appspot.com/6348077 <davecheney> additionally, now it's not a prereq, the branch is much cleaner <m_3> davecheney: <m_3> yo <m_3> davecheney: my reply probably just got out on smtp <davecheney> m_3: cool <m_3> tomorrow'd probably be great for Ultimo if that works for you <davecheney> yeah, that sounds good #juju-dev 2012-07-04 <davecheney> niemeyer: thanks for the review <niemeyer> davecheney: np! <davecheney> i have in the back of my mind the request to rename those packages <davecheney> worker/agent/etc sound like good solutions, but I'd liek to give it a little more thought <davecheney> maybe go for a walk and a sit before proposing anything <niemeyer> davecheney: "worker" seems the best option so far <niemeyer> davecheney: We already use agent nowadays with a different meaning <davecheney> niemeyer: yeah, all the good options are taken <davecheney> niemeyer: lbox has just started to panic on me <davecheney> pls hold for the paste <davecheney> http://paste.ubuntu.com/1074143/ <davecheney> is this related to the issue you raised in the mailing list ? <niemeyer> davecheney: Can you please update it? <niemeyer> davecheney: It is related indeed <niemeyer> davecheney: I've backported it from tip to stable for now, though, so we can continue moving forward while Brad fixes it <davecheney> niemeyer: will do <niemeyer> Alright, bed time here <fwereade_> davecheney, since you have been doing Provisioner lately, I was wondering whether you had wisdom to impart re handling of environ configs <davecheney> fwereade_: possibly, what are you trying to do ? <fwereade_> davecheney, because I'm getting very uncomfortable with the usage of naked ConfigNodes <fwereade_> davecheney, I'm not *quite* sure yet what I'm aiming for -- I'm working on getting Initialize to set up the environment node <fwereade_> davecheney, and it seems to me that Initialize alone should be responsible for "name" and "type", which should otherwise be unsettable <davecheney> fwereade_: i see your concern <davecheney> for the moment we've solved it via contract, by saying the environs should refuse to update those keys <fwereade_> davecheney, yeah, I just can't really articulate it further in a useful way <davecheney> once set <davecheney> from my point of view, i'm happy with ConfigNode providing a very simple interface <davecheney> but, i'm not the architect of this project :) <fwereade_> davecheney, ah, sorry, totally missed your important point -- what is it that enforces those non-changes? <davecheney> a loose agreement inside the environ itself <davecheney> backed entirely by a comment <fwereade_> davecheney, ok, gotcha <davecheney> environs/ec2/ec2.go#SetConfig <fwereade_> davecheney, the trouble is that EnvironConfig is acessible to any numpty with a State ;) <davecheney> i would be less concerned with people changeing the name of the environ than pinching the secrets which are stored in the same key <fwereade_> davecheney, sure, that too ;p <davecheney> fwereade_: are you protecting against a back actor, or a well meaning, but misguided mistake ? <fwereade_> davecheney, well-meaning mistakes are my primary concern really <fwereade_> davecheney, I am just generally concerned that which-environ-to-use is a subtler problem than it may at first appear <fwereade_> davecheney, this is mainly on the client side <davecheney> fwereade_: i concur <davecheney> fwereade_: but I have a plan to un fuck it a bit <fwereade_> davecheney, I would like a clean way to express the stuff we need to do on first access to an env <davecheney> i'm going to figure out, in cloudinit, what the real internal IP of machine/0 is, so we can write it into the state then and there <davecheney> fwereade_: on a related note, how does jujutest come up with the dnsname for the first machine as i-3.example.com ? <fwereade_> davecheney, er, not sure I'm afraid <fwereade_> davecheney, continuing above I feel that it should be obvious and easy to use a juju.Conn in such a way that: <fwereade_> davecheney, on bootstrap, we use the environments.yaml config exclusively <fwereade_> davecheney, on subsequent operations, we push the missing keys from environments.yaml into state, and subsequently use only the config from state <davecheney> fwereade_: gotta fly -- tonight is my night to help out at the bowls club <fwereade_> davecheney, np, have fun, take care :) <davecheney> catch you on the other side <fwereade_> heya TheMue <TheMue> fwereade_: Hi <TheMue> fwereade_: Successfully get your branch in? ;) <fwereade_> TheMue, yeah, thanks :) <fwereade_> TheMue, hope the merge doesn't hurt too much :) <TheMue> fwereade_: So far my new code is totally outside, it only uses state. So it should be no problem when merging it into my branch. <jamespage> morning juju dev's <jamespage> is anyone able to help me diagnose why one of the charms in launchpad is not accessible in the charm store? <james_w> jamespage, you know about http://jujucharms.com/tools/store-missing ? <jamespage> james_w, no I did not - thanks <james_w> but unless you're asking about shelr.tv that's probably not going to be very useful <jamespage> james_w, no - my query is with reference to the 'ubuntu' charm <james_w> ah <james_w> indeed, not very useful <james_w> I've no more ideas, sorry, I'll return you to waiting for a dev <niemeyer> Hey there! <TheMue> Heya niemeyer <fwereade_> niemeyer, heyhey <james_w> hi niemeyer <james_w> jamespage, has a question about why ubuntu isn't in the charm store: http://jujucharms.com/tools/store-missing <niemeyer> james_w: In a meeting, will be with you on a sec <Aram> hey all. <niemeyer> Aram: Heya <niemeyer> james_w: Okay, so how can I help? <james_w> <jamespage> is anyone able to help me diagnose why one of the charms in launchpad is not accessible in the charm store? <james_w> <jamespage> james_w, no - my query is with reference to the 'ubuntu' charm <niemeyer> james_w: Which branch is that? <james_w> lp:charms/ubuntu <niemeyer> james_w: We've debugged this problem before, actually <niemeyer> james_w: Clint asked me about it back in May <niemeyer> james_w: This was the issue: https://pastebin.canonical.com/66332/ <james_w> hah <james_w> niemeyer, where did you find that output? <niemeyer> james_w: From Tom <james_w> it would be great to have it on http://jujucharms.com/tools/store-missing <niemeyer> james_w: This is supposed to be integrated in the charm-store proper <niemeyer> james_w: In the API <james_w> niemeyer, I suspect that's a bug in the bzr freshness check for package branches <niemeyer> james_w: So anyone can query it and put wherever <james_w> niemeyer, aside from fixing the bug you can disable that check <niemeyer> james_w: How do you mean? <james_w> niemeyer, I assume it's the MISSING line that is throwing it off? <niemeyer> james_w: It's the fact it's outputting an error, yeah <james_w> niemeyer, so anything on stderr causes the import to bail? <niemeyer> james_w: No, but this certainly does: <niemeyer> % bzr checkout --lightweight lp:~charmers/charms/precise/ubuntu/trunk <niemeyer> Most recent Ubuntu version: MISSING <niemeyer> Most recent Ubuntu version: MISSING <niemeyer> Most recent Ubuntu version: MISSING <niemeyer> james_w: Well, or maybe I'm lying <niemeyer> james_w: Let me check reality <niemeyer> james_w: The command is really failing <james_w> non-zero exit code? <niemeyer> james_w: No, I'm lying again, sorry.. I did something silly that resulted in echo $? == 1 <niemeyer> [niemeyer@gopher ~/trunk]% bzr revision-info <niemeyer> Most recent Ubuntu version: MISSING <niemeyer> Most recent Ubuntu version: MISSING <niemeyer> 2 heatherjohnson@example.com <niemeyer> [niemeyer@gopher ~/trunk]% echo $? <niemeyer> 0 * niemeyer checks implementation <james_w> ok <niemeyer> james_w: It's getting the combined output <niemeyer> james_w: I'll fix that and invite Tom to redeploy <james_w> niemeyer, fix it to do what? <niemeyer> james_w: To get stdout <james_w> ok <james_w> we can also make the message go away <niemeyer> james_w: It's slightly worse for debugging <niemeyer> james_w: But I'd rather have it working instead ;-) <niemeyer> james_w: That's certainly a better option overall for bzr <niemeyer> james_w: Having every command spitting out an arbitrary number of irrelevant messages sounds bad <james_w> right, bzr shouldn't activate the check for that url <james_w> but we can also have the charm importer tell bzr to just never even bother thinking of doing the check <niemeyer> james_w: How do we do that? <james_w> I'm looking <niemeyer> james_w: Cheers <james_w> bazaar.conf: launchpad.packaging_verbosity = off <james_w> there may be a way to set it on the command-line, but I don't think so <james_w> and you can escalate https://bugs.launchpad.net/bzr/+bug/1020935 if you like <james_w> ah, https://bugs.launchpad.net/bzr/+bug/888615/comments/4 <james_w> that's presumably better than the config file <niemeyer> james_w: I'll probably just pick stdout <james_w> niemeyer, that harms debugging? <niemeyer> and let it complain <james_w> can you capture stderr and include it if the command fails or something so that you can have both? <niemeyer> james_w: It could, because interleaving stderr/stdout means errors are in context, but in practice this seems to have made more harm than good, so I'm happy to adapt to reality <james_w> yeah <james_w> I've always found doing both to be tricky, I wish more libraries allowed for getting at both individually and an interleaved stream <niemeyer> james_w: Yeah, it's seldomly done because one can't actually do it perfectly unless it's done at the kernel level due to buffering <niemeyer> james_w: The correct way to interleave is to use a single fd, and then the individual streams are gone <james_w> niemeyer, yeah, agreed it's not easy <james_w> I still want it though :-) <niemeyer> james_w: +1 :) <niemeyer> Lunch time <niemeyer> Quiet day #juju-dev 2012-07-05 <davecheney> niemeyer: howdy <davecheney> sorry i'm a bit slow off the mark today <davecheney> m_3 and I are checking out a coworking space in the city today <davecheney> niemeyer: short status report, got ec2 local tests working with a provisoiner working last night <davecheney> but there are a few problems that need to be solved before I can propose it <niemeyer> davecheney: Heya <niemeyer> davecheney: Neat, sounds good <davecheney> niemeyer: the main one is we ask ec2test for the DNS name of machine/0 <davecheney> whcih returns something liek i-3.example.com <davecheney> which breaks the local test because it then uses that hostname in the stateinfo passed to anyone who wants it <davecheney> so I need to add some kind of hook to either ec2test to say 'always return localhost' as the dns name, or to be able to override it if the test is running in local mode <davecheney> niemeyer: the second problem is ec2 is hard coded to expect the zkport to be 2181 <davecheney> but that is simpler to fix <davecheney> i'll propose a fix real soon now <niemeyer> davecheney: Hmm <niemeyer> davecheney: Wait, you mean you're putting a worker within the ec2 tests? <davecheney> niemeyer: by worker, do you mean provisioner ? <niemeyer> davecheney: a juju-core/worker <davecheney> niemeyer: yes, the tests that roger left bootstrap an environment then try to do a state.AddMachine <niemeyer> davecheney: Within the *ec2* tests? <niemeyer> davecheney: I may be wrong, and will be happy to look at it, but it doesn't sound entirely right <davecheney> niemeyer: yes, roger extended jujutest/livetest to start a machine in the provisioned environment <davecheney> hang on, will get a link <davecheney> niemeyer: sorry, m_3 just turned up <davecheney> https://codereview.appspot.com/6347044/ << not a proposal <niemeyer> davecheney: No worries, I'll actually have to step out for dinner <davecheney> this is a grab bag of rogers stuff and my additions <davecheney> niemeyer: the key change is https://codereview.appspot.com/6347044/patch/10001/11012 <davecheney> specifically, if hasProvisioner then testprovisioner <niemeyer> davecheney: I'm indeed a bit lost.. in a sentence or two, what is this testing? <davecheney> niemeyer: the changes to jujutest test that after bootstrapping, a provisioner is running, and it can start an instance with state.AddMachine <niemeyer> davecheney: I can't see how we could possibly be testing that without a functional test <niemeyer> davecheney: How can we be testing that after bootstrap a provisioner is running? This is entirely dependent on details of ec2 itself <davecheney> yes, i haven't got that to work in the ec2 amazon tests yet because of the push secrets problem <davecheney> this only works when we make the local provisioning agent in process for local_test.go <davecheney> but in theory, once the secrets are pushed (faux juju deploy) the test will also pass on the real ec2 <niemeyer> davecheney: The point is that there's nothing being tested there <niemeyer> davecheney: The live real tests can do that on real EC2 <niemeyer> davecheney: If we fake the precise point we want to test, what's the purpose of the test? <davecheney> niemeyer: this test should (ha!) work on the real ec2 as well <davecheney> so at the moment all I can prove is the test case works when we fake out all the moving parts behind it <davecheney> that is, localtest continues to work with a test plan that expects a provisioner <davecheney> once the pushing of secrets is fixed, we can remove the 'hasprovisioner' field as it will always be true <niemeyer> davecheney: I understand, but this doesn't seem to be going in a good direction <niemeyer> davecheney: All this faking isn't buying us stability or anything <niemeyer> davecheney: It's messing up the code instead <davecheney> niemeyer: should I put this on hold until we know how to fix the pushing of secrets and can do a real test in a real ec2 ? <niemeyer> davecheney: No, I think we should fix the problems we have to fix for getting the real EC2 running <davecheney> niemeyer: cool, that means resolving the secrets problem <niemeyer> davecheney: The secrets problem is resolved in deploy <niemeyer> davecheney: There's no *real* problem.. we just need to do the same thing we used to do <niemeyer> davecheney: Sorry, I'm kind of stating that with the belief
{'channel': 'juju-dev', 'month': 7}