#juju 2011-10-01 <hazmat> argh.. hotspot down to edge networking <hazmat> fail <jimbaker> hazmat, i'm trying to ssh into the local lxc containers with juju ssh. but i'm getting a pw prompt. looking at the code, i would expect it to just enable me to use my public key. what am i missing here? <hazmat> jimbaker, it should use your key <hazmat> jimbaker, although i had a minor accident today where i tried to ssh into the container from a root shell.. it needs to be the shell of the user who bootstrapped <hazmat> SpamapS, are you going to the wikipedia hack days in nola? <hazmat> i was just considering going down there <jimbaker> hazmat, ok, it's not working for me then. i'm using local-origin-passthrough however <jimbaker> does that depend on some other branches? <hazmat> jimbaker, hmmm. let me push my latest <hazmat> jimbaker, that's the one i'm working against <jimbaker> hazmat, ok <hazmat> i gave up on my last attempt after my hotspot switched to edge <hazmat> although i was definitely able to ssh into my container <hazmat> jimbaker, looks like the latest is pushed <jimbaker> hazmat, definitely annoying that, although i find it's more of a transitory thing - maybe 5 min. assuming a place w/ a normal good signal <hazmat> jimbaker, i started putting the customize script output into the data-dir under units <jimbaker> hazmat, i was trying the latest <hazmat> that's what setups the container pretty much.. there's lots of logs on disk in the data-dir .. machine agent, container logs, customize script logs <hazmat> jimbaker, where you doing it from a root shell? <jimbaker> hazmat, no, just letting it ask for sudo as usual <hazmat> jimbaker, you can poke at the constructed container to see if the key is there its under /var/lib/lxc <hazmat> there should be an authorized key file under /home/ubuntu/.ssh <jimbaker> hazmat, it's there <hazmat> i'm definitely able to login <hazmat> via ssh against the ubuntu user <jimbaker> hazmat, yeah, that's what it was prompting me for <hazmat> jimbaker, how are you doing ssh? using juju? <jimbaker> hazmat, yes <jimbaker> juju ssh wordpress/0, usual stuff <hazmat> ah.. i've been doing ssh by hand * hazmat tries juju ssh <jimbaker> hazmat, i don't know enough about lxc, so i've been using the tool here ;) <hazmat> jimbaker, dig @(316)288-4715 container-name <hazmat> will give the ip address <hazmat> jimbaker, your actually farther than i am atm, i haven't gotten the unit agent running (which means no juju ssh) <hazmat> i committed and pushed the fixes, but wasn't able to do a final run, due to lack of networking <hazmat> trying again <jimbaker> hazmat, i really was just tried juju ssh 0 in this case <hazmat> jimbaker, you can't do that <hazmat> i mean you can <hazmat> but its not what you think <jimbaker> yeah, that makes sense <hazmat> jimbaker, juju ssh 0 <hazmat> is to login into a machine <hazmat> but all the units are on localhost <jimbaker> exactly <hazmat> one machine <hazmat> so its rather pointless and we don't mess with the host user's authorized keys in their home directory <jimbaker> i think the more relevant thing is the data-dir <jimbaker> at least i might see something <hazmat> so that won't work unless you've explicitly set it up <jimbaker> basically none of the service units start, just like your experience <hazmat> jimbaker, are you doing this juju-origin set? <hazmat> w/ <jimbaker> hazmat, i tried, that didn't work <jimbaker> so now, no <hazmat> jimbaker, yeah.. ppa should work <hazmat> jimbaker, except it fails because of the protocol version increment, but the unit agents are running with it <hazmat> i'm still debugging why they don't come up with juju-origin <hazmat> although at this point, its time for a weekend <jimbaker> hazmat, sounds like a reasonable decision to me too! enjoy <hazmat> bcsaller, have you ever seen the pty allocation error <hazmat> it keeps coming up for me, very annoying.. basically needs a reboot <bcsaller> I've never gotten it, no <_mup_> juju/local-origin-passthrough r416 committed by emily84@example.com <_mup_> add python-yaml dep for branch installs, and switch address map s/lxc/local <hazmat> chr <hazmat> all working now, nice <_mup_> juju/unit-info-cli r422 committed by emily84@example.com <_mup_> missing bin/unit-get <_mup_> Bug #863816 was filed: We need a cross-provider way to allow units to get pub/priv address info <juju:In Progress by hazmat> < https://launchpad.net/bugs/863816 > <_mup_> Bug #864164 was filed: Must complain if charm hooks aren't executable <juju:New> < https://launchpad.net/bugs/864164 > #juju 2011-10-03 <_mup_> Bug #865163 was filed: default-series option has surprising behaviour <juju:New> < https://launchpad.net/bugs/865163 > <niemeyer> Good mornings! <rogpeppe> niemeyer: hiya! <fwereade> heya niemeyer <niemeyer> Hey folks! * niemeyer grabs coffee before engaging into juju awesomeness <hazmat> g'morning <niemeyer> hazmat: Yo <hazmat> niemeyer, greeting <niemeyer> hazmat: Alan seemed pleasantly surprised with juju <hazmat> niemeyer, awesome! <hazmat> niemeyer, they've been setting up a saas system with plone.. ploud.. i'm glad you guys had a chance to talk <niemeyer> hazmat: Spent good part of Saturday's afternoon hacking a django "platform" charm with Sidnei.. very cool stuff <hazmat> niemeyer, sweet! <niemeyer> hazmat: and he was around for part of it <hazmat> niemeyer, did you guys use bash? or python? <niemeyer> hazmat: Bash for the low level charm integration.. but it's a surprisingly small amount of code <niemeyer> hazmat: Checks the branch out from Launchpad, and puts it live <hazmat> niemeyer, indeed, checking out revision control is the only main thing i used python for <hazmat> on a wsgi charm <hazmat> niemeyer, virtualenv for requirements.. but figuring out the db dependency from config was a little odd <hazmat> made me want runtime dependency declarations <niemeyer> hazmat: Ah, interesting.. we didn't go so far <niemeyer> hazmat: We were thinking about a python-level or app-level hook for installing dependencies <hazmat> niemeyer, yeah.. as it is right now i've got optional relations for lots of things, and i just populate a conf file with the properties for the app <niemeyer> hazmat: Ah, sweet! <hazmat> using gunicorn for serving up the app ( easy multi-process using a pre-fork model) <niemeyer> hazmat: We were going in the direction of an nginx server <hazmat> niemeyer, nginx is a good front end, but i still expect that to be a separate unit.. within a unit the socket bind, fork, saves an extra hop <niemeyer> hazmat: Hadn't heard of gunicorn before <hazmat> which is needed for the frontend anyways <hazmat> niemeyer, its a nice front end to wsgi apps supports several options.. gevent, pre-fork workers, etc. <niemeyer> hazmat: Nice, we should talk to sidnei about this <hazmat> niemeyer, sounds good <hazmat> niemeyer, just a heads up... i'm going to be working on solving transient zk disconnect issues and upstartifying things this week <niemeyer> hazmat: Good timing.. on a call with fwereade and we were just talking about that :-) <hazmat> fwereade, niemeyer, cool, how's the remote repo working coming? <niemeyer> hazmat: Wanna join? <hazmat> niemeyer, definitely <niemeyer> hazmat: sent <robbiew> niemeyer: fyi, I updated the call to use my conference number <robbiew> 1075684916 <niemeyer> robbiew: Joining <niemeyer> Lunch time! <niemeyer> brb <robbiew> FYI -> https://juju.ubuntu.com/Testing <robbiew> :) <SpamapS> cool <SpamapS> http://summit.openstack.org/sessions/view/103 <SpamapS> lol.. look at the example they use for server templates. ;) <SpamapS> "A server template could be used, for example, to build a server containing a pre-installed WordPress system and database" <niemeyer> robbiew: Wow! <niemeyer> robbiew: Integrates very neatly * robbiew is so happy IS added the rawhtml option to our wiki ;) <niemeyer> robbiew: +1 :) <m_3> robbiew: rawhtml... yay * rogpeppe is off. see ya tomorrow. <robbiew> rogpeppe: cya <niemeyer> rogpeppe: Cheers! On a call now, but will review your branches before you're back :) <_mup_> Bug #865550 was filed: Provide automation for including apparmor profiles in charms <juju:New> < https://launchpad.net/bugs/865550 > <niemeyer> Four hours of meetings should be good enough for a Monday * niemeyer tries to actually review some code now <_mup_> juju/unit-info-cli r423 committed by emily84@example.com <_mup_> copy host resolv.conf to container before attempting any resolutions, in the customize chroot, this was being masked by the local package cache * hazmat heads out to doctor appt, bbiab <niemeyer> hazmat: Good luck there! :) <niemeyer> hazmat: When you're back, would like to ping you on a minor <fwereade> niemeyer: ping <niemeyer> fwereade: Yo <fwereade> thanks for the review <fwereade> niemeyer: one I'm about to address and merge, one addressed in a followup, one I intend to address in a followup <niemeyer> fwereade: Cool <niemeyer> fwereade: Branches seem to be missing pre-reqs, btw <fwereade> really? crap, let me check <niemeyer> fwereade: The require-default-series one is.. hmm.. interesting <niemeyer> fwereade: Yeah, it's fine as I can imagine what the stack is <niemeyer> fwereade: But it's good to have that in place in general <fwereade> niemeyer: consider that one speculative, I wasn't sure you'd like it but itseemed better to implement while you were away than to moon around wondering <fwereade> niemeyer: definitely <niemeyer> fwereade: It's good speculation.. it seemed wrong and I was tempted to ask for more information <fwereade> niemeyer: yeah, looks like I've missed at least one :( sorry <niemeyer> fwereade: But pondering on the problem for a while, you have a point <fwereade> niemeyer: it's clint's point really :) <niemeyer> fwereade: _Today_, detecting the revision will yield surprising results more often than it will be useful <niemeyer> fwereade: Well, SpamapS has a point then :-) <fwereade> niemeyer: detecting the series, right? <niemeyer> fwereade: Yeah, sorry <niemeyer> fwereade: Detecting is really the long term solution <niemeyer> fwereade: But, there are two points that make it suboptimal: <niemeyer> 1) There's a single series that work today <niemeyer> 2) We do auto-updating of the environment config, which can yield surprising results <fwereade> niemeyer: 3) we want people to be able to run juju from non-ubuntu systems <fwereade> niemeyer: I do feel it would be nice to be smarter about it, but I think I've convinced myself at least that Least Surprise is the right principle to follow at this point <fwereade> niemeyer: the fact that auto-updating can *still* break this solution is (I think) a point against auto-updating, rather than against the solution <niemeyer> fwereade: 3) is kind of irrelevant in this specific case <fwereade> niemeyer: feels somewhat relevant to the long-term case, but I don't really feel that specific point is going to be a fruitful avenue of discussion right now ;) <niemeyer> fwereade: The fact someone wants to be able to work in environment B isn't good reasoning for not doing something in environment A which is actually more common right now <fwereade> niemeyer: granted <niemeyer> fwereade: Yeah, least surprise is good too <niemeyer> fwereade: and it's actually exactly what we were going at with the original approach, though <niemeyer> fwereade: If you're siting on a machine running Natty, doing things remotely with Natty is expected <niemeyer> fwereade: The problem, and the reason why it has my sympathy, are points 1) and 2) above <niemeyer> fwereade: For 1), natty doesn't exist <niemeyer> (for juju) <fwereade> niemeyer: well, indeed :) <niemeyer> fwereade: re. 2), auto-updating means we'd get different behavior on an _existing_ environment post-bootstrap <niemeyer> fwereade: Which is quite awkward <fwereade> niemeyer: yep <niemeyer> fwereade: I'm happy to move forward with the simplistic way for now.. <niemeyer> fwereade: We'll have to change this once there are other releases, but we have some time to think <niemeyer> fwereade: I'll just check it up with hazmat <fwereade> niemeyer: cool -- I agree it's not perfect, but it feels like a sensible short-term solution <niemeyer> fwereade: +1 <niemeyer> fwereade: Awesome.. just reviewing docs, but I think we're settled as far as our conversation goes <niemeyer> fwereade: We need the auto-revision bumping now, and the fake testing server <niemeyer> fwereade: Probably in that order, so that we can get things rolling into the Ubuntu front while we sort out the testing and all <niemeyer> fwereade: WDYT? <fwereade> yep, sounds good to me <fwereade> niemeyer, in case you didn't see, sounds good to me <niemeyer> fwereade: Cool! <niemeyer> fwereade: Thanks for checking out <niemeyer> fwereade: I'll finish reviewing the doc branch, but we're in sync I think <fwereade> niemeyer: cool <fwereade> niemeyer: there will be another doc tweak at some stage, to fix up the draft I mention <fwereade> niemeyer: but, well, it is a draft :) <niemeyer> fwereade: Yeah, cool :-) <Aram> hello. <niemeyer> Aram: Hey! <niemeyer> hazmat: Can you please check out this when you have a moment: https://code.launchpad.net/~fwereade/juju/require-default-series/+merge/77878 <niemeyer> I'm going to step out for some exercising and be back later today <fwereade> niemeyer: (I've assumed it was; the other place is the new-user tutorial, where every extra character counts, IMO) <fwereade> niemeyer: (and now I've convinced myself I'm wrong: the full charm url will show up in status, and it'll be clearer that the two places reference the same thing if they look like the same thing) * hazmat checks out branch <niemeyer> fwereade: I'm here, but you're probably not.. :p <SpamapS> niemeyer: so, how is the eureka push going? <SpamapS> I've been distracted with other Ubuntu stuff all day <niemeyer> SpamapS: Very well.. the only critical bit is the client-side store work <niemeyer> SpamapS: fwereade's branches are in pretty good shape, though.. we'll probably have everything in by tomorrow, or wednesday the latest <niemeyer> SpamapS: fwereade will then work on a fake server the rest of the week so we can be sure that the store support _actually_ works <niemeyer> SpamapS: and then finish the real store in the next couple of weeks <SpamapS> We can always release an updated client if the API has to change <niemeyer> SpamapS: That's cool, but at least the that work we're finishing tomorrow/wednesday should really be in <niemeyer> Aram: ping <Aram> niemeyer: pong. <niemeyer> Aram: Hey! <SpamapS> niemeyer: indeed, its a bit more of an abuse of the process to SRU a whole new feature in than it is to just SRU a new API version in. :) <niemeyer> SpamapS: Yeah, and also because my _hope_ is that we'll have zero incompatible changes in the 11.10=>12.04 time frame <SpamapS> +1 from me on that <niemeyer> SpamapS: So would really like to have the store work now, so that if we have to SRU changes they use the same user interface <SpamapS> would go a long way to building up user support if they're able to smoothly transition to the PPA and/or 12.04 without having to do anything to their charms/running envs/etc. <niemeyer> SpamapS: Exactly <niemeyer> SpamapS: That's my hope really <SpamapS> Seems like the last of the big structural local storage changes are in with juju-origin and the local charm repository organization. <niemeyer> SpamapS: juju-origin is kind of minor in that regard <SpamapS> I do think we'll have to see some backflips in code as the ZK topology changes.. <niemeyer> SpamapS: the charm url user interface is the big deal <niemeyer> hazmat: ping <niemeyer> SpamapS: another bit we have to sort out is being able to tag formulas as incompatible with a given revision in a nice way <niemeyer> SpamapS: But that seems fine for an SRU <SpamapS> yeah <SpamapS> thats another one where it fits nicely the SRU desire to make sure interoperability with remote services is maintained <niemeyer> Right <hazmat> niemeyer, pong <niemeyer> hazmat: Yo! <niemeyer> hazmat: How're things going there? Still churning, or more like heading to a beer? :) <hazmat> niemeyer, really hoping to have the unit get cli and addresses in relations by default in for oneiric <niemeyer> hazmat: Neat <hazmat> niemeyer, both in the queue <niemeyer> hazmat: I'll check that out today still <SpamapS> hazmat: that would be *saweet* <niemeyer> hazmat: Awesome <hazmat> niemeyer, i'm still running into occasional issues with the local provider stuff, think it might be because i switch local networks so often <niemeyer> hazmat: Hmm <hazmat> not sure, still also seeing an occasional lxc problem when i try to ssh into a machine <hazmat> i committed a fix for the network issue, it always manifests as xmlrpc.launchpad.net temp failure in resolution <SpamapS> I'll gladly do a mass bug file/fix in the charm repo once that is done, as that gives us much better reliability in non-ec2 environments <hazmat> not sure if its a good fix though <hazmat> works for me atm though <hazmat> pty allocation error re lxc problem <hazmat> only a reboot fixes it once it shows up <hazmat> niemeyer, having a look over fwereade's branches atm, still churning <hazmat> and that's not mutually exclusive to a beer ;-) <hazmat> tdd ftw ;-) <hazmat> saw the new db pkg for go <hazmat> elmo was at the surge conference saw him the last day we chatted a bit <niemeyer> hazmat: :-) <niemeyer> hazmat: On that last branch, check-latest-formulas, I think we need something like <niemeyer> /charm-info?charms=<urls> <niemeyer> So that we can get metadata + bundle-sha256 at once <niemeyer> for all the charms <hazmat> niemeyer, sounds good, as we learned rest purisms fails for efficiency <niemeyer> Yeah <niemeyer> It feels bad to do _3_ requests per charm download #juju 2011-10-04 * niemeyer gets some food <SpamapS> Its important to have a nice clear easy REST API to use.. but its vital that you also provide optimisations for batch operations. Its why SQL is so popular.. easy to get one row, easy to get all rows. <niemeyer> jimbaker: ping <niemeyer> hazmat: Still around? <hazmat> niemeyer indeed <niemeyer> hazmat: Cool, sorted out already, again! :-) <niemeyer> hazmat: Review queue pretty much empty <hazmat> niemeyer, nice * hazmat crashes <niemeyer> hazmat: Cheers <_mup_> juju/unit-info-cli r424 committed by emily84@example.com <_mup_> remove the manual copy of host resolv.conf, since customize runs in a chroot, directly modify the resolv.conf output to point to dnsmasq, fix indentation problem <_mup_> juju/env-origin r381 committed by emily84@example.com <_mup_> Merged trunk <SpamapS> FYI r378 caused a segfault when building on natty <SpamapS> https://launchpadlibrarian.net/81865589/buildlog_ubuntu-natty-i386.juju_0.5%2Bbzr378-1juju1~natty1_FAILEDTOBUILD.txt.gz * ejat just wondering â¦ is someone doing charm for liferay :) <hazmat> hmm <hazmat> SpamapS, its a problem with the zk version there <hazmat> 3.3.1 has known issues for juju <hazmat> applies primarily to libzookeeper and python-libzookeeper <hazmat> SpamapS, all the distro ppas (minus oneiric perhaps) should have 3.3.3 <_mup_> Bug #867420 was filed: Add section mentioning expose to the user tutorial. <juju:In Progress by rogpeppe> < https://launchpad.net/bugs/867420 > <TeTeT> just updated my oneiric install, juju seems to have a problem: <TeTeT> Errors were encountered while processing: <TeTeT> /var/cache/apt/archives/juju_0.5+bzr361-0ubuntu1_all.deb <TeTeT> E: Sub-process /usr/bin/dpkg returned an error code (1) <TeTeT> was a transient problem, apt-get update, apt-get -f install seemed to have fixed it <hazmat> interestingly simulating transient disconnection of a bootstrap node for extended periods of time seems to be fine <fwereade> heya niemeyer <SpamapS> hazmat: ahh, we need to add a versioned build dep then <niemeyer> Hello! <hazmat> niemeyer, g'morning <niemeyer> fwereade: How're things going there? <niemeyer> hazmat: Good stuff in these last few branches <fwereade> niemeyer: tolerable :) <niemeyer> fwereade: ;-) <hazmat> niemeyer, yeah.. finally fixed the local provider issue wrt to customization, so all is good there, still seem some occasionally lxc pty allocation errors, but haven't deduced to a reliable reproduction strategy for upstream <hazmat> niemeyer, i did play around with the disconnect scenarios some more, at least for a period of no active usage (no hooks executing, etc), we tolerate zookeeper nodes going away transiently fairly well <niemeyer> hazmat: By zookeeper nodes you mean the server themselves? <niemeyer> servers <hazmat> niemeyer, yeah.. the zookeeper server going away <niemeyer> hazmat: Neat! <niemeyer> It's a good beginning :) <niemeyer> hazmat: we should talk to rogpeppe about the issues we debated yesterday <niemeyer> hazmat: re. making things not fail when possible <rogpeppe> i'm here! <hazmat> niemeyer, for the single server case, the session stays alive, if the client reconnects within the the session timeout period after the server is back up. and the clients all go into poll mode every 3s when the zk server is down (roughly 1/3 session time i believe) <rogpeppe> (afternoon, folks, BTW) <hazmat> niemeyer, theres a few warnings in the zk docs about not trusting library implementation that magic things for the app <hazmat> regarding error handling <hazmat> rogpeppe, hola <niemeyer> hazmat: Well, sure :) <niemeyer> hazmat: That's what the whole session concept is about, though <niemeyer> rogpeppe: This goes a bit in the direction you were already thinking <niemeyer> rogpeppe: You mentioned in our conversations that e.g. it'd be good that Dial would hold back until the connection is actually established <niemeyer> rogpeppe: This is something we should do, but we're talking about doing more than that <rogpeppe> i don't know if this is relevant, or if it's a problem with gozk alone, but i never got a server-down notification from a zk server, even when i killed it and waited 15 mins. <niemeyer> rogpeppe: Try bringing it up afterwards! :-) <hazmat> rogpeppe, session expiration is server governed, clients don't decide that <niemeyer> rogpeppe: It's a bit strange, but that's how it works.. the session times out in the next reconnection <rogpeppe> niemeyer: yeah, i definitely think it should <hazmat> rogpeppe, the clients go into a polling reconnect mode, turning up the zookeeper debug log verbosity will show the activity <rogpeppe> hazmat: but what if there's no server? surely the client should fail eventually? <niemeyer> rogpeppe: So, in addition to this, when we are connected and zk disconnects, we should also block certain calls <niemeyer> rogpeppe: Well.. all the calls <hazmat> rogpeppe, nope.. they poll endlessley in the background, attempting to use the connection will raise a connectionloss/error <hazmat> rogpeppe, at least until the handle is closed <niemeyer> rogpeppe: So that we avoid these errors ^ <hazmat> rogpeppe, that's why we have explicit timeouts for connect <niemeyer> rogpeppe: In other words, if we have a _temporary_ error (e.g. disconnection rather than session expiration), we should block client calls <hazmat> above libzk <rogpeppe> hazmat: but if all users are blocked waiting for one of {connection, state change}, then no one will try to use the connection, and the client will hang forever <niemeyer> rogpeppe: Not necessarily.. as you know it's trivial to timeout and close a connection <niemeyer> rogpeppe: I mean, on our side <rogpeppe> so all clients should do timeout explicitly? <niemeyer> rogpeppe: <-time.After & all <rogpeppe> sure, but what's an appropriate time out? <niemeyer> rogpeppe: Whatever we choose <niemeyer> rogpeppe: But that's not what we're trying to solve now <rogpeppe> sure <niemeyer> rogpeppe: What we have to do is make the gozk interface bearable <niemeyer> rogpeppe: Rather than a time bomb <hazmat> so we're trying to make recoverable error handling subsumed into the client <rogpeppe> [note to future: i'd argue for the timeout functionality to be inside the gozk interface, not reimplemented by every client] <niemeyer> [note to future: discuss timeout with rogpeppe] <hazmat> by capturing a closure for any operation, and on connection error, wait till the connection is restablished, and rexecing the closure (possibly with additional error detection semantics) <rogpeppe> hazmat: are we talking about the gozk package level here? <niemeyer> hazmat: I think there's a first step before that even <rogpeppe> or a higher juju-specific level? <niemeyer> rogpeppe: Yeah, internal to gozk <hazmat> rogpeppe, which pkg isn't relevant, but yes at the zk conn level <hazmat> http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling <niemeyer> hazmat: Before we try to _redo_ operations, we should teach gozk to not _attempt_ them in the first place when it knows the connection is off <hazmat> hmm <hazmat> yeah.. thats better <hazmat> we can basically watch session events and hold all operations <hazmat> niemeyer, +1 <niemeyer> hazmat: Cool! <hazmat> hmm <hazmat> niemeyer, so there is still a gap <niemeyer> rogpeppe: Does that make sense to you as well? * rogpeppe is thinking hard <niemeyer> hazmat: There is in some cases, when we attempt to do something and the tcp connection crashes on our face <hazmat> niemeyer, internally libzk will do a heartbeat effectively to keep the session alive, if the op happens before the heartbeat detects dead we still get a conn error <niemeyer> hazmat: Let's handle that next by retrying certain operations intelligently <rogpeppe> i think the first thing is to distinguish between recoverable and unrecoverable errors <hazmat> rogpeppe, its a property of the handle <niemeyer> rogpeppe: That's the next thing, after the initial step we mentioned above <hazmat> libzk exposes a method for it to return a bool <hazmat> recoverable(handle) <niemeyer> rogpeppe: For blocking operations on certain connection states, we're actually preventing the error from even happening <rogpeppe> preventing the error being exposed to the API-client code, that is, yes? <niemeyer> rogpeppe: No <hazmat> rogpeppe, yup <hazmat> :-) <rogpeppe> lol <niemeyer> rogpeppe: Preventing it from happening at all <hazmat> the error never happens <hazmat> because we don't let the op go through while disconnected <niemeyer> rogpeppe: The error never happens if we don't try the call <rogpeppe> ok, that makes sense. <rogpeppe> but... what about an op that has already gone through <rogpeppe> ? <hazmat> next step is to auto recover the error for ops that we can do so without ambiguity, because there is still a gap on our detection of the client connectivity <rogpeppe> and then the connection goes down <niemeyer> rogpeppe: That's the next case we were talking about above <niemeyer> rogpeppe: If the operation is idempotent, we can blindly retry it behind the lib client's back <rogpeppe> niemeyer: do we need to? i thought it was important that clients be prepared to handle critical session events <niemeyer> rogpeppe: If the operation is not idempotent, too bad.. we'll have to let the app take care of it <hazmat> rogpeppe, effectively the only only ops i've seen ambiguity around is the create scenario, and modifications without versions <niemeyer> rogpeppe: Do we need to what? <rogpeppe> do we need to retry, was my question. <hazmat> so this might be better structured as a library on top of the connection that's specific to juju <niemeyer> rogpeppe: Yeah, because otherwise we'll have to introduce error handling _everywhere_, doing exactly the same retry <niemeyer> hazmat: Nah.. let's do it internally and make a clean API.. we know what we're doing <rogpeppe> does zookeeper do a 3 phase commit? <hazmat> niemeyer, famous last words ;-) <rogpeppe> i.e. for something like create with sequence number, does the client have to acknowledge the create before the node is actually created? <niemeyer> hazmat: Well, if we don't, we have larger problems ;-) <hazmat> rogpeppe, its a paxos derivative internally. everything forwards to the active leader in the cluster <hazmat> writes that is <hazmat> it transparently does leader election as needed <niemeyer> rogpeppe: The _client_ cannot acknowledge the create <hazmat> rogpeppe, the client doesn't ack the create, but the error recovery with a sequence node is hard, because without the server response, we have no idea what happened <rogpeppe> niemeyer: why not? i thought the usual process was: write request; read response; write ack; server commits <niemeyer> rogpeppe: What's the difference? <niemeyer> rogpeppe: write ack; read response; write ack; read response; write ack; read response; server commits <rogpeppe> niemeyer: the difference is that if the server doesn't see an ack from the client, the action never happened. <niemeyer> rogpeppe: Doesn't matter how many round trips.. at some point the server will commit, and if the connection crashes the client won't know if it was committed or not <hazmat> ? there's client acks under the hood? <niemeyer> hazmat: There isn't.. and I'm explaining why it makes no difference <hazmat> ah * hazmat dogwalks back in 15 <niemeyer> hazmat: Cheers <rogpeppe> if the connection crashes, the client can still force the commit by writing the ack. it's true that it doesn't know if the ack is received. hmm. byzantine generals. <niemeyer> Yeah <rogpeppe> i'm slightly surprised the sequence-number create doesn't have a version argument, same as write <niemeyer> rogpeppe: Hmm.. seems to be sane to me? <rogpeppe> that would fix the problem, at the expense of retries, no? <niemeyer> rogpeppe: It's atomic.. it's necessarily going to be version 0 <rogpeppe> ah, child changes don't change a version number? * rogpeppe goes back to look at the modify operation <niemeyer> rogpeppe: It changes, but it makes no sense to require a given version with a sequence number <niemeyer> rogpeppe: The point of using the sequence create is precisely to let the server make concurrent requests work atomically <niemeyer_> Hmm <niemeyer_> Weird <niemeyer_> Abrupt disconnection <rogpeppe> niemeyer: but we want to do that with node contents too - that's why the version number on Set <rogpeppe> niemeyer_: and that's the main problem with the lack of Create idempotency <rogpeppe> anyway, we could easily document that Create with SEQUENCE is a special case <rogpeppe> and can return an error without retrying <niemeyer_> rogpeppe: We don't even have to document it really.. the error itself is the notice <rogpeppe> i think it would be good if the only time a session event arrived at a watcher was if the server went down unrecoverably <rogpeppe> actually, that doesn't work <rogpeppe> watchers will always have to restart <niemeyer_> rogpeppe: That's how it is today, except for the session events in the session watch <niemeyer_> rogpeppe: Not really <niemeyer_> rogpeppe: If the watch was already established, zk will keep track of them and reestablish internally as long as the session survives <rogpeppe> but what if the watch reply was lost when the connection went down? <niemeyer_> rogpeppe: Good question.. worth confirming to see if it's handled properly <rogpeppe> i'm not sure how it can be <rogpeppe> the client doesn't ack watch replies AFAIK <niemeyer_> rogpeppe: There are certainly ways it can be.. it really depends on how it's done <niemeyer_> rogpeppe: E.g. the client itself can do the verification on connection reestablishment <niemeyer_> Another alternative, which is perhaps a saner one, is to do a 180â° turn and ignore the existence of sessions completely <niemeyer_> Hmmm.. <rogpeppe> niemeyer_: that would look much nicer from a API user's perspective <niemeyer_> I actually like the sound of that <niemeyer_> rogpeppe: Not even thinking about API.. really thinking about how to build reliable software on top of it <rogpeppe> aren't those closely related things? <niemeyer_> rogpeppe: Not necessarily.. an API that reestablishes connections and knows how to hanndle problems internally is a lot nicer from an outside user's perspective <rogpeppe> niemeyer: don't quite follow <niemeyer> rogpeppe: Don't worry, it's fine either way * hazmat catches up <niemeyer> hazmat: I think we should do a U turn <hazmat> niemeyer, how so? <hazmat> hmm.. verifying watch handling while down sounds good <hazmat> connection down that is <niemeyer> hazmat: We're adding complexity in the middle layer, and reality is that no matter how complex and how much we prevent the session from "crashing", we _still_ have to deal with session termination correctly <hazmat> session termination is effectively fatal <rogpeppe> when does a session terminate? <hazmat> the only sane thing to do is to restart the app <niemeyer> hazmat: we're also constantly saying "ah, but what if X happens?".. <hazmat> rogpeppe, a client is disconnected from the quorum for the period of session timeout <niemeyer> hazmat: Not necessarily.. we have to restart the connection <hazmat> niemeyer, and reinitialize any app state against the new connection <niemeyer> hazmat: Yes <hazmat> ie. restart the app ;-) <niemeyer> hazmat: No, restart the app is something else <niemeyer> hazmat: Restart the app == new process <hazmat> doesn't have to be a process restart to be effective, but it needs to go through the entire app init <niemeyer> hazmat: So, the point is that we have to do that anyway <niemeyer> hazmat: Because no matter how hard we try, that's a valid scenario <hazmat> rogpeppe, the other way a session terminates is a client closes the handle, thats more explicit <hazmat> rogpeppe, that can be abused in testing by connecting multiple clients via the same session id, to simulate session failures <hazmat> niemeyer, absolutely for unrecoverable errors that is required <niemeyer> hazmat: So what about going to the other side, and handling any session hiccups as fatal? It feels a lot stronger as a general principle, and a lot harder to get it wrong <rogpeppe> when you say "reinitialize any app state", doesn't that assume that no app state has already been stored on the server? <hazmat> for recoverable errors local handling inline to the conn, seems worth exploring <rogpeppe> or are we assuming that the server is now a clean slate? <hazmat> we need to validate some of the watch state <hazmat> rogpeppe, no the server has an existing state <niemeyer> hazmat: The problem is that, as we've been seeing above, "recoverable errors" are actually very hard to really figure <hazmat> rogpeppe, the app needs to process the existing state against its own state needs and observation requirements <niemeyer> hazmat: rogpeppe makes a good point in terms of the details of watch establishment <rogpeppe> so presumably we know almost all of that state, barring operations in progress? <niemeyer> hazmat: and I don't have a good answer for him <hazmat> niemeyer, that's why i was going with a stop/reconnect/start for both error types as a simple mechanism <hazmat> for now * hazmat does a test to verify watch behavior <niemeyer> hazmat: Yeah, but the problem we have _today_ and that I don't feel safe doing that is that we don't have good-but-stay-alive semantics in the code base <niemeyer> erm.. <niemeyer> good stop-but-stay-alive <rogpeppe> i *think* that the most important case is automatic retries of idempotent operations. <hazmat> niemeyer, we do in the unit agents as a consequence of doing upgrades, we pause everything for it <rogpeppe> but that's hard too. <niemeyer> hazmat: I seriously doubt that this will e.g. kill old watches <hazmat> niemeyer, effectively the only thing that's not observation driven is the provider agent does some polling for runaway instances <hazmat> niemeyer, it won't kill old watches, but we can close the handle explicitly <niemeyer> hazmat: and what happens to all the deferreds? <hazmat> niemeyer, their dead, when the session is closed <hazmat> at least for watches <niemeyer> hazmat: What means dead? Dead as in, they'll continue in memory, hanging? <hazmat> niemeyer, yeah... their effectively dead, we can do things to clean them up if that's problematic <hazmat> dead in memory <niemeyer> hazmat: Yeah.. so if we have something like "yield exists_watch", that's dead too.. <hazmat> we can track open watches like gozk and kill them explicitly (errback disconnect) <niemeyer> hazmat: That's far from a clean termination <hazmat> niemeyer, we can transition those to exceptions <niemeyer> hazmat: Sure, we can do everything we're talking about above.. the point is that it's not trivial <hazmat> it seems straightforward at the conn level <hazmat> to track watches, and on close kill them <niemeyer> hazmat: Heh.. it's straightforward to close() the connection, of course <niemeyer> hazmat: It's not straightforward to ensure that doing this will yield a predictable behavior <hazmat> so back to process suicide ;-) <niemeyer> hazmat: Cinelerra FTW! <rogpeppe> this is all talking about the situation when you need to explicitly restart a session, right? <hazmat> rogpeppe, yes <niemeyer> rogpeppe: Yeah, control over fault scenarios in general <hazmat> restart/open a new session <rogpeppe> restart is different, i thought <rogpeppe> because the library can do it behind the scenes <rogpeppe> and reinstate watches <rogpeppe> redo idempotent ops, etc <hazmat> rogpeppe, but it can't reattach the watches to the all extant users? <rogpeppe> i don't see why not <hazmat> perhaps in go that's possible with channels and the channel bookeeping <hazmat> against the watches <niemeyer> hazmat, rogpeppe: No, that doesn't work in any case <rogpeppe> ? <niemeyer> The window between the watch being dead and the watch being alive again is lost <rog> of course <rog> doh <rog> except... <rog> that the client *could* keep track of the last-returned state <rog> and check the result when the new result arrives <rog> and trigger the watcher itself if it's changed <niemeyer> rog: Yeah, we could try to implement the watch in the client side, but that's what I was talking above <rog> expect... i don't know if {remove child; add child with same name} is legitimately a no-op <niemeyer> rog: We're going very far to avoid a situation that is in fact unavoidable <niemeyer> rog: Instead of doing that, I suggest we handle the unavoidable situation in all cases <rog> force all clients to deal with any session termination as if it might be unrecoverable? <niemeyer> rog: Yeah <niemeyer> rog: Any client disconnection in fact <niemeyer> rog: let's also remove the hack we have in the code and allow watches to notice temporary disconnections <rog> this is why proper databases have transactions <niemeyer> rog: Uh.. <niemeyer> rog: That was a shoot in the sky :-) <rog> if the connection dies half way through modifying some complex state, then when retrying, you've got to figure out how far you previously got, then redo from there. <niemeyer> rog: We have exactly the same thing with zk. <niemeyer> rog: The difference is that unlike a database we're using this for coordination <niemeyer> rog: Which means we have live code waiting for state to change <niemeyer> rog: A database client that had to wait for state to change would face the same issues <hazmat> rog, databaess still have the same issue <rog> yeah, i guess <rog> what should a watcher do when it sees a temporary disconnection? <rog> await reconnection and watch again, i suppose <hazmat> so watches don't fire if the event happens while disconnected <rog> i wonder if the watch should terminate even on temporary disconnection. <niemeyer> rog: It should error out and stop whatever is being done, recovering the surrounding state if it makes sense <niemeyer> rog: Right, exactly <rog> and is that true of the Dial session events too? the session terminates after the first non-ok event? <rog> i think that makes sense. <rog> (and it also makes use of Redial more ubiquitous). [of course i'm speaking from a gozk perspective here, as i'm not familiar with the py zk lib] <hazmat> interesting, i get a session expired event.. just wrote a unit test for watch fire while disconnected, two server cluster, two clients one connected to each, one client sets a watch, shutdown its server, delete on the other client/server, resurrect the shutdown server with its client waiting on the watch, gets a session expired event <hazmat> hmm. its timing dependent though <niemeyer> rog: Yeah, I think so too <hazmat> yeah.. this needs more thought <niemeyer> hazmat: Yeah, the more we talk, the more I'm convinced we should assume nothing from a broken connection <hazmat> niemeyer, indeed <niemeyer> hazmat: This kind of positioning also has a non-obvious advantage.. it enables us to more easily transition to doozerd at some point <niemeyer> Perhaps not as a coincidence, it has no concept of sessions * niemeyer looks at Aram <hazmat> niemeyer, interesting.. i thought you gave up on doozerd <hazmat> upstream seems to be dead afaik <niemeyer> hazmat: I have secret plans! <niemeyer> ;-) <hazmat> niemeyer, cool when i mentioned it b4 you seemed down on it <hazmat> it would be nice for an arm env to go java-less <niemeyer> hazmat: Yeah, because it sucks on several aspects right now <hazmat> rog, on ReDial does gozk reuse a handle? <niemeyer> hazmat: But what if we.. hmmm.. provided incentives for the situation to change? :-) <hazmat> niemeyer, yeah.. persistence and error handling there don't seem well known <hazmat> niemeyer, indeed, things can change <rog> hazmat: no, i don't think so, but i don't think it needs to. <hazmat> that's one important difference between gozk/txzk.. the pyzk doesn't expose reconnecting with the same handle, which toasts extant watches (associated to handle) when trying to reconnect to the same session explictly <rog> (paste coming up) <hazmat> libzk in the background will do it, but if you want to change the server explicitly at the app/client level its a hoser <rog> if you get clients to explicitly negotiate with the central dialler when the connection is re-made, i think it can work. <rog> i.e. get error indicating that server is down, ask central thread for new connection. <rog> store that connection where you need to. * niemeyer loves the idea of keeping it simple and not having to do any of that :) <rog> yeah, me too. <rog> but we can't. at least i think that's the conclusion we've come to, right? <niemeyer> rog: Hmm, not my understanding at least <niemeyer> rog: It's precisely the opposite.. we _have_ to do that anyway <niemeyer> rog: Because no matter how hard we try, the connection can break for real, and that situation has to be handled properly <niemeyer> rog: So I'd rather focus on that scenario all the time, and forget about the fact we even have sessions <rog> niemeyer: so you're saying that we have to lose all client state when there's a reconnection? <niemeyer> rog: Yes, I'm saying we have to tolerate that no matter what <rog> so the fact that zk has returned ok when we've created a node, we have to act as if that node might not have been created? <rog> s/the fact/even if/ <niemeyer> rog: If zk returned ok, there's no disconnection <rog> niemeyer: if it returned ok, and the next create returns an error; that's the scenario i'm thinking of <rog> that's the situation where i think the node creator could wait for redial and then carry on from where it was <niemeyer> rog: If it returned an error, we have to handle it as an error and not assume that the session is alive, because it may well not be <niemeyer> rog: and what if the session dies? <rog> i'm not saying that it should assume the session is still alive <rog> i'm saying that when it gets an error, it could ask the central thread for the new connection - it might just get an error instead <rog> the node creator is aware of the transition, but can carry on (knowingly) if appropriate <niemeyer> rog: and what about the several watches that are established? <rog> niemeyer: same applies <niemeyer> rog: What applies? <rog> the watch will return an error; the code doing the watch can ask for a new connection and redo the watch if it wishes. <hazmat> niemeyer, so we're back to reinitializing the app on any connectoin error, disregarding recoverable <rog> no redoing behind the scenes, but the possibility of carrying on where we left off <niemeyer> rog: The state on which the watch was requested has changed <niemeyer> rog: Check out the existing code base <hazmat> niemeyer, so interestingly we can be disconnected, not know it, and miss a watch event <niemeyer> rog: It's not trivial to just "oh, redo it again" <rog> niemeyer: it doesn't matter because the watcher is re-requesting the state, so it'll see both the state and any subsequent watch event <niemeyer> hazmat: Yeah, that's exactly the kind of very tricky scenario that I'm concerned about <rog> the watcher has to deal with the "state just changed" scenario anyway when it first requests the watch <hazmat> niemeyer, actually we get notification from a session event that we reconnected <niemeyer> hazmat: As Russ would say, I don't really want to think about whether it's correct or not <niemeyer> rog: No.. please look at the code base <rog> niemeyer: sorry, which bit are you referring to? <niemeyer> rog: We're saying the same thing, in fact.. you're just underestimating the fact that "just retry" is more involved than "request the new connection and do it again" <niemeyer> rog: juju <niemeyer> rog: lp:juju <niemeyer> rog: This concept touches the whole application <rog> niemeyer: i've been exploring it a bit this morning, but haven't found the crucial bits, i think. what's a good example file that would be strongly affected by this kind of thing? <niemeyer> hazmat: We do.. the real problem is ensuring state is as it should be when facing reconnections <niemeyer> rog: I'm serious.. this touches the whole app <hazmat> niemeyer, right we always have to reconsider state on reconnection <niemeyer> rog: Check out the agents <rog> niemeyer: ah, ok. i was looking in state <rog> thanks <niemeyer> rog: state is good too <niemeyer> rog: Since it's what the agents use and touches this concept too <niemeyer> rog, hazmat: So, my suggestion is that the first thing we do is to unhide temporary failures in gozk <hazmat> niemeyer, the test becomes alot more reliable when we have multiple zks in the cluster setup for the client to connect to <rog> sgtm <niemeyer> rog, hazmat: Then, let's watch out for that kind of issue very carefully in reviews and whatnot, as we build a reliable version <niemeyer> hazmat: The same problem exists, though.. <hazmat> niemeyer, indeed <hazmat> but it minimizes total disconnect scenarios with multiple zks <niemeyer> hazmat: Even if it _immediately_ reconnects, the interim problems may have created differences that are easily translated into bugs very hard to figure out <niemeyer> hazmat: and again, we seriously _have_ to handle the hard-reconnect across the board <hazmat> niemeyer, agreed <hazmat> niemeyer, i'm all in favor of simplifying and treating them the same <hazmat> recoverable/unrecoverable conn errors <niemeyer> hazmat: So no matter how much we'd love to not break the session and have a pleasant API, the hard reconnects means we'll need the good failure recovered either way <niemeyer> So we can as well plan for that at all times <hazmat> niemeyer, its detecting the conn error that i'm concerned about atm <niemeyer> hazmat: My understanding is that the client always notifies about temporary issues <hazmat> niemeyer, based on its internal poll period to the server <hazmat> niemeyer, a quick disconnect masks any client detection <hazmat> ^transient <niemeyer> hazmat: Really!? <hazmat> it seems the server will attempt to expire the client session, but i've seen once we're instead it shows a reconnect <hazmat> s/we're/where <niemeyer> hazmat: I can't imagine how that'd be possible <niemeyer> hazmat: The client lib should hopefully notify the user that the TCP connection had to be remade <hazmat> niemeyer, fwiw here's the test i'm playing with (can drop into test_session.py ).. http://paste.ubuntu.com/702290/ <hazmat> for a package install of zk.. ZOOKEEPER_PATH=/usr/share/java <hazmat> for the test runner <niemeyer> hazmat: Hmm <niemeyer> hazmat: That seems to test that watches work across reconnections <niemeyer> hazmat: We know they can work <hazmat> niemeyer, they do but we miss the delete <niemeyer> hazmat: Or am I missing something? <niemeyer> hazmat: Ah, right! <hazmat> with no notice <niemeyer> hazmat: So yeah, it's total crack <hazmat> niemeyer, actually most of the time we get an expired session event in the client w/ the watch <hazmat> like 99% <hazmat> if i connect the client to multiple servers it sees the delete <hazmat> w/ the watch that is <niemeyer> hazmat: Hmm.. interesting.. so does it keep multiple connections internally in that case, or is it redoing the connection more quickly? <hazmat> niemeyer, not afaick, but its been a while since i dug into that <hazmat> niemeyer, but as an example here's one run http://paste.ubuntu.com/702291/ <hazmat> where it does get the delete event <hazmat> but that's not guaranteed in allop <niemeyer> hazmat: If you _don't_ create it on restart, does it get the notification? <niemeyer> hazmat: Just wondering if it might be joining the two events <hazmat> niemeyer, no it still gets the deleted event if gets an event, else it gets session expired <hazmat> but its easy to construct it so it only sees the created event <hazmat> if toss a sleep in <hazmat> perhaps not <hazmat> it seems to get the delete event or session expiration.. i need to play with this some more and do a more thought out write up <hazmat> in some cases it does get the created event, obviously the pastebin has that <niemeyer> hazmat: I see, cool <niemeyer> On a minor note, filepath.Rel is in.. can remove our internal impl. now <hazmat> niemeyer, cool <niemeyer> That was a tough one :) <niemeyer> fwereade: Leaving to lunch soon.. how's stuff going there? <niemeyer> fwereade: Can I do anything for you? <niemeyer> jimbaker: How's env-origin as well? <jimbaker> niemeyer, just need to figure out the specific text for the two scenarios you mention <niemeyer> jimbaker: Hmmm.. which text? <jimbaker> niemeyer, from apt-cache policy <niemeyer> jimbaker: Just copy & paste from the existing test? Do you want me to send a patch? <jimbaker> niemeyer, well it's close to being copy & paste, but the difference really matters here <jimbaker> if you have a simple patch, for sure that would be helpful <niemeyer> jimbaker: Sorry, I'm still not sure about what you're talking about <niemeyer> jimbaker: It seems completely trivial t ome <niemeyer> jimbaker: Sure.. just a sec <jimbaker> niemeyer, i was not familiar with apt-cache policy before this work. obviously once familiar, it is trivial <niemeyer> jimbaker: I'm actually talking about the request I made in the review.. <niemeyer> jimbaker: But since you mention it, I actually provided you with a scripted version saying exactly how it should work like 3 reviews ago <jimbaker> niemeyer, i'm going against http://carlo17.home.xs4all.nl/howto/debian.html#errata for a description of the output format <hazmat> the python-apt bindings are pretty simple too.. i used them for the local provider.. although its not clear how you identify a repo for a given package from it <jimbaker> niemeyer, if you have a better resource describing apt-cache policy, i would very much appreciate it <jimbaker> hazmat, one advantage of such bindings is the data model <hazmat> jimbaker,
{'channel': 'juju', 'month': 10}