#ubuntu-ensemble 2011-06-01 <SpamapS> 2011-05-31 16:17:46,019 ERROR Formula %r is the latest revision known <SpamapS> DOH <SpamapS> Heh.. munin really is too much for a t1.micro. :-/ <SpamapS> http://ec2-50-16-148-122.compute-1.amazonaws.com/munin/ <SpamapS> mmmmmm... graphs <SpamapS> hazmat: hit the problem again. :( <bcsaller> SpamapS: nice graphs :) <SpamapS> bcsaller: I'm besieging the cluster righ tnow.... about to get more interesting! :) <bcsaller> press record :) <SpamapS> top - 00:00:59 up 6:56, 1 user, load average: 12.00, 4.46, 1.65 <SpamapS> 2011-05-31 16:59:30,485 provision:ec2: ensemble.agents.provision INFO: Starting machine id:11 ... <SpamapS> bcsaller: I'm thinking of writing a "test monster" formula that will crawl and try to destroy a website.. :) <bcsaller> terrifying <SpamapS> blammo.. load drops to 4 new box up <SpamapS> I also wonder if my t1.micro's are getting throttled <SpamapS> might be interesting to reboot a few as c1.medium <SpamapS> hmm. I really should be hitting the 10.x address so I can just leave this running all night. ;) <hazmat> SpamapS, do you have the relation hook installing munin-node? <SpamapS> hazmat: yes <SpamapS> hazmat: will have to discuss later, super late heading out the door.. :-P <SpamapS> hazmat: any tips though, let me know I'll give it a shot <hazmat> SpamapS, me too.. cheers <hazmat> SpamapS, i have to rebuild the txzk packages to get that fix pushed out.. i'll have a look at it in the morning <_mup_> ensemble/expose-provision-service-hierarchy r255 committed by amy08@example.org <_mup_> Updated watch_exposed_flag (about to be moved into separate branch) <_mup_> ensemble/expose-watch-exposed-flag r240 committed by amy08@example.org <_mup_> Changes to watch_exposed_flag moved from expose-provision-service-hierarchy <_mup_> ensemble/expose-provision-service-hierarchy r256 committed by amy08@example.org <_mup_> Removed watch_exposed_flag changes in this branch <_mup_> ensemble/expose-provision-service-hierarchy r257 committed by amy08@example.org <_mup_> Merged in expose-watch-exposed-flag branch <_mup_> ensemble/expose-provision-service-hierarchy r258 committed by amy08@example.org <_mup_> Nonexistent service terminates corresponding watch on exposed flag <_mup_> ensemble/expose-provision-service-hierarchy r259 committed by amy08@example.org <_mup_> PEP8, removed unnecessary debug statements <_mup_> ensemble/expose-provision-service-hierarchy r260 committed by amy08@example.org <_mup_> Better docstrings + cleanup <_mup_> Bug #791035 was filed: removed formula components are not cleaned up when upgrading <Ensemble:New> < https://launchpad.net/bugs/791035 > <_mup_> Bug #791042 was filed: *-relation-broken has no way to identify which remote service is being broken <Ensemble:New> < https://launchpad.net/bugs/791042 > <kim0> ERROR Formula %r is the latest revision known <kim0> Is %r supposed to be substitued for by something <kim0> anyone around today <niemeyer> Good morning all! <kim0> Morning :) <niemeyer> kim0: Hey! * kim0 trying to write a new formula <kim0> hmm .. in debug-hooks, # relation-list --format json <kim0> why am I getting No ENSEMBLE_AGENT_SOCKET/-s option found <hazmat> kim0, are you in the window that popped up for the hook or just the default terminal? <hazmat> re debug <kim0> the one that poped <hazmat> kim0, if you do ENV | grep ENSEMBLE what do you see? <hazmat> er. env <kim0> sorry, I had closed the window .. perhaps I shouldn't have <hazmat> kim0, no worries, if it happens again let me know, i'd be happy to take a look <kim0> hazmat: does debug-hooks execute the hook + give me shell, or just shell <hazmat> kim0, hook + shell <hazmat> the interactive shell replaces the hook execution <hazmat> the exit of the shell is the considered the exit of the hook <kim0> so the original hook script is NOT executed <hazmat> kim0, correct <niemeyer> hazmat: Good morning * hazmat is on his 3rd unity reboot of the last 24hrs. <hazmat> niemeyer, g'morning <niemeyer> hazmat: From the logs, it looks like SpamapS got the same issue again last night <hazmat> niemeyer, he did, i did a fix for the top level symptom which is committed on txzookeeper trunk <hazmat> niemeyer, i'm still tracking down in the zookeeper client where/why the event occurs, its a session event <hazmat> so it hits all the watchers extant <niemeyer> hazmat: I'm not sure that really fixes it.. right.. it's a session timeout, so it'll continue breaking <hazmat> niemeyer, indeed <niemeyer> hazmat: Btw, the quotes were inconsistent, but that's minor <hazmat> niemeyer, i'm wondering if we need some lower level bookeeping in txzk client to be able to attach a handler for session events, what's unclear to me atm is if its a session expired scenario or not ( ie. recreate all watches and ephemerals) <niemeyer> hazmat: Is the session issue happening before or after the already called error? <hazmat> before <niemeyer> hazmat: SpamapS said something about a micro machine being too small for whatever he was doing.. it may just be locked up completely for so long that it times out <hazmat> niemeyer, yeah.. i think we need to up our session timeout values on the zk server, there's a faq on this for ec2 <hazmat> and the latencies inherent in that environment <niemeyer> hazmat: Sounds good, if he's able to reproduce it easily, it would be good to try again under more controlled conditions <hazmat> but that doesn't excuse that we need some handling for this behavior <niemeyer> hazmat: Indeed <niemeyer> hazmat: In gozk, I redirect session events to a specific channel <hazmat> niemeyer, yeah.. i'm going to try and reproduce via manipulation of the ec2 firewall on the bootstrap node <niemeyer> hazmat: and as an experiment, I'm panicing the application if the session event isn't handled in a given amount of time <hazmat> niemeyer, yeah.. that's basically along the lines of what i was thinking for txzk <niemeyer> hazmat: IOW, if the app isn't acknowledging the fact a session event happened <hazmat> use a dedicated session event handler, and route those events there. <hazmat> we'd also need to track the outstanding watches if we want to reduce the redunancy from the zk c client <niemeyer> hazmat: Either way, we'll likely have to kill all the running callbacks <hazmat> which notifies on all watchers <kim0> hmm, why does relation-list only result in ["mysql/0"] (no database, user, password ) ..etc This is in db-relation-changed second invocation <niemeyer> kim0: That's the relation name, not really what you want <niemeyer> kim0: Check relation-get <hazmat> niemeyer, it depends.. does it mean the session is expired.. or not <niemeyer> hazmat: I think the only thing coming through those channels is up/down notes <niemeyer> hazmat: But it'd be worth double checking <kim0> relation-get - --format json results in {} <niemeyer> kim0: That means the other side hasn't relation-set anything <kim0> facing this consistently for the past hour .. I used remove-relation and add-relation many times <kim0> the other side is mysql from examples <niemeyer> kim0: Hmm <kim0> should I login and check if a DB has been created for my instance (drupal) ? <hazmat> kim0, if your debugging both sides interactively <kim0> just one side <hazmat> the values aren't flushed for the remote to see till the hook/debug is ended <hazmat> er. the debug window for the hook is ended <kim0> hazmat: I'm not debugging mysql side, so that shouldn't be a problem corrrect <niemeyer> kim0: Kind of.. <niemeyer> kim0: Try logging out and in again, just to try it out <niemeyer> kim0: From the debug-hook session <hazmat> kim0, are you in the relation join event or the changed event? <kim0> hazmat: in "changed" in its 2nd invocation <hazmat> kim0, hmm. that sounds like the right place then <kim0> yeah and I've waited for mysql to create its stuff <niemeyer> kim0: Try logging out and in <niemeyer> kim0: From the debug session <kim0> ok <niemeyer> kim0: and then do the relation-get <niemeyer> Oh, I think there's a bug, now that you mention it.. <kim0> niemeyer: so I closed the screen window <kim0> opening it again <niemeyer> kim0: Yeah, I suspect it won't work <niemeyer> kim0: Our example is doing some magic it shouldn't <kim0> niemeyer: so I should kill mysql and start a new one ? <niemeyer> kim0: Follow with me so that you can actually fix it and try again <kim0> ok <niemeyer> kim0: Go to the mysql formula, and look at the joined hook <kim0> opened <niemeyer> kim0: You see that check under the comment # Determine if ... <kim0> yeah <niemeyer> kim0: You see, it's bumping out of the hook if the _service_ already has a database created <kim0> which makes sense to me ?! <kim0> what's the problem <kim0> ah <niemeyer> kim0: The problem is that it bumps out without ever letting the _relation_ to know of the settings <kim0> so it's not relation-set'ing <niemeyer> kim0: yep <kim0> got it <kim0> doh <kim0> but it must have relation-set'ed the first time <kim0> is that lost ? <niemeyer> kim0: If you <niemeyer> kim0: Lost? <kim0> why is the relation-set from first time not available ? <niemeyer> kim0: Oh, yeah, it seems to be <kim0> ok <hazmat> if the service name is reused <kim0> I've just been remove-relation and add-relation <hazmat> ie relate worpdress mysql.. destroy wordpress, recreate wordpress and relate <niemeyer> hazmat: Ugh.. that's bad <hazmat> yeah.. i'm not sure this was a problem before with the mysql formula <niemeyer> hazmat: The service name is not being "reused".. the service is still around <hazmat> kim0, are you using the example or principia? <kim0> hazmat: example <niemeyer> hazmat: It's just another relation <hazmat> niemeyer, yes.. its another relation, but the association to the same related service name in the db is still present <niemeyer> hazmat: Yes, but we shouldn't destroy the database like that no matter what <hazmat> i think the principia (based on the original python) doesn't have this issue <hazmat> yeah.. it doesn't <hazmat> this is a bug in the mysql formula <hazmat> in the examples directory <hazmat> the original python one would just skip the db creation, but still set the credentials.. the shell script one exits the hook if the db is already present without setting credentials <hazmat> iotw. the principia mysql formula should work fine here <kim0> ok I'll destroy everything and start again <niemeyer> kim0: Thanks for brining this up, we'll have to fix the example somehow <niemeyer> hazmat: This is another use case for having a name for the actual relation <niemeyer> hazmat: Or, an identifier <hazmat> niemeyer, you mean a generated name? <niemeyer> hazmat: The identifier we talked about before <niemeyer> hazmat: relation-N <hazmat> right.. to prevent anon relations <hazmat> and make them addressable <niemeyer> hazmat: Yeah.. this would be the right thing to use in this case <niemeyer> rather than the service name <niemeyer> then the logic would be right <niemeyer> "If a database name with the relation name already exists, stop, otherwise create it and set the info" <niemeyer> But, one thing at a time <niemeyer> hazmat: What's your plan for the session issue? <hazmat> debatable, we've allowed service names to be reusable (as opposed to unit names), so the question is are there scenarios where a service would want to pickup a previously set up database (a/b upgrades perhaps).. fwiw. the logic was right before it was rewritten.. given that we do need relation names to make them addressable in other context that seems like a good choice <hazmat> niemeyer, setup a test environment and manipulate the firewall and zk server to reproduce <hazmat> might need to send an email to the list as well, i've seen some contradictory information in regards to this <hazmat> niemeyer, have a look at http://zookeeper-user.578899.n2.nabble.com/watcher-semantics-for-session-events-in-the-C-client-td6206081.html <niemeyer> hazmat: The right thing wouldn't be to give the same database for all relations <niemeyer> hazmat: So the original one wasn't proper either <hazmat> the watches aren't freed after the session event by the client <hazmat> niemeyer, working vs. proper <niemeyer> hazmat: ? <hazmat> niemeyer, it worked in this exact scenario <hazmat> you can add/remove relations all day with the original, and the remote service will get the correct credentials and events <niemeyer> hazmat: Ok, there are several things we might that that would work <niemeyer> hazmat: I'm trying to figure how we should actually solve this issue <hazmat> niemeyer, which issue session or relation? <niemeyer> hazmat: Both :) <hazmat> well that clarifies ;-) <niemeyer> hazmat: So.. for relation I'll file a bug <niemeyer> hazmat: Let's put it aside for now <hazmat> niemeyer, sounds good <niemeyer> hazmat: Interesting indeed <niemeyer> hazmat: (the link) <hazmat> niemeyer, so the ppa will auto update with the new package for txzk since there was a merge to trunk? <hazmat> oh.. i need to increment the version i think <niemeyer> hazmat: It will, but it will be daily <niemeyer> hazmat: No <hazmat> ah.. its got the revno on it <niemeyer> hazmat: It will use the revision from the repo <hazmat> niemeyer, can those builds be triggered by hand? <hazmat> yup.. i see it now.. okay requested a rebuild of txzk <niemeyer> hazmat: When you want to bump up due to a change you've done recently, go to the recipe page, and click on the button <hazmat> wow.. the build wait time is much better now <hazmat> 4-8m <hazmat> is much better than 12-36hrs <niemeyer> hazmat: Btw, it's good to check if your revision isn't already built <niemeyer> hazmat: It was, in this case <niemeyer> txzookeeper - 0.2.1-0ensemble38~oneiric1 <niemeyer> hazmat: That 38 is the revno, which is the tip atm <hazmat> niemeyer, ah good point <niemeyer> hazmat: Don't use "Request build", in general <niemeyer> hazmat: When there is a revno unprocessed, there will be a button at the top, right below "Build schedule" <hazmat> ic <niemeyer> hazmat: https://bugs.launchpad.net/ensemble/+bug/791370 <_mup_> Bug #791370: We need a relation identifier for hooks <Ensemble:New> < https://launchpad.net/bugs/791370 > <_mup_> Bug #791370 was filed: We need a relation identifier for hooks <Ensemble:New> < https://launchpad.net/bugs/791370 > <niemeyer> We should address this sooner rather than later <hazmat> niemeyer, more immediately the formula should probably be fixed as well <niemeyer> hazmat: It depends a bit on how long we take to get to it <niemeyer> hazmat: E.g. if Ben is available today, it might be a good brain-break task <niemeyer> I've assigned the bug to him <niemeyer> Hah, just in time for bcsaller to be unable to say no! <bcsaller> oh no <hazmat> niemeyer, ;-) here's the diff https://pastebin.canonical.com/48037/ <bcsaller> what did I walk into? <hazmat> bcsaller, a bug in the example mysql formula <niemeyer> bcsaller: https://launchpad.net/bugs/791370 <_mup_> Bug #791370: We need a relation identifier for hooks <Ensemble:Confirmed for bcsaller> < https://launchpad.net/bugs/791370 > <niemeyer> hazmat: How does that solve anything? <hazmat> bcsaller, and more generically identifiers for relations <hazmat> niemeyer, it allows the formula to work in if add/remove relation is used or the service is destroyed and recreated <niemeyer> hazmat: How so? <hazmat> oh.. i need to remove the exit 0 as well <hazmat> niemeyer, it will setup credentials for the service and set them on the relation always this way <niemeyer> hazmat: Ahhhh, yeah, if you fix it, maybe ;-) <hazmat> niemeyer, can i get a +1 on the updated trivial https://pastebin.canonical.com/48038/ <niemeyer> hazmat: So it'll use a different user per relation? <hazmat> niemeyer, yes <niemeyer> hazmat: Cool <hazmat> hmmm.. i wonder what the original did here <niemeyer> hazmat: +1 if you test it :-) <hazmat> this doesn't seem right either <hazmat> it shouldn't be resetting the identity every time on join <hazmat> it should be checking for the value on the relation <niemeyer> Yeah, indeed <hazmat> niemeyer, oddly though the original (in principia) did this.. and it worked <hazmat> for scaling mediawiki <hazmat> but i take that as some mysql aberation <niemeyer> hazmat: Apparently we didn't test non-trivial scenarios, though <niemeyer> hazmat: Well, or maybe you did with the old one <hazmat> i did with the old one.. but i still wonder about this <_mup_> Bug #791382 was filed: Session events are not being handled properly <Ensemble:Confirmed for hazmat> < https://launchpad.net/bugs/791382 > <niemeyer> Lunch time, biab. <SpamapS> Hmm.. <SpamapS> so scrolling back a bit.. <SpamapS> the principia formula right now uses a flag file to know whether or not the relationship has been broken.. <SpamapS> If its been broken, then it creates a new user <SpamapS> lp:~ensemble-composers/principia/oneiric/mysql/trunk btw <SpamapS> hazmat: re the munin server, I had it happen yet again after a full shutdown/bootstrap.. and I tracked it down to heavy network + CPU. When munin was doing its updates and I would add or remove relations, the problem would happen. <SpamapS> hazmat: so I think your hunch is correct that we probably need to think about longer timeouts for ZK <_mup_> ensemble/expose-provision-service-hierarchy r261 committed by amy08@example.org <_mup_> Tests verifying all service units are 'checked' upon destroying a service or unexposing it <_mup_> ensemble/expose-provision-service-hierarchy r262 committed by amy08@example.org <_mup_> Doc strings <_mup_> ensemble/expose-provision-service-hierarchy r263 committed by amy08@example.org <_mup_> Removed debug logging output <niemeyer> Yo <SpamapS> hrm.. augeas lenses have murky copyrights * SpamapS is starting to develop copyright policies for formulas <kim0> yuck <SpamapS> Yeah <niemeyer> SpamapS: What, seriously? <niemeyer> SpamapS: What's the copyright like? <SpamapS> I'm surprised that augeas was allowed into Debian w/ such an inaccurate copyright file <SpamapS> the copyright file claims that it is Copyright 2007, 2008 Red Hat Inc. <SpamapS> but that copyright is only in the man pages and tests <SpamapS> most of it is clearly Copyright 2007-2010 David Lutterkort <SpamapS> And the lenses have almost no copyrights specified, and there is no blanket copyright specification <SpamapS> the *license* is very clearly LGPL-2.1 <SpamapS> but most people seem to get Copyright wrong. <kim0> Does the released column on https://ensemble.ubuntu.com/kanban/dublin.html refer to released in past week ? <hazmat> SpamapS, it doesn't need to check a flag file, the relation is different if its broken, just checking for the value should do the trick <hazmat> kim0, roughly.. everything since the last milestone, its cumulative. the milestone started roughly right after uds <SpamapS> hazmat: well my point was simply that the database name should remain the same, as we want to preserve the data most of the time. <hazmat> SpamapS, ah.. indeed * kim0 rings a little shiny bell <SpamapS> I actually think even better would simply be to check for the existence of the user.. since I'm also revoking the user perms on broken <kim0> Let's have our irc meeting in 1:15 hours <kim0> Things to talk about <kim0> - Creating the principia distribution and principia-tools project <kim0> - The Ensemble ppa being up <kim0> - Plus any recent development done <SpamapS> kim0: I'm going to miss it unfortunately.. have to take child to doctor. But next time! <kim0> oh it would have been nice to have you on this one .. <kim0> no problem though <SpamapS> Yes.. definitely. :( <SpamapS> just found out about it about 15 min ago <kim0> SpamapS: all the best to our little sick friend <SpamapS> He's fine.. its just not possible to convince his mother of that without a PhD. ;) <kim0> HAHA <hazmat> SpamapS, :-) <kim0> it's amazing women are women everywhere I suppose <hazmat> perhaps more specifically.. the nature of a mom to be a mom <hazmat> i wish it were true <kim0> yeah, they're hard wired to care for children more than others can imagine .. <niemeyer> SpamapS: Ah, that should be fine then <niemeyer> SpamapS: We care mostly about license details, rather than copyright <niemeyer> SpamapS: For using Augeas, that is <niemeyer> SpamapS: +1 on having that cleared up for formulas <niemeyer> bcsaller: Have an interview now, will push your review to completion after that <niemeyer> sidnei_! <sidnei_> yo <sidnei_> so anyone wrote an ensemble formula for plone yet? :) <hazmat> sidnei_, not yet ;-) <hazmat> sidnei_, i was thinking it would be a zodb server formula, plone app server formula, varnish formula <sidnei> hazmat, yeah, that's what i thought too <sidnei> just wondering if i can get off the business of building the installer for windows and provide ensemble formulas instead :) <niemeyer> That'd be *awesome* <hazmat> the varnish is a little wierd, in that its probably passing VCL directly via the service relation <hazmat> for more advanced config.. and namespacing across multiple domains and different apps might be a little strange with varnish and service relations.. as its config is effectively code <hazmat> but simple stuff should just work ootb <hazmat> sidnei, did you see enfold's new plone hosting system? <sidnei> hazmat, yup, got early access and all :) <hazmat> its insanely fast to setup a new plone with their stuff, i still haven't figure out how they did that <hazmat> its way faster than the process start time <sidnei> hazmat, i don't think they're setting up a new process, but i could be wrong. possibly shared instance but separate zodb? <hazmat> sidnei, that makes sense, although there's some cache thrashing/mem ballooning possible with something like that <sidnei> hazmat, indeed. <sidnei> could ask alan but he's not online right now <hazmat> sidnei, he told me secret sauce last i asked ;-) <sidnei> haha * hazmat switches into concurrent mode <_mup_> ensemble/expose-hook-commands r240 committed by amy08@example.org <_mup_> Hook skeleton for open-port, close-port commands <kim0> koolhead17: hey <kim0> there ? <koolhead17> kim0: hello <kim0> hey :) <kim0> koolhead17: can u join the online chat in 2 mins <kim0> and tell us what you've been up to <koolhead17> yes sure. am very much here. <kim0> cool <kim0> niemeyer: hazmat sidnei jimbaker bcsaller ready for some irc talk <hazmat> niemeyer, that problem that SpamapS had seems pretty easy to reproduce.. start client with watch against local zk, restart zk, boom. <kim0> let's hit #ubuntu-cloud <niemeyer> hazmat: Sweet! <hazmat> kim0, rock on <niemeyer> hazmat: Well, "sweet!" :) <koolhead17> cool <niemeyer> COMMUNITY WEEKLY MEETING ROLLING ON #ubuntu-cloud <niemeyer> ^^^ <niemeyer> :-)) <niemeyer> Ok, standup? <niemeyer> Let's try to do a "real" one.. I have to stop talking and review some code :) <koolhead17> haha <jimbaker> niemeyer, sounds good <niemeyer> hazmat, bcsaller: standup? <koolhead17> obino: jimbaker hello guys <bcsaller> niemeyer: mumble <hazmat> niemeyer, we're all hanging in mumble <jimbaker> koolhead17, hi <_mup_> Bug #791501 was filed: Ensemble images should use the ppa <Ensemble:New> < https://launchpad.net/bugs/791501 > <hazmat> niemeyer, on the call you where saying we shouldn't divert all session events to a separate channel if the session is still alive? <niemeyer> hazmat: The opposite <hazmat> okay.. cool, that's what i thought <niemeyer> hazmat: We shouldn't divert the event if the session is really dead <hazmat> or.. i should say.. yeah.. that makes more sense <niemeyer> hazmat: Since we have to crash whoever is waiting on the watch <hazmat> niemeyer, yeah.. in the case of disconnect we can still divert if the handler is a kill switch <niemeyer> hazmat: Btw, it's good to double check what I said in the meeting re. the behavior of sessions. That link you posted has someone saying the opposite, but I don't recall this being the case. <hazmat> as we want to crash fast in that case, not go through every watcher callback failing <hazmat> niemeyer, yeah.. i'm running some tests now <niemeyer> hazmat: I recall a specific feature announcement saying that "now reestablishment of connections will preserve watches" <hazmat> if connecting to a different server? <niemeyer> hazmat: But this guy's email is from March.. so it's strange <niemeyer> hazmat: Yeah, given the semantics of zk, switching server is really not a big deal <niemeyer> hazmat: <niemeyer> "If the client connects to a different ZooKeeper server, it will send the session id as a part of the connection handshake. <niemeyer> " <niemeyer> http://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html <hazmat> i wonder how vmware with activestate stealing the thunder of cloudfoundry <hazmat> ^feels <niemeyer> "Another parameter to the ZooKeeper session establishment call is the default watcher. Watchers are notified when any state change occurs in the client. For example if the client loses connectivity to the server the client will be notified, or if the client's session expires, etc..." <niemeyer> hazmat: The distinction between precisely those two examples is what we were talking about <hazmat> niemeyer, from the same page "If you are watching for a znode to come into existance, you will miss the event if the znode is created and deleted while you are disconnected." <hazmat> i guess in that particular case its moot <niemeyer> hazmat: Yeah, sure, but we don't care .. righ <niemeyer> t <hazmat> hmm <niemeyer> hazmat: re. cloudfoundry, I don't see it as stealing. I think vmware is probably delighted. <hazmat> its a separate private cloud foundry installation hawking closed source features <niemeyer> hazmat: Yeah, but it's still announced as Cloud Foundry.. much better for them than anything else ActiveState pushed. <bcsaller> niemeyer: I think we are missing a high level architecture doc. I got asked for something and realized we don't really have anything we can point people at that want to eval the overall system design. Unless you know of something I'm not seeing? <hazmat> niemeyer, another useful doc http://outerthought.org/blog/435-ot.html <niemeyer> bcsaller: Indeed <niemeyer> bcsaller: I started such a doc in the old specifications section in the wiki <niemeyer> bcsaller: But it's poor and unfinished <niemeyer> bcsaller: Funny enough, I think the best we'd have today is our glossary <bcsaller> maybe I'll take a stab at it a little later today <niemeyer> bcsaller: Sweet! <niemeyer> bcsaller: https://wiki.canonical.com/Ensemble/Specifications/0010 <bcsaller> thanks <niemeyer> bcsaller: Not sure there's anything useful there <niemeyer> bcsaller: It hasn't been reviewed this century <bcsaller> understood :) <_mup_> ensemble/expose-hook-commands r241 committed by amy08@example.org <_mup_> Completed wiring up hook commands for open-port, close-port <hazmat> looking at some of the callback oriented test code of txzookeeper is like reassembling a puzzle. <jimbaker> hazmat, indeed, inlineCallbacks do make for a better experience. in talking to guido van rossum at the amazon python day event, he clarified his position on async code - it was the callback oriented code that made his head hurt, not async code per se <bcsaller> niemeyer: thanks for the review <niemeyer> bcsaller: No problem, excited to see this close to completion #ubuntu-ensemble 2011-06-02 <niemeyer> Night all <_mup_> txzookeeper/session-event-handling r40 committed by amy08@example.org <_mup_> allow connection using existing session, test session expiration, additional symbol name translation for exceptions. <_mup_> txzookeeper/session-event-handling r41 committed by amy08@example.org <_mup_> pep8isms <kim0> do I really need to type "yes" to the ssh authenticity question <niemeyer> kim0: How do you mean? <kim0> ensemble status <kim0> I get the ssh yes/no prompt <kim0> same for debug-hooks <kim0> niemeyer: that is normal right ? <niemeyer> kim0: You mean the prompt asking you if the fingerprint for the server is valid? <kim0> yes <niemeyer> kim0: Yeah, that's usual when connecting to a new server <kim0> Would be great if Ensemble would get the machine log .. and verify it for me * kim0 grabs his wish list bag <kim0> Also if ensemble had a presistent connection to bootstrap node :) and perhaps run locally under a screen session with "watch status" and debug-log ..etc all running <kim0> niemeyer: I think I am see'ing strange behaviour which I hope someoen can help me with since I really want to write this "write a formula doc". I just launched a mysql SU, and a drupal SU (based on an almost empty new formula) <kim0> fired debug-hooks drupal/0 .. works <kim0> add-relation drupal mysql <kim0> I am not getting any new windows in the debug-hooks screen <niemeyer> Hmm <niemeyer> kim0: Thinking <kim0> sure <kim0> I think I saw this yesterday <kim0> when I closed the debug-hooks screen .. <kim0> hooks suddenly started firing <kim0> it's like it was stuck <niemeyer> kim0: That's normal <kim0> but I always blame myself :) <niemeyer> kim0: Hooks are serially executed <kim0> well there were not opening new windows in screen session <kim0> they should right ? <niemeyer> kim0: You won't get another hook window until you stop the existing one <kim0> there was no existing one .. I was waiting for it <kim0> just like now .. there's only window 0 in screen <niemeyer> kim0: and what's 0? <hazmat> kim0, it was disabled (the ssh fingerprint confirm prompt) <kim0> niemeyer: just a shell <hazmat> but it leaves things open to man in the middle <hazmat> we should pull it down though automatically <niemeyer> hazmat: Do you have any ideas of what might be going on for kim0? <kim0> hazmat: yeah .. check my wish list :) we could ec2-get-console-output and verify it :) <niemeyer> kim0: WE can do better than that <niemeyer> kim0: We should inject the host key <niemeyer> kim0: That's in our wishlist already :) <hazmat> niemeyer, man in the middle was the primary reason fingerprint checking was renabled yes? * hazmat reads through log <niemeyer> hazmat: That's right <kim0> niemeyer: cool ! <kim0> niemeyer: cloud-init can inject host key already indeed .. that's even better <hazmat> indeed, we should probably make use of that, but we need to store in zk for multi-client access <hazmat> kim0, okay.. so you've got a debug hook session on drupal or mysql? <hazmat> when doing the add relation <kim0> drupal <kim0> hazmat: debug-hooks drupal/0 <kim0> hazmat: add-relation drupal mysql <kim0> that's it .. no new window in screen <hazmat> kim0, okay.. so you do debugs for install & start? <hazmat> or are you debugging after start? <kim0> the sequence was <kim0> deploy mysql <kim0> deploy drupal <kim0> debug-hooks drupal/0 <kim0> debug-log <kim0> add-relation mysql drupal <hazmat> kim0, could you paste your debug-log <kim0> I got a "install" or "start" hook here can't remember .. which I closed <kim0> I expected to get the db-relation-changed one after it .. but didn't <kim0> sure <hazmat> kim0, by closing are you exiting the shell or just closing the window? <hazmat> hmm <hazmat> i don't think i've tested closing the window instead of exiting the shell <kim0> hazmat: ctrl + d <kim0> exit shell <kim0> hazmat: log http://paste.ubuntu.com/616692/ <hazmat> hmm that should be fine <niemeyer> hazmat: Should be equivalent <kim0> hazmat: status â http://paste.ubuntu.com/616694/ <hazmat> niemeyer, yeah.. but on the close window case, there is still a callback to screen to close the window after the process exit <hazmat> but ctrl +d vs. exit is equiv <niemeyer> hazmat: I don't understand that distinction <hazmat> kim0, odd it seems like the unit hasn't picked up the relation <kim0> :s <kim0> can u connect to the env ? <niemeyer> hazmat: The callback will execute after the shell process exits, right? Either option would kill it <niemeyer> hazmat: IOW, closing the window also terminates the shell <niemeyer> hazmat: This would happen if the hook wasn't executed <hazmat> niemeyer, right, but we have another process checking on the shell and then instructing screen to kill the window, which is probably just a noop at that point <hazmat> niemeyer, its unrelated to what kim0 is seeing <niemeyer> hazmat: I mean, the relation not showing up <niemeyer> kim0: Can you please paste ps auxw for that machine? <niemeyer> kim0: The drupal one <kim0> niemeyer: from the debug-hooks screen is ok right ? <niemeyer> hazmat: Hmm.. unless we're running the shell script with -e, and screen exits with 1 because the window wasn't there? * niemeyer doing guess work <niemeyer> kim0: Yeah <kim0> http://paste.ubuntu.com/616696/ <niemeyer> hazmat: "install".. there's an old hook running still <kim0> I hope I didn't do something stupid at the end :) <niemeyer> kim0: I suspect your window 0 has the install hook running <niemeyer> kim0: Can you please paste "env" from that window <kim0> http://paste.ubuntu.com/616698/ <hazmat> niemeyer, window 0 is never used for hooks its .. its always a shell <kim0> window 0 is always there <kim0> yeah <niemeyer> hazmat: It's trivial to shift windows around <hazmat> niemeyer, but the names are distinct on the windows <kim0> http://paste.ubuntu.com/616699/ is the install hook itself <niemeyer> Ok, but that's not the case either way <niemeyer> Still, we have a hook running <niemeyer> hazmat: Ok <hazmat> the debug stuff names the windows by hook , except window 0 which is named 'shell' afaicr * kim0 nods <niemeyer> kim0: What's in /tmp/tmpLjxVDG-install <kim0> niemeyer: http://paste.ubuntu.com/616700/ <kim0> scary script <hazmat> so it seems somehow the debug window was ended but the underlying debug process is still alive. <niemeyer> hazmat: Yeah, it's still in the sleep loop <niemeyer> hazmat: Which confirms your initial theory <kim0> I probably closed the window too fast, if you think it needs time to do anything <hazmat> it might be a different signal gets sent besides HUP that needs to be caught here <hazmat> kim0, it shouldn't matter <hazmat> we should never rely on user timing <kim0> yeah I know <niemeyer> hazmat: TERM, KILL <niemeyer> hazmat: Wait.. the HUP is catching the outside signal <niemeyer> hazmat: That's not the problem.. that script is still running <hazmat> yeah.. its not in the screen process <niemeyer> kim0: One more: /proc/1585/environ <kim0> http://paste.ubuntu.com/616708/ <kim0> niemeyer: not sure why it has no newlines <kim0> doh <niemeyer> hazmat: We should monitor it from outside instead of expecting it to do stuff before it dies <kim0> sorry .. pastebinit error <niemeyer> kim0: That's the file format indeed <kim0> niemeyer: http://paste.ubuntu.com/616709/ <kim0> this is complete <niemeyer> hazmat: e.g. writing to hook.pid when the process starts * kim0 probably just uncovered a pastebinit bug <hazmat> niemeyer, yeah.. and then just doing something like kill -0 `cat hook.pid` for the sleep condition <niemeyer> hazmat: RIght * hazmat files a bug <niemeyer> hazmat: Another handy issue for a brain breaker.. will paste that conversation in a bug. <niemeyer> hazmat: Oh, ok :) <niemeyer> hazmat: Please paste the log for context <niemeyer> hazmat: Thanks <niemeyer> kim0: Alright.. we know what's wrong <kim0> great :) <niemeyer> kim0: For fixing your problem right now, <niemeyer> kim0: kill 1585 <kim0> got it <kim0> thanks <niemeyer> kim0: np <kim0> wonder why no one else is hitting this <niemeyer> kim0: Thanks a lot for your help uncovering the bug <niemeyer> kim0: It's the way the debug-hook window was closed <kim0> ah so you close it clike ctrl-a c <kim0> ok <kim0> not c .. whatever closes windows :) <kim0> ew <kim0> ok probably hitting a new one <kim0> I killed the process .. got the window for db-relation-changed <kim0> relation-get inside it says â No ENSEMBLE_AGENT_SOCKET/-s option found <kim0> env dump http://paste.ubuntu.com/616713/ <kim0> hazmat: could you please as well * hazmat looks <hazmat> kim0, that's the env from window 0 ? <hazmat> or the debug window? <kim0> hazmat: no win 1 <kim0> the db-relation-changed window <kim0> db-relation-joined actually <hazmat> it doesn't look like it has the debug environment variables sourced <niemeyer> That's the shell isn't it? <niemeyer> SUDO_COMMAND=/usr/bin/byobu -xRS drupal-0-hook-debug -t shell <niemeyer> kim0: I think the paste is bogus <niemeyer> kim0: Or maybe I just misunderstand what the variables mean <kim0> I can repaste manually <kim0> anything to look for ? <niemeyer> kim0: Just thinking how to get the pid for the parent shell <kim0> ps -elf | grep $$ ? <kim0> it'd be listed in ppid field <niemeyer> kim0: echo $BASHPID <kim0> 9229 <niemeyer> kim0: echo $PPID <kim0> 935 * kim0 feels like a shell <niemeyer> Ok, cool <niemeyer> kim0: Hehe :-) <kim0> :) <niemeyer> kim0: Yeah, looks like a failure in source it indeed <niemeyer> sourcing <kim0> we don't log those steps somewhere ? <niemeyer> Can't imagine how that could happen, though, even if we killed the process <niemeyer> kim0: Nope, this is the bootstrapping of debugging itself.. we might indeed have to log it in the future <niemeyer> kim0: Can you please paste the new process list so we can reach the new hook <kim0> http://paste.ubuntu.com/616722/ <kim0> guess no one uses debug-hooks really :) * kim0 afk for 5 mins <niemeyer> kim0: We do, but we don't generally kill processes in the middle <hazmat> kim0, just me ;-) <hazmat> need to run a quick errand, back in a bit <niemeyer> kim0: Let me know when you're back.. we can follow on a bit if you're interested <kim0> niemeyer: back <niemeyer> kim0: Ok, let's see /tmp/tmpR1UhMY-db-relation-joined then <kim0> niemeyer: http://paste.ubuntu.com/616737/ <niemeyer> kim0: Ok, please figure ENSEMBLE_DEBUG from /proc/9187/environ, and list $ENSEMBLE_DEBUG/env.sh <kim0> niemeyer: can't see ENSEMBLE_DEBUG .. paste http://paste.ubuntu.com/616742/ <niemeyer> kim0: Hmm.. I guess it wasn't exported * niemeyer htinks <niemeyer> thinks <niemeyer> kim0: Ok, let's try to find by force: cd /tmp && find -name env.sh <niemeyer> kim0: Will probably see more than one <kim0> niemeyer: 3 of em <niemeyer> kim0: Ok, let's fine the one with db-relation-joined <kim0> http://paste.ubuntu.com/616746/ <kim0> http://paste.ubuntu.com/616747/ <kim0> http://paste.ubuntu.com/616748/ <kim0> niemeyer: I tried grep'ing .. doesn't have joined in them <niemeyer> kim0: Well, that's likely the issue then.. let me check <niemeyer> kim0: That's weird.. <niemeyer> kim0: All of them have ENSEMBLE_AGENT_SOCKET <kim0> maybe none of them is sourced <niemeyer> kim0: This is the right one for db-relation-joined: http://paste.ubuntu.com/616747/ <niemeyer> kim0: Can you please paste the hook.sh file living in the same directory? <niemeyer> kim0: Well, that's the thing <niemeyer> kim0: There's no easy way for bash to be executed without this being sourced <kim0> niemeyer: http://paste.ubuntu.com/616754/ <niemeyer> kim0: As you can see.. <niemeyer> kim0: That prior paste is from /tmp/tmp.Sj2ilkd53B/env.sh, right? <kim0> double checking <kim0> should be yes <niemeyer> kim0: Ok, so there's really no way for bash to be executed without it being sourced, which is awkward.. <niemeyer> kim0: You got a bash, without the env variables, but the only way for that bash to have come up, was through the sourcing line <niemeyer> Hmmm <kim0> thinking as well <kim0> niemeyer: parent process for the Window1 shell, is byobu, not the hook.sh ? <niemeyer> kim0: Yeah, that's strange <kim0> hook.sh should still be running if it fired us right <niemeyer> kim0: Indeed <niemeyer> kim0: This would also justify the previous issue as well, interestingly <niemeyer> kim0: Hmm <niemeyer> kim0: Let me do a local test, hold on <kim0> niemeyer: pstree -p .. if that's helpful http://paste.ubuntu.com/616757/ <hazmat> nice.. half way to my walking desk finished <kim0> hazmat: walking desk ? <hazmat> kim0, treadmill with keyboard tray and monitor stand <kim0> oh that's new to me ... sounds cool indeed :) <hazmat> kim0, http://opinionator.blogs.nytimes.com/2010/02/23/stand-up-while-you-read-this/ ... http://www.nytimes.com/2008/09/18/health/nutrition/18fitness.html <hazmat> kim0, lots of a good evidence for the benefits vs sitting in a chair all day <kim0> yeah that's intuitive <hazmat> the only problem is that the treadmill weighs 250 pounds.. just carried it up the stairs.. so most of the way done on the setup * hazmat catches up the irc log to get up to speed on debug-hooks <kim0> hazmat: congrats :) send pics to warthogs :) <kim0> niemeyer: I'm going for a late lunch .. I have inserted your ssh key into ec2-67-202-22-46.compute-1.amazonaws.com (drupal/0) should you want to login to it <niemeyer> kim0: Cheers <niemeyer> kim0: Will check it out <kim0> cool <niemeyer> hazmat: I suspect both issues likely boil down to the way the shell is being executed <niemeyer> hazmat: It's doing a two-step execution, and it's not entirely clear why <hazmat> niemeyer, what's strange is that it works sometimes <hazmat> writing up a reply to tom for his questions on list <niemeyer> hazmat: It first creates a window, which spawns an outside shell by screen itself <niemeyer> hazmat: then overwrites a shell onto it <niemeyer> hazmat: I suspect we may be hitting some race within screen itself <niemeyer> hazmat: Is there a reason why you coded it like that, or is it safe to change? <hazmat> niemeyer, its safe to change, i thought that creation was per your suggestion <niemeyer> hazmat: It's unrelated to my suggestion <hazmat> the openstack nova screen setup does a similiar setup <niemeyer> hazmat: It's executing two shells for no reason <niemeyer> hazmat: Rather than only hook.sh <hazmat> right <hazmat> so instead of creating the window it should just exec in the named window? <niemeyer> hazmat: The "screen" command of screen takes an executable as an argument <niemeyer> hazmat: -X screen -t .. hook.sh <niemeyer> hazmat: The shell I'm seeing in kim0's is the shell from the first screen command, not the one from the exec <hazmat> niemeyer, that sounds good to me <kim0> hope that bug got caught <niemeyer> kim0: Sounds like so.. <niemeyer> kim0: Will try something this afternoon <kim0> awesome <niemeyer> kim0: Thanks for all your help <kim0> All thanks to you :) * hazmat lunches * niemeyer too <kim0> hmm <kim0> state: install_error <kim0> if service is having install_error .. any facility to figure out what went wrong <kim0> ok I could figure it out <niemeyer> kim0: I was kind of expecting that.. <niemeyer> kim0: Is that the service we were debugging? <kim0> niemeyer: I shutdown the env and started a fresh <niemeyer> kim0: Oh, ok <niemeyer> kim0: ensemble log, ensemble debug-hook, etc <kim0> niemeyer: does the log not provide hooks stdout any more ? <kim0> is it supressed by default <niemeyer> kim0: It does, but you have to turn it on earlier <niemeyer> We should really have a feature where it logs by default <niemeyer> and rotates them out after a while <niemeyer> kim0: Otherwise, the best bet is logging in the machine and checking logs <niemeyer> kim0: You should be able to retry, though <niemeyer> kim0: Run debug-hook <niemeyer> kim0: and then run ensemble resolved with the --retry argument <kim0> niemeyer: hmm .. service unit is stuck somehow <kim0> here is status http://paste.ubuntu.com/616891/ <kim0> I hope you don't mind all the questions <niemeyer> kim0: amy08@example.org.. I'm actually going to fix some of the issues you found today <kim0> yeah .. the basic workflow should be smoother .. <kim0> so, I had an error in a hook, now I have no idea how to nudge things and get them back <niemeyer> kim0: resolved, as I mentioned above <kim0> bin/ensemble resolved --retry drupal/1 <kim0> tried this <niemeyer> kim0: Ok, what happened next? <kim0> debug-log only shows mysql related messages <kim0> and status is the same <niemeyer> kim0: Have you actually fixed the original reason why your hook failed? <kim0> niemeyer: yes I did <niemeyer> kim0: Why was it failing before? <kim0> niemeyer: ssh'ed into the machine .. and ran it <kim0> niemeyer: some cd to a non existent directoyy <kim0> niemeyer: I ran the script inside the instance .. it is fine now <niemeyer> kim0: Is it returning successfully now? (exit status 0) <kim0> checking again <kim0> can't really check again accurately <kim0> ensemble-log giving errors because it's running outside the environment <kim0> apt-get saying packages already installed ..etc <kim0> but yeah it seems correct <niemeyer> kim0: So how did you run it befor? <niemeyer> e <kim0> niemeyer: in a debug-hooks session <kim0> /var/lib/ensemble/units/drupal-1/formula/hooks/install <niemeyer> kim0: If apt-get install is failing, it won't work as a hook either <kim0> it's just saying .. the package is already installed <kim0> the script is fine trust me :) <niemeyer> kim0: :) <niemeyer> kim0: If you have already executed it by hand, you can just say "resolved" <niemeyer> kim0: Without --retry <niemeyer> kim0: But I suspect that fuzzing may have triggered something else.. that "state: null" isn't really great <kim0> yeah <niemeyer> kim0: Try the resolved trick <kim0> did that <niemeyer> kim0: This just states to Ensemble "I have resolved the problem" <kim0> still null <niemeyer> kim0: Ok, try redeploying the fixed formula then <niemeyer> kim0: We'll have to investigate a bit that scenario <niemeyer> kim0: (--retry with a broken script, etc) <kim0> how do I redeploy <niemeyer> kim0: But first, I'll fix the debug-hook stuff we debugged this morning <kim0> ok np .. <niemeyer> kim0: Same thing you did earlier? <kim0> I'll pick this up later <kim0> fresh environment .. ok <kim0> In the tutorial .. I'm assuming the formula is going to have errors <niemeyer> kim0: Nope <niemeyer> kim0: Just remove the unit <niemeyer> kim0: and add it again <kim0> ok <niemeyer> kim0: Yeah, that's a good thing <niemeyer> kim0: You can also upgrade the formula in general <kim0> says, error state cannot be upgraded <niemeyer> kim0: This is what we're preparing Ensemble to be able to do <kim0> or so <niemeyer> kim0: Error? Change, upgrade.. <kim0> says like, formula is in error state .. so it cannot be upgraded <kim0> I lost the exact message though <niemeyer> kim0: Yeah, you have to resolve it first.. <kim0> Ok .. I'll need to try this again (recovering from a formula with errors ) <kim0> and will discuss again with you <niemeyer> kim0: Because otherwise we can't assume a known state <niemeyer> kim0: Imagine an install hook failed in the middle <kim0> that's what it was :D <kim0> so I just need to know the recommended recovery steps <niemeyer> kim0: Right.. simply upgrading won't really yield a working system necessarily <niemeyer> kim0: Because half of it executed <kim0> how to know what went wrong .. release a fix .. start recovering <niemeyer> kim0: What we want is this: <niemeyer> kim0: install failed: check logs, fix it or retry, upgrade if wanted <niemeyer> kim0: With debug-hook if desired, to understand what's going on <kim0> then use "resolved" right ? <niemeyer> kim0: Right, after the "fix it or retry" <niemeyer> kim0: Or during it actually.. retry is done with resolved <kim0> is there a way to upload the fixed new hook ? <niemeyer> kim0: upgrade-formula <kim0> which refuses to work in error state ? <niemeyer> kim0: Yes, which is the right thing to do <kim0> ok .. so I'd still need to upload my fixed hook <niemeyer> kim0: The error state must be acknowledged by the administrator <niemeyer> kim0: and if an install hook blows up in the middle, upgrading a new install hook with the error fixed won't necessarily make it work <niemeyer> kim0: mkdir foo, run twice, breaks <kim0> so our recommended approach is kill the instance, and start a fresh machine ? <niemeyer> <niemeyer> kim0: What we want is this: <niemeyer> <niemeyer> kim0: install failed: check logs, fix it or retry, upgrade if wanted <niemeyer> <niemeyer> kim0: With debug-hook if desired, to understand what's going on <kim0> the "fixing and trying" cycle is what I'm trying to grasp <niemeyer> kim0: Fixing it means fixing the actual problem within the formula.. <kim0> and what about the trying <niemeyer> kim0: Sorry, within the service unit <kim0> on a new instance ? <niemeyer> kim0: If there's nothing to do.. you just run "resolved" <kim0> ah <kim0> so I fix the problem manually .. then run resolved <niemeyer> kim0: Yes, that's one way to do it <niemeyer> kim0: The other way to do it is to code an idempotent hook <niemeyer> kim0: This enables you to run resolved and upgrade a new formula <kim0> and use, resolved --retry <kim0> right ? <niemeyer> kim0: If that's what you want to do <niemeyer> kim0: The problem is really quite simple <niemeyer> kim0: When a hook fails, Ensemble will stop running hooks until the admin acknowledges it <niemeyer> kim0: If you run ensemble resolved, it forgets about the old hook, and continues execution <niemeyer> kim0: If you want to run the old hook again before continuing, you run resolved --retry <niemeyer> kim0: and that's it <niemeyer> kim0: Whether you upgrade the formula, change the hook in place to try things out, run debug-hook, run ensemble log, etc, is really up to you <niemeyer> kim0: We're coding tools to give you everything you need to understand how things are behaving <niemeyer> kim0: and fixing them <kim0> I guess the workflow I had in mind is .. install hook blows up .. I ssh into machine .. figure out why it blew up .. then I fix the hook *locally* .. then somehow ensemble would upload the new version and run that <kim0> but the workflow you explained is perfectly fine <kim0> thanks <niemeyer> kim0: Well, maybe there are further options we can develop around this <kim0> Yeah, recovering from broken formulas should be smooth .. since people are going to make all sorts of mistakes :D <kim0> niemeyer: thanks for all the explanation and patience :) <niemeyer> kim0: No worries <niemeyer> kim0: Good to talk about that stuff.. it's important to learn how other people feel about the system too * kim0 nods * SpamapS just discovered resolved yesterday btw <SpamapS> would have saved me quite a few remove-relation/add-relation cycles ;) <niemeyer> SpamapS: Sorry about that :-) <SpamapS> Yes shame on you guys for making the thing work welle nough to survive hundreds and hundreds of remove/adds. <_mup_> ensemble/close-zk-port r240 committed by amy08@example.org <_mup_> Do not open zk port on AWS firewall. <niemeyer> SpamapS: Yeah, good thing we are in a polishing cycle.. :) <_mup_> Bug #791973 was filed: Ensemble shouldn't open the EC2 firewall for zk access <Ensemble:Confirmed> < https://launchpad.net/bugs/791973 > <niemeyer> hazmat: sudo -u ubuntu screen -dmS $SESSION_NAME <niemeyer> hazmat: -u ubuntu? Shouldn't this be root? <niemeyer> Also, this command seems to create a new session, irrespective of whether there's an existing one with the same name <hazmat> its connecting to the ubuntu user's screen session from the login shell <niemeyer> hazmat: Yes, but isn't the ubuntu user connecting to root's screen using sudo? <hazmat> niemeyer, no its not that requires a setuid binary screen program <niemeyer> hazmat: Using sudo? <hazmat> niemeyer, that would be fine <hazmat> i thought you where referring to multi-user screen <niemeyer> hazmat: ssh -t ubuntu@%s sudo byobu -xRS %s-hook-debug -t shell <hazmat> sounds good <niemeyer> hazmat: The debug-hook command connects a session from root <niemeyer> hazmat: Which means doing -u ubuntu would put the session in another place <niemeyer> Ok, I'll fix that as well <niemeyer> So there's apparently no way to start a screen session unless it doesn't yet exist.. :( <niemeyer> Ok, I'll give tmux a try.. <niemeyer> smoser! <niemeyer> smoser: Was just reminding of you while hacking a shell script <niemeyer> smoser: exec &> foo.. that's an awesome trick I learned with you recently :) <smoser> that is bash only <smoser> exec > foo 2>&1 <smoser> would be the posix shell equivalent <niemeyer> smoser: Nice <niemeyer> smoser: Will use that <niemeyer> hazmat: If a sigspec is EXIT
{'channel': 'ubuntu-ensemble', 'month': 6}