text: #juju-dev 2012-10-29
  <rog> niemeyer: mind works find, i *think*
  <rog> s/mind/mine/
  <rog> s/find/fine/ :-)
  <Aram> rog: any idea how to rewrite code that uses /srv in Plan 9 to run under plan9port?
  <Aram> I'm trying to port history(1).
  <Aram> davecheney: http://paste.ubuntu.com/1315554/
  <davecheney> Aram: ty
  <Aram> cheers
  <niemeyer> rogpeppe: https://codereview.appspot.com/6720049/
  <davecheney> $ ls -1 /tmp | wc -l
  <davecheney> 20176
  <davecheney> protip - don't leave tests running in a loop
  <davecheney> http://paste.ubuntu.com/1315724/ << 24 hour stress test
  <davecheney> no GOMAXPROCS tomfoolery
  <davecheney> just while true ; do go test ./... ; done
  #juju-dev 2012-10-30
  <SpamapS> Hey can somebody from juju-core please come to this session at 11:00:  http://summit.ubuntu.com/uds-r/meeting/21326/servercloud-r-juju-ci/
  <rogpeppe> SpamapS: will do
  <SpamapS> rogpeppe: ty!
  <SpamapS> anybody from go-aware team please join us in B3-M1
  <SpamapS> davecheney: around?
  <SpamapS> rogpeppe: ?
  <SpamapS> niemeyer: ?
  <davecheney> SpamapS: still looking for go team ?
  <SpamapS> davecheney: got 'em, but thanks
  <davecheney> npo
  <davecheney> https://codereview.appspot.com/6818053
  <davecheney> ^ quite trivial
  <fss> niemeyer: ping :-)
  <niemeyer> fss: Hey
  <fss> how is that CL going? (https://codereview.appspot.com/6631063/)
  <fss> niemeyer: I have another doubt, but I don't know if it's apropriate to ask it here, it's about juju and vpc... (does it work?)
  <niemeyer> fss: It doesn't work yet, but it must
  <fss> niemeyer: is there a ticket in lp?
  <niemeyer> fss: Not that I can recall.. please feel free to file one if you cannot fine it
  <niemeyer> find it
  <fss> niemeyer: right, I will do this
  <fss> that
  <niemeyer> Aram: https://codereview.appspot.com/6775076
  <niemeyer> fss: Thanks!
  <niemeyer> and it's lunch time here
  <davecheney> https://codereview.appspot.com/6775079/
  <davecheney> ^ rogpeppe fwereade mildly trivial
  <davecheney> except for the change to environs/dummy
  <davecheney> rogpeppe: https://codereview.appspot.com/6822056/diff/1/testing/mgo.go
  <davecheney> far less contraversial
  <niemeyer> fwereade: https://codereview.appspot.com/6823050
  #juju-dev 2012-10-31
  <davecheney> https://bugs.launchpad.net/juju-core/+bug/1073458
  <davecheney> ahh s3, my friend
  <davecheney> lucky(~/src/launchpad.net/juju-core) % juju deploy mysql &
  <davecheney> [1] 21606
  <davecheney> lucky(~/src/launchpad.net/juju-core) % juju deploy memcacheerror: cannot put charm: cannot make S3 control bucket: Your previous request to create the named bucket succeeded and you already own it.
  <fwereade> niemeyer, https://codereview.appspot.com/6820065 (bbs, ciggie)
  <niemeyer> davecheney: https://codereview.appspot.com/6819056
  <davecheney> amazon: eventual inconsistency !
  <davecheney> mramm: https://bugs.launchpad.net/juju-core/+bug/1073458
  <davecheney> sorry to be an arse,
  <davecheney> could you please put the final word in this issue
  <davecheney> then it's all legal like
  <fwereade> niemeyer, https://codereview.appspot.com/6814055 reproposed with ResolveUnit in Conn
  <niemeyer> fwereade: Done
  <fwereade> niemeyer, cheers
  <niemeyer> fss: Morning
  <davecheney> niemeyer: there is no LICENCE file in py-juju
  <niemeyer> davecheney: http://bazaar.launchpad.net/~juju/juju/trunk/view/head:/COPYING
  <davecheney> niemeyer: ty
  <davecheney> all: https://codereview.appspot.com/6818063
  <davecheney> ^ hopefully uncontraversial
  <niemeyer> fss: ping
  <niemeyer> davecheney: LGTM
  <davecheney> https://code.launchpad.net/~dave-cheney/+recipe/juju-core
  <davecheney> can someone else please try to build this
  <davecheney> i've used up my quota
  <davecheney> Aram:  cat /proc/$$/status | grep Cap
  <davecheney> rogpeppe: can you go here
  <davecheney> https://code.launchpad.net/~dave-cheney/+recipe/juju-core
  <davecheney> and request a new precise build pls
  <Aram> davecheney: http://paste.ubuntu.com/1320767/
  <davecheney> rogpeppe: https://codereview.appspot.com/6811061
  <davecheney> ^ trivial
  <rogpeppe> davecheney: replied
  <davecheney> rogpeppe: ty
  <davecheney> good point about running the tests
  <davecheney> fwereade: https://codereview.appspot.com/6811061
  <davecheney> ^ looking for a 2nd LGTM
   * fwereade looks
  <fwereade> davecheney, LGTM
  <davecheney> ty
  <davecheney> open question: should the version inside juju be linked (in some way) to the version of the deb ?
  <davecheney> rogpeppe: what is the name of the juju public bucket ?
  <rogpeppe> davecheney: juju-dist, i think
  <davecheney> rogpeppe: is the name encoded in the source somewhere ?
  <rogpeppe> davecheney: in environs/cloudinit, it's used to get the mongodb binaries
  <davecheney> rogpeppe: got it
  <davecheney> ta
  <davecheney> cloudinit died: http://paste.ubuntu.com/1321037/
  <davecheney> niemeyer: http://ec2-23-22-1-96.compute-1.amazonaws.com/?p=4
  <fss> niemeyer: hi :-)
  <fss> niemeyer: you pinged earlier, but I wasn't in the computer
  <niemeyer> davecheney: http://bazaar.launchpad.net/~niemeyer/golang/stable-packaging/view/head:/rules
  <niemeyer> fss: Heya
  <davecheney> rogpeppe: http://paste.ubuntu.com/1321157/
  <davecheney> ^ why did the tests break ?!
  <davecheney> test 0
  <davecheney> tools_test.go:565: c.Assert(toolsList.Public, HasLen, len(test.expect))
  <davecheney> ... obtained []*state.Tools = []*state.Tools{(*state.Tools)(0xf8405834b0), (*state.Tools)(0xf840583550)}
  <davecheney> ... n int = 1
  <bigjools> hazmat: when you have a moment, can you please look at the provisioning agent stack trace here please? https://bugs.launchpad.net/maas/+bug/1068086
  <bigjools> bootstrap works, deployment goes bang like that
  <davecheney> jam: cmd/juju/status.go
  <davecheney> rogpeppe: http://paste.ubuntu.com/1321181/
  <fss> niemeyer: ptal https://codereview.appspot.com/6823060
  <niemeyer> davecheney: fss wants to know when we'll have the release out? :-)
  <davecheney> niemeyer: how long will he be online for :)
  <niemeyer> fss: ^ (!!)
  <davecheney> niemeyer: there is one problem with the version string we have chosen
  <davecheney> actually, it's not that bad
  <fss> niemeyer, davecheney: :)
  <davecheney> niemeyer: rogpeppe https://codereview.appspot.com/6821058
  <davecheney> ^ small fix so we can use tools in the 1.x range
  <davecheney> ty to rogpeppe for the debug
  <davecheney> niemeyer: rogpeppe ignore the change to export_test.go
  <fss> niemeyer, davecheney: that's something nice to read
  <niemeyer> davecheney: Looks strange
  <davecheney> preproposed
  <davecheney> reproposed
  <niemeyer> davecheney: Where are those names being put into the storage?
  <davecheney> dummy always calls putFakeTools from environs.Open
  <niemeyer> davecheney: Ah, super
  <davecheney> yes, super, lower case s
  <davecheney> it was never a problem when version.Current was 0.x
  <davecheney> in fact, i'll add a test case for that
  <davecheney> niemeyer: have you lost your room key ?
  <niemeyer> davecheney: I did
  <niemeyer> I'm in the room behind.. if you're leaving, can you please drop it here
  <niemeyer> ?
  <niemeyer> davecheney: ?
  <davecheney> niemeyer: yes, i have your key
  <niemeyer> davecheney: Cool.. I shouldn't be long, thanks
  <davecheney> kk
  <davecheney> all: i will be doing the 1.9.1 release at 6pm
  <davecheney> if you have any branches to land, please do so before then
  <davecheney> oh, it is always 6pm
  <davecheney> you'd better hurry :)
  <davecheney> niemeyer: rogpeppe https://codereview.appspot.com/6823061
  <fss> davecheney: which 6pm? :)
  <davecheney> fss: uhhh, good point
  <davecheney> might have to be someone elses 6pm :)
  <fss> davecheney: it's 4 pm here, is this 6pm enough? :)
  #juju-dev 2012-11-01
  <davecheney> TheMue: https://code.launchpad.net/~dave-cheney/+recipe/juju-core
  <davecheney> niemeyer: https://code.launchpad.net/~gophers/+archive/go/+build/3946897
  <davecheney> rogpeppe: niemeyer fwereade https://codereview.appspot.com/6814065
  <davecheney> trivial, please approve before checking in anything else
  <niemeyer> davecheney: +1
  <bigjools> hazmat: FYI that bug is https://bugs.launchpad.net/maas/+bug/1068086
  <bigjools> hazmat: someone on the maas-devel list said he fixed it by upgrading juju but hasn't said what versions he had
  <hazmat> bigjools, danke
  <bigjools> so when I know I'll tell you
  <davecheney> http://paste.ubuntu.com/1322955/
  <davecheney> ^ this is with 1.9.1
  <davecheney> mramm: niemeyer http://ec2-184-73-150-172.compute-1.amazonaws.com/
  <davecheney> ^ juju 1.9.1
  <davecheney> all: https://docs.google.com/a/canonical.com/document/d/161Nt-AGqH50ldIQVKR8zIZO-cm7OyD-UkrU_-N_EhE4/edit
  <davecheney> ^ juju-core 1.9.1 release notes
  <davecheney> in progress
  <davecheney> contributions welcome
  <davecheney> TheMue: https://docs.google.com/a/canonical.com/document/d/161Nt-AGqH50ldIQVKR8zIZO-cm7OyD-UkrU_-N_EhE4/edit
  <davecheney> niemeyer: https://docs.google.com/a/canonical.com/document/d/161Nt-AGqH50ldIQVKR8zIZO-cm7OyD-UkrU_-N_EhE4/edit
  <davecheney> can i have some of your time to talk about changes to environments.yaml
  <davecheney> nag: http://codereview.appspot.com/6818074/
  <davecheney> fss: ping
  <davecheney> https://codereview.appspot.com/6810065
  <davecheney> ^ trivial
  <davecheney> fwereade: are you working on adding relations to juju status ?
  <fwereade> davecheney, no, that hasn't really been on my mind
  <davecheney> fwereade: taking into account travel and recovery
  <davecheney> i wouldn't get to it til mid next week
  <davecheney> does that sound ok ?
  <fwereade> davecheney, if you're offering to do it then I am content with whatever schedule works for you :)
  <davecheney> cool, i'll assign the issue to myself for 1.9.2
  <fwereade> davecheney, lovely, tyvm
  <davecheney> np
  #juju-dev 2012-11-02
  <davecheney> all: i have some time to kill between checking out and my flight
  <davecheney> if anyone wants to do some juju hacking (read, bug triage) i'll be in the lobby bar
  <mcclurmc> davecheney: how was the party last night? sorry i had to miss that
  <davecheney> mcclurmc: blurey
  <mcclurmc> lol
  <davecheney> once the bar tab ran out
  <davecheney> i spent a few hours playing giant jenga with some eastern europeans
  <davecheney> then stumbled back to the late bus
  <mcclurmc> yeah, that looked like fun (jenga)
  #juju-dev 2013-10-28
  <mattyw> If I've just uploaded a new proposal and I want to leave a comment other than "please take a look" shall I just do it on Rietveld or is there an lbox way of doing it that puts it in launchpad as well?
  <rick_h_> mattyw: I'm not following 100%. lbox will sync the description/comment to the LP MP. LP supports the description and commit message.
  <mattyw> rick_h_, I've already pushed the change, and I want to make a comment about it, there doesn't seem to be an option for that in lbox?
  <rick_h_> mattyw: sure thing, you have to either reply at the bottom or do a "Publish ... comments"
  <rick_h_> mattyw: and it'll give you an input to comment.
  <rick_h_> mattyw: we do 'code reviewer comments' by going through as if we were a reviewer putting in-line comments and then publishing them before asking for a review that way.
  <rick_h_> mattyw: you do have to be signed in to comment? Look in the upper right and check for the login link perhaps?
  <mattyw> rick_h_, I can see the reply button, just wonderd if it sync with lp automatically or if I should be using lbox
  <rick_h_> mattyw: yes, comments in lbox (reitveld) will be sync'd over to LP. see https://code.launchpad.net/~rharding/charmworld/proof-config2/+merge/192243 for an example where the comments were all made in reitveld, but got copied over ok.
  <rick_h_> mattyw: we just use lbox to sync the code. So if you edit the MP code, you need to re-lbox it to update the diff on reitveld
  <rick_h_> and final submit is lbox submit
  <mattyw> rick_h_, ok thanks - I'm going to blame my cognitive speed on my jet lag :)
  <mattyw> rick_h_, while I have your attention, I was told you might be able to answer the following (or point me to someone who can)
  <mattyw> rick_h_, ^^ it doesn't look like the jujucharms pages for charms not in the store get updated, is this on purpose?
  <rick_h_> mattyw: no, not on purpose. You're talking about jujucharms.com or manage.jujucharms.com?
  <rick_h_> mattyw: if there's an error on a charm that a revision introduces it'd not get updated. See http://manage.jujucharms.com/tools/proof-errors if the charm in question is in there.
  <mattyw> rick_h_, it's on jujucharms.com. I mean pages like this: https://jujucharms.com/fullscreen/search/~mattyw/precise/docker-3/?text=docker. Even though I've updated the readme the latest version doesn't appear, just wondered if this was intentional
  <rick_h_> mattyw: it's failing lint http://manage.jujucharms.com/~mattyw/precise/docker I'm not sure if that's causnig it to not update.
  <mattyw> rick_h_, I'll check it out, thanks :)
  <rick_h_> mattyw: sure thing. if that doesn't help it out we can follow up on it tomorrow. /me is actual away today
  <rick_h_> mattyw: it should re-check things after a new commit around 15-30min
  <rick_h_> mattyw: I'd also suggest you redo the readme like http://paste.mitechie.com/show/1055/ as the `` is markdown syntax and it renders it funny and might cause issues (thuogh shouldn't be breaking updating it)
  <mattyw> rick_h_, thanks for the suggestions, didn't realise you were away otherwise I'd have left you in peace
  <rick_h_> mattyw: np, quiet day around here so can irc between loads of cleaning/laundry :)
  <abentley> Why does juju chown the environments/*.jenv?  I'm getting a failure because of it: http://pastebin.ubuntu.com/6320043/
  <hazmat> abentley, cause it has secrets
  <abentley> hazmat: I can understand chmodding, but why does it chown?
  <hazmat> abentley, its a fallout from local provider needing sudo to bootstrap
  <hazmat> that stills sounds fishy.. but ie. prior to the chown (on environments dir) if you bootstrapped local, you wouldn't be able to use other providers because of perm errors on creating jenv in that dir
  <abentley> hazmat: If it must chown, it would be nice to chown to gid/uid of environments.yaml.  That way, secrets are not disclosed to other users.
  <abentley> hazmat: If it succeeded at chowning to 1000, then user "ubuntu" would know the contents of user "jenkins" (107) environments.yaml.
  <hazmat> abentley, sounds reasonable re gid/uid of env.yaml
  <hazmat> wathces seem to be interacting badly with container units
  <hazmat> hmm.. upgrader gone amuck trying to downgrade and stopping agents
  <sidnei> uhm, no thumper around
  #juju-dev 2013-10-29
  <rogpeppe> i'm taking another swap day today, BTW
  <mattyw> rogpeppe, I don't remember allowing that
  <rogpeppe> mattyw: oh
  <rogpeppe> mattyw: in which case i guess i'll just have to kill you
  <smoser> hey. whats the state of the null /none/manual provider ?
  <smoser> is that in 1.16 ?
  <natefinch> smoser: the manual provider is in 1.16 but it's not really ready for primetime. There's still some bugs around it.
  <smoser> natefinch, thats ok.
  <smoser> hey....
  <smoser> http://paste.ubuntu.com/6324341/
  <smoser> that seems to imply that "Due to needing newer versions of LXC the local provider does require a newer kernel than the released version of 12.04. Therefore we install Linux 3.8 from the LTS Hardware Enablement Stack"
  <smoser> which is declared as fact at https://juju.ubuntu.com/docs/config-local.html
  <smoser> is incorrect
  <smoser> can we have that doc fixed please ?
  <smoser> or someone point me to why that statement exists  (hazmat?)
  <natefinch> smoser: I don't know the exact details, but we've definitely seen problems with 12.04.  Not only the old LXC but also the old MongoDB.  Are you sure you're not already running the newer kernel on that machine?
  <smoser> natefinch, i'm using the cloud archive
  <smoser> which is the solution designed by CTS for some supportable solution of juju on 12.04
  <smoser> ie, that is probably what doc should suggest.
  <natefinch> smoser: yeah, it sounds like the doc is just out of date.  I haven't really kept up on the exact nuances of the local provider, which is my only hesitation.  definitely we should make sure the documentation is accurate and suggests the least painful way to get it running on 12.04
  <hazmat> smoser, not sure why that verbiage appeared.
  <hazmat> smoser, specifically around manual provider it works, but you need to manually remove the $JUJU_HOME/environments/$manual_env_name.jenv to destroy/reset it
  <hazmat> and it leaves juju stuff on the machines in question..
  <hazmat> smoser lp:juju-core/docs
  <hazmat> smoser, there are performance issues with older kernels (including 3.8) and container networking that have been documented, not sure its relevant in this context though
  <smoser> hazmat, hallyn was unaware of such issues
  <smoser> so links to such documentation might be useful.
  <hazmat> smoser, i saw it referenced by the docker  community last month re this thread https://groups.google.com/forum/#!msg/docker-user/txAd5BiVapU/AfXvssMqkr4J
  <smoser> i think performance of veth is probably not a good enough reason to sugges tsomeone use a HWE kernel that is not necessary otherwise.
  <smoser> at least without citing that as the reason
  <smoser> HWE kernels have real drawbacks, forcing you to upgrade or lose security updates earlier than 12.04 LTS kernel.
  <sinzui> natefinch, do you have time to review two small branches that backport fixes to 1.16.1? https://codereview.appspot.com/18610043/ and https://codereview.appspot.com/18640043/
  <natefinch> sinzui: sure
  <natefinch> sinzui: the first one already got a LGTM from jam, so that's fine. I can lgtm the other one.
   * sinzui thinks his review emails are being eaten again
  <hatch> any word on a fix for the lxc/apt issue coming down the pipe? I'd really like to demo the GUI next week on lxc :)
  <sinzui> hatch is this the node problem?
  <hatch> sinzui: when I try and deploy the gui it fails with some apt-get stuff
  <hatch> I was told it was an lxc bug
  <hatch> happens on 12.04 and 12:10 with the most recent juju-core
  <sinzui> hatch, these are the bugs that I want to close today/tomorrow with a release https://launchpad.net/juju-core/+milestone/1.16.1
  <sinzui> hatch, there aren't any bugs about lxc reported that also speak of apt
  <hatch> ahh ok so that doesn't help unfortuately....I don't know what the true cause of it is, it could be lxc related and not juju related
  <sinzui> all apt bugs relate to firewalls in fact.
  <hatch> hmm ok I'll try and get some more information and file a formal bug so that someone can tell me one way or another if it's juju related or not
  <sinzui> I am sure it is a bug and maybe even reported. We want to tag it juju-gui so that I can drive a fix for it
  <hatch> sinzui: looks like after doing some updates after the sprint it's working again, so something must have landed to fix it :)
  <sinzui> okay hatch. If you do have bugs you care about, tag then with juju-gui so that I can arrange for fixes right away
  <hatch> will do thanks!
  <hazmat> smoser, re hwe sounds reasonable to me, i don't have any insight into why we doc'd hwe kernels for precise, we're not using any lxc functionality that's only later kernel rels afaics. the delta might be related to  lxc userspace tooling and how  a newer version of that interacts with older kernel versions, dunno.
  <sodre> hello. I am facing some issues with juju on an Openstack private cloud. Could someone explain point to some documentation on that ?
  <sodre> I am running Havana deployed on top of MAAS
  <sodre> hi Thumper, can you help me ?
  <thumper> maybe, what's your problem?
  <sodre> I am trying to run Juju on top of a Havana
  <sodre> on a private cloud.
   * thumper thinks 
  <thumper> is your problem django?
  <sodre> I've deployed Havana using Juju/MAAS. But now I am having issues deploying VMs on top of Havana using juju.
  <thumper> ISTR that there is an issue with dependencies
  <thumper> MAAS needing one version and havana needing another
  <sodre> a version of Juju ?
  <thumper> no
  <thumper> djanog
  <thumper> but when juju deploys a machine
  <thumper> it adds the cloud-tools archive
  <thumper> which then installs a version of django that havana can't use
  <thumper> this is a known bug and something we are going to try to work around
  <thumper> but not sure of the fix just yet
  <thumper> does this sound like your problem may be related to that?
  <sodre> okay. What is the side effect of that django bug? My issue right now is that Juju cannot bootstrap _on top of_ Havana.
  <sodre> Havana, seems to behave just fine by itself.
  <thumper> I'm not entirely sure what the side effect it, just that it doesn't work
  <thumper> this could be it
  <thumper> IF it is the same issue
  <thumper> what stage does it fail?
  <sodre> after I setup my environments.yaml file to point to my Havana install
  <sodre> I call juju bootstrap.
  <sodre> it uploads all the tools to Swift/RadosGW
  <sodre> then the following error comes up:
  <thumper> can you see that the machine has started? can you ssh to it?
  <sodre> 2013-10-29 19:47:26 ERROR juju supercommand.go:282 cannot start bootstrap instance: index file has no data for cloud {RegionOne http://m1basic-05.vm.draco.metal.as:5000/v2.0} not found
  <thumper> ok... that sounds like something else
  <thumper> maybe
  <sodre> I think so.
  <thumper> so... perhaps this is simple stream related
  <sodre> how do people usually load up their images into a Private OpenStack install for using with Juju?
  <thumper> we now use simple streams to find the instances
  <sodre> exactly...
  <thumper> I don't know
  <thumper> wallyworld may be able to help when he starts in a few hours
  <thumper> as he did a lot of the simplestream stuff
  <sodre> okay.
  <thumper> sodre: which TZ are you in?
  <sodre> I'm on EST
  <thumper> so already 5pm?
   * thumper looks at the date applet
  <sodre> 4:05
  <thumper> oh, no falling back just yet
   * thumper thinks
  <thumper> gah, TZ confused again
  <thumper> anyway...
  <sodre> :)
  <thumper> wallyworld is in brisbane, AU
  <sodre> okay...
  <thumper> and not likely to start for a few hours
  <thumper> or stry smoser
  <thumper> s/stry/try/
  <sodre> Let me pm him.
  <thumper> sodre: he is in this channel
  <thumper> smoser: <sodre> how do people usually load up their images into a Private OpenStack install for using with Juju?
  <thumper> smoser: any idea?
  <sodre> ahh.. got it
  <sodre> I haven't used IRC in such a long time.
  <thumper> I understand, before working at canonical I hadn't used irc at all
  <thumper> ok, not quite at all, but very briefly
  <sodre> Do most people at canonical work from home ?
  <smoser> ah. sodre yes most people work from home.
  <smoser> the suggested way to load images into glance is at https://code.launchpad.net/~smoser/simplestreams/example-sync
  <sodre> that is nice. I guess there is always development going ton.
  <sodre> Alright, let me clone that branch.
  <thumper> smoser: thanks
  <sodre> thanks thumper and smoser... I am still trying to get it to work... I'll be back soon.
  <thumper> sodre: good luck
  <sodre> smoser: are you still available ?
  <smoser> not really
  <smoser> qiuck....
  <sodre> for some reason the PUT requests are going itno /swift/simplestreams instead of swift/v1/simplestreams
  <sodre> any idea why that would be ?
  <sodre> Also, running swift from the command-line places things in the correct place.
  <smoser> unm...
  <smoser> i'm not sure.
  <smoser> that code there *is* used in production
  <smoser> so it is known working
  <sodre> okay. I am using radosgw, maybe it should default to v1
  <sodre> I'll give the code a quick read. Maybe I can figure out why it is not giving placing the v1 in PUT.
  <bigjools> wallyworld_: just for you: http://www.git-tower.com/blog/git-cheat-sheet/
  <wallyworld_> :-(
  <thumper> troll!
  <bigjools> like a boss
  <bigjools> I was using git yesterday as one of my home projects uses it
  <bigjools> hadn't been in for a while and needed to pull a particular branch to compile
  <bigjools> took me 45 minutes to work out how
  <bigjools> winning!
  <bigjools> who knew you had to pull *everything* and then "checkout"
  <bigjools> wallyworld_: I am low on coffee.  If you could stop by the local bean on the way over that'd be awesome
  <wallyworld_> it will cost you
   * bigjools bends over
  <wallyworld_> it's on the wrong side of the road so it will cost you a lot
   * thumper covers his eyes
  <thumper> there are children in this channel
   * wallyworld_ unips
   * bigjools goatses
  <thumper> WORK CHANNEL!!
  <bigjools> thumper: there sure are!
   * thumper cracks a whip
  <sodre> wallyworld_: do you work on simplestreams as well ?
  <wallyworld_> the juju side of things, yes
  <sodre> I have a question :)
  <wallyworld_> sure
  <sodre> I am trying to use sstream-mirror-glance, but the URLs it gives to the swift client lacks the /v1  part.
  <sodre> e.g. when I run it, I get the following error
  <sodre> Container PUT failed: http://m1basic-04.vm.draco.metal.as:80/swift/simplestreams 405 Method Not Allowed
  <sodre> However, if I  run ' swift post simplestreams '
  <sodre> everything goes fine.
  <sodre> I have logs from the Swift back-end as well, if you would like to see them.. +3lines.
  <wallyworld_> sodre: i have no knowledge of the lp:simplestreams project and associated tools. i just do the juju-core lib
  <wallyworld_> which is a separate codebase written in Go
  <sodre> got it
  <wallyworld_> sorry :-(
  <sodre> np
  <wallyworld_> but it does seems perhaps there is a bug there
  <sodre> yeah, in the Swift log, it shows that the sstream-mirror-glance code did a PUT /swift/simplestreams
  <sodre> while the swift CLI, did a POST followed by a PUT /swift/v1
  <sodre> at least I found out why... the code in simplestreams/openstack.py strips the version out :)
  <sodre> no I wonder why they are doing that.
  <wallyworld_> good question :-)
  <sodre> wallyworld_ : The simplestreams images are getting loaded into glance after I commented out the "strip_versions" call.
  <wallyworld_> \o/
  <sodre> now I have another question which you might be able to help.
  <sodre> I am trying to deploy juju on top of Havana.
  <sodre> Havana itself seems to be working okay, i.e. I can launch CirrOS
  <sodre> create Networks, assign floating ips, etc...
  <sodre> let me clean up my bootstrap version
  <sodre> so I have sourced my admin-openrc.sh
  <sodre> I ran juju bootstrap
  <sodre> and then at the end I get the following error.
  <sodre> ERROR juju supercommand.go:282 cannot start bootstrap instance: index file has no data for cloud {RegionOne http://m1basic-05.vm.draco.metal.as:5000/v2.0}
  <sodre> I imagine that happens because it did a get for /swift/v1/admin-juju/streams/v1/index.json
  <sodre> and it failed.
  <sodre> how do I tell Juju to find /streams/v1/index.json in a different container ?
  <wallyworld_> sodre: you set the tools-url config option if you want to get simplestreams metadata from a non-default container
  <sodre> okay...  let me try taht.
  <wallyworld_> or, ensure the index file you are using has metadata for RegionOne and your endpoint
  <wallyworld_> sodre: actually
  <wallyworld_> not tools-url
  <wallyworld_> you want image-metadata-url for images
  <wallyworld_> i think that's what it can't find
  <wallyworld_> if it complains about tools, you need tools-url
  <thumper> Time to go and hit something, more emails after lunch
   * thumper -> gym
  <sodre> okay.
  <sodre> humm... not yet...
  <sodre> I'll play with it once I get home.
  <sodre> ttyl
  <sinzui> wallyworld_, do you understand what went wrong in this azure bootstrap. I see evidence that abentley's CI testing has placed current tools in juju-dist/testing/tools, but I don't see how out environments could collide and break: http://pastebin.ubuntu.com/6326844/
   * wallyworld_ looks
  <wallyworld_> sinzui: i've not seen that error before. it looks like it is failing to load image metadata from cloud-images.canonical.com
   * sinzui ponders setting image-metadata-url: http://cloud-images.ubuntu.com/releases
  <wallyworld_> that may work, let me check the code
  <sinzui> wallyworld_, okay. I think so too. I am not testing changes yet :) This was a stable deploy for an upgrade test
  <wallyworld_> sinzui: btw, trunk has all the changes except for the --public option with sync tools (but you can create the json file for that by hand in the interim). i hope to land today in any case
  <sinzui> okay
  <sinzui> wallyworld_, I was starting the blessing/cursing of 1.16 r1977. Once the stable is updated, I was going to return to automate releases
  <wallyworld_> rightio
  <sinzui> damn. image-metadata-url did change anything
  <wallyworld_> did not?
  <sinzui> hmm, maybe the sjson is the issue
  <sinzui> 1.16 shouldn't be looking at it yet
  <sinzui> well there is evidence that the sjson is understood, but I don't think it is 100% vetted
  <wallyworld_> 1.16 should use sjson if it is there
  <adam_g> does anyone know what the status of the juju side of https://bugs.launchpad.net/maas/+bug/1239488 ?
  <_mup_> Bug #1239488: [SRU] Juju api client cannot distinguish between environments <verification-done> <MAAS:Fix Released by julian-edwards> <maas (Ubuntu):Triaged> <maas (Ubuntu Saucy):Fix Committed> <https://launchpad.net/bugs/1239488>
  <wallyworld_> sinzui: the above bug will be fixed with the 1.16 release you are doing now, right?
  <sinzui> I don't think so wallyworld_ The bug does not claim to affect juju, let alone have a bug targeted to the 1.16.1 release
  <sinzui> These are the bug fixes we want to close https://launchpad.net/juju-core/+milestone/1.16.1
  <wallyworld_> there were juju changes to fix that maas env bug
  <adam_g> sinzui, i was told there was a corresponding juju change for the multi-tenant usage to actually work, have not been able to find what that buig is, tho
  <wallyworld_> bigjools: ^^^^^?
  <wallyworld_> you made changes to juju-core right?
  <bigjools> yes
  <wallyworld_> was the branch linked to a bug?
   * bigjools digs
  <adam_g> are those released ?
  <wallyworld_> not yet i don't think
  <bigjools> https://bugs.launchpad.net/juju/+bug/1081247
  <_mup_> Bug #1081247: maas provider releases all nodes it did not allocate [does not play well with others] <maas-provider> <pyjuju:Triaged> <juju-core:Fix Committed by julian-edwards> <juju-core 1.16:Fix Committed by rogpeppe> <MAAS:Invalid> <https://launchpad.net/bugs/1081247>
  <adam_g> bigjools, thanks
  <bigjools> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1229275
  <_mup_> Bug #1229275: [maas] juju destroy-environment also destroys nodes that are not controlled by juju <maas-provider> <theme-oil> <pyjuju:Triaged> <juju-core:Fix Committed by thumper> <juju-core 1.16:Fix Committed by rogpeppe> <juju-core (Ubuntu):Triaged> <maas (Ubuntu):Triaged> <juju-core (Ubuntu Saucy):Triaged> <maas (Ubuntu Saucy):Triaged> <https://launchpad.net/bugs/1229275>
  <bigjools> also that, but it has the same branches linked
  <adam_g> sinzui, when is 1.16.1 due?
  <wallyworld_> bigjools: that bug is against the wrong projevt
  <wallyworld_> that may be why sinzui  didn't see it
  <bigjools> wallyworld_: ha!
  <wallyworld_> to include it for the next 1.16 release
  <bigjools> wallyworld_: it does have a juju-core task
  <wallyworld_> oh, so it does
  <bigjools> I invalidated pyjuju
  <bigjools> does this url make you feel better? https://bugs.launchpad.net/juju-core/+bug/1081247
  <_mup_> Bug #1081247: maas provider releases all nodes it did not allocate [does not play well with others] <maas-provider> <pyjuju:Invalid> <juju-core:Fix Committed by julian-edwards> <juju-core 1.16:Fix Committed by rogpeppe> <MAAS:Invalid> <https://launchpad.net/bugs/1081247>
  <wallyworld_> yes :-)
   * wallyworld_ relocates, bbiab
   * bigjools prepares for his arrival
   * bigjools chuckles, evilly
  <sinzui> sorry bigjools wallyworld_ I had to jump into a hangout.
  <bigjools> no worries sinzui
  <sinzui> I am not rushing the 1.16.1 release. I can wait for a critical bug fix. I just need to know what needs merging
  #juju-dev 2013-10-30
  <wallyworld_> thumper: if you have a moment, i'd love a review on https://codereview.appspot.com/16550046/ so i can land it for curtis. no real hurry or anything
   * thumper looks at wallyworld_'s work
  <wallyworld_> \o/ thanks
  <thumper> wallyworld_: I suggest we skip the package review today
  <thumper> nothing has been suggested and axwalk isn't around
  <wallyworld_> sounds good to me
  <thumper> cool
   * thumper goes back to writing emails
  <wallyworld_> thanks for review
  <thumper> np
  <thumper> sometimes I feel that if I don't suggest anything, you'll think I didn't read it
  <sodre> hi guys.. a quick question...
  <sodre> maybe not so quick.
  <sodre> I had a juju debug-hooks session going before I left the office.
  <sodre> That session got killed somehow. When I tried to start a new debug-hooks session from home, juju complains with the following error:
  <sodre> duplicate session: nova-cloud-controller/0
  <sodre> Connection to m1basic-07.vm.draco.metal.as closed.
  <sodre> ERROR exit status 1
  <sodre> is there a way to reset that status (possibly without reboot) ?
  <hazmat> sodre, you can login into the machine and kill the tmux session or reattach
  <sodre> that was my first thought :)
  <hazmat> tmux list-sessions shows nothing?
  <sodre> aha... :) my bad. I looked for byobu.
  <sodre> okay.. now I'm back! Thanks hazmat!
  <hazmat> np
  <bigjools> thumper: are you dealing with vlan stuff on the juju side?
  <bigjools> I have a vague plan of how I want to do this and need to talk to you guys
  <hazmat> bigjools, vlans as tag constraints or something more intrisinic?
  <hazmat> afaicr part of the issue is that provider specific tags are all exposed globally atm
  <hazmat> with something more intrinsic and specific to maas
  <hazmat> though i guess could just continue to stick more one use globals on.
  <bigjools> hazmat: it's all up for discussion. I don't necessarily like tags being used for everything. Not sure it'll work here either.
  <sodre> guys, is there a good starting point for deploying juju on a Private openstack install? I am having trouble with juju bootstrap
  <bigjools> hazmat: point is - nodes may be on multiple vlans and juju can ask for a specific vlan
  <hazmat> bigjools, you can allocate a machine on multiple vlans?
  <hazmat> gotcha
  <bigjools> hazmat: only if the machine has multiple network interfaces
  <hazmat> sodre, let's switch out to #juju and pastebin your $ juju bootstrap -v --debug
  <sodre> got it
  <hazmat> sodre, likely its an issue around syncing tools (juju sync-tools)..
  <sodre> could be.
  <hazmat> bigjools, right, but its a possibility, why would tags be a bad match here outside of the reuse of the semantic .. the and logic seems similiar
  <bigjools> hazmat: it could work for sure, we just need to discuss options and see if that is indeed the right way. I hate to jump on the first idea that comes up :)
  <hazmat> fair enough
   * bigjools lunches
  <hazmat> thumper, kinda of cool albeit not working, local provider and add-machine --to=lxc:1 ... the sub container doesn't  seem to be functional sadly.
  <hazmat> er.. not really create it appears
  <hazmat> any openstack/goose devs around?
  <thumper> hazmat: agreed, we don't support nested containers yet
  <thumper> bigjools: not sure who will be dealing with it yet
  <thumper> bigjools: we have to deliver a big hunk of stuff prior to network aware juju
  <thumper> bigjools: but I do have a big picture understanding of it
  <thumper> hazmat: please stop telling people to use "-v", if using --debug, verbose is implied
  <hazmat> really need some keyword calls for help here.. sodre is on #juju but having some issues getting a private openstack installation going .. looks like tool and image metadata issues.. but also notable as an annoyance is the upload of 34mb of tools every bootstrap and the failure of sync-tools
  <hazmat> thumper, sorry habbit
  <thumper> :)
  <bigjools> thumper: ok thanks
   * thumper looks at wallyworld_ for synctools problems that hazmat is watching someone have
  <wallyworld_> hmmm?
  <wallyworld_> k, /me read #juju
  <hazmat> wallyworld_, http://pastebin.com/t0H8DVVG
  <hazmat> wallyworld_, why is it doing uploads like that every bootstrap?
  <wallyworld_> hazmat: it uploads when it can't find tools
  <wallyworld_> iow it does a --upload-tools for you
  <wallyworld_> so the user doesn't have to type it
  <hazmat> wallyworld_, sure but every bootstrap?
  <wallyworld_> but if tools are there, it won't upload anything
  <wallyworld_> only if tools are missing
  <wallyworld_> once set up correctly, it won't do it
  <hazmat> umm.. he's done multiple bootstraps.. that's just his most recent run
  <wallyworld_> in that case, theres a problem finding tools
  <hazmat> yeah..
  <wallyworld_> otherwise it wouldn't upload
  <hazmat> wallyworld_, interestingly sync-tools through an error.. http://pastebin.com/BJJUrasS
  <hazmat> wallyworld_, way past EOD here.. hopefully you can help him out.. going to try and finish up some ODS demo work.
  <wallyworld_> huh, never seen that eror before
  <wallyworld_> ok, no problem
  <wallyworld_> thanks for helping
  <jam> wallyworld_: I'm off for a run, but the "invalid literal t" stuff looks like a bug in goose, search for something about json marshalling
  <jam> specificalyl, we make a list request
  <jam> and expect to get JSON back
  <jam> but end up getting just a plain listing
  <jam> which doesn't JSON.Unmarshal well
  <wallyworld_> oh, ok
  <bigjools> rvba: can you think of why the response to Azure's GetOSImages would get truncated inside the Go code? curl works ok.
  <bigjools> rvba: wallyworld_ is debugging and it only reads ~16k of a 159k response
  <rvba> bigjools: sounds fishyâ¦ I don't recall seeing thatâ¦ let me have a quick look at the code and see if anything comes backâ¦
  <wallyworld_> i blame ms... or go
  <jam> rvba: sounds a lot like an File.Read() rather than ioutil.ReadAll()
  <wallyworld_> jam: ioutil.ReadAll() fails the same way, so i have done my own read loop to try and see where it gets to
  <jam> wallyworld_: I wonder if it is getting something like EINTR and not retrying
  <wallyworld_> jam: could be. all i know is that reading from response.Body fails
  <wallyworld_> it is expecting 160kb response (which is absurd in itself but that's another story)
  <jam> wallyworld_: sure. I haven't seen it myself, I just know that it is easy to do a object.Read() when you actually want ioutil.ReadAll()
  <wallyworld_> but the connection is closed after reading 16k
  <jam> wallyworld_: there can certainly be a lot of OS Images ...
  <wallyworld_> jam: have you seen the xml. sooooo verbose
  <wallyworld_> all sorts of crap in there besides image details
  <rvba> wallyworld_: could this be related somehow to the change made in revision 226? (the "request.Close = true" thingy)?
  <wallyworld_> rvba: no, that change would have improve the situation if anything afaik
  <wallyworld_> cause it would have stopped go from reusing connections which may have been closed
  <rvba> I see.
  <wallyworld_> time to pick up the kid from school. will have to continue debugging when i get back
  <rogpeppe> mornin' all
  <dimitern> morning all
  <mgz> mornin'
  <dimitern> mgz, did we change the standup time to be 1h earlier? or my tz is wrong again in the calendar
  <mgz> hm, we just switched to summer time
  <mgz> so, I think it should have moved an hour forwards for both of us
  <dimitern> what's your local time now?
  <mgz> 10:20
  <dimitern> hmm, so the clock seems right
  <dimitern> but on the calendar I can see the standup at 11:45, whereas before it was at 12:45
  <mgz> right, I think the event is pinned on gmt, so, moves an hour with summer time ending
  <wallyworld> rvba: you around?
  <rvba> wallyworld: yep
  <wallyworld> rvba: running the azure test example utility, any idea why i get "The versioning header is not specified or was specified incorrectly"
  <wallyworld> gwacl/example/management/run.go
  <wallyworld> i had to hack out the retrier to stop the network connection close issue
  <rvba> wallyworld: hum, IIRC it should "just work", let me have a look.
  <rvba> wallyworld: what did you have to do?
  <wallyworld> instead of using the retrier: retrier := session.retryPolicy.getForkedHttpRetrier(session.client)
  <wallyworld> i just did a session.client.Do(req) directly
  <wallyworld> that allows the images to be listed ok
  <rvba> Weird.
  <wallyworld> there's a separate issue with the conection getting closed
  <wallyworld> so that hack just gets past that bit
  <wallyworld> i think there may be a real issue with the header version
  <wallyworld> since juju bootstrap also fails with that error on azure all of a sudden
  <rvba> Maybe the version we use is not supported anymoreâ¦
  <rvba> That would be weirdâ¦ but anything is possible.
  <wallyworld> yeah, maybe
  <wallyworld> thanks for looking :-)
  <rvba> I'm still looking :)
  <wallyworld> so we use "x-ms-version", "2013-03-01"
  <wallyworld> is that the header it is refering to?
  <rvba> Yes
  <wallyworld> ok, maybe there's anew one? where do we look to find out?
  <jam> mgz, dimitern: Generally we've left the meeting at GMT so that when we have 5 different daylight switchover times triggering asynchronously we don't have to bump it around. But we can discuss changing the time if it works better for people.
  <rvba> Somewhere in the documentation.  But the point of using version headers like this is exactly to avoid trouble when the API changes :).
  <jam> My guess is that when Nate changes TZ in a couple weeks we might all want to move earlier
  <wallyworld> yeah, i would have thought so too
  <jam> but we'll have to check with wallyworld if that is going to work
  <wallyworld> might work :-)
  <jam> wallyworld: don't you switch on to daylight savings time? or is Queensland DST agnostic
  <wallyworld> sadly QLD is retarded. no DST here :-(
  <rvba> http://msdn.microsoft.com/en-us/library/windowsazure/gg592580.aspx says "2013-03-01" is still the most recent version.
  <wallyworld> hmmm.
  <wallyworld> maybe there's an azure bug
  <dimitern> jam, nah, it's alright I just have to remember it's 1h later
  <jam> rvba: it also says "last updated November 19, 2012"
  <rvba> jam: indeed :)
  <dimitern> so are we using git now?
  <rvba> Let me try to run the example scriptâ¦
  <jam> dimitern: not as in "today we commit with git" but as in "we'll be working on migrating now"
  <wallyworld> rvba: ok. let me know if you have to hack out the retrier
  <dimitern> jam, i see
  <jam> I'm trying to decide my personal involvement.
  <wallyworld> rvba: my command line was: go run -a run.go -cert=/home/ian/.juju/azure.pem -location="West US" -subscriptionid="<blah>" -pause=true
  <dimitern> it would be useful to have a "crash course" comparison between common bzr and git commands
  <jam> dimitern: http://www.git-tower.com/blog/git-cheat-sheet-detail/
  <dimitern> jam, cheers
  <rvba> wallyworld: ta
  <jam> dimitern: http://doc.bazaar.canonical.com/migration/en/survival/bzr-for-git-users.html is the reverse
  <jam> but it probably has the correlations
  <wallyworld> jam: i had a quick look at git today - it is weird. git pull doesn't actually update your branch
  <wallyworld> and you get the whole repo whether you want it or not
  <jam> wallyworld: so git has a concept of a bunch of local branches that track remote branches
  <jam> wallyworld: and it doesn't support remote ops like "log"
  <jam> so you *have* to pull locally
  <jam> to be able to see the history
  <wallyworld> :-(
  <jam> wallyworld: and under that circumstance, it doesn't make sense to always merged
  <jam> merge
  <wallyworld> and you need to checkout after pull to see the changes?
  <jam> wallyworld: "git merge" will also "fast forward" like "bzr pull' unless you pass "--no-ff" or configure it to disable that.
  <jam> wallyworld: well, you can "git log" locally
  <jam> there is probably a "bzr cat" equivalent.
  <rvba> wallyworld: the command just hangs, no output, no nothing :/
  <wallyworld> i wonder how to set up a lightwieght checkout scenario
  <wallyworld> rvba: really? hmmmm
  <rvba> Oh wait, it crashed after a while
  <dimitern> so nothing in git like qbzr?
  <jam> wallyworld: each "git repository" is a working tree + repository + multiple branches
  <rvba> With "The versioning header is not specified or was specified incorrectly."
  <jam> so you generally just have a repo and switch between the branches in there
  <jam> essentially the colocated version of bzr by default
  <wallyworld> rvba: right, that's what i see, except i have the conn closed issue too
  <rvba> Gosh, what a mess.
  <wallyworld> yeah :-(
  <natefinch> morning all
  <jam> wallyworld: I know it is possible to play with ENV args like GIT_REPO to allow you to say the history is "over there". And I've seen comments about other bits of sharing histories between working trees.
  <jam> wallyworld: but AFAICT the *common* behavior is that you have just one tree that you switch between branches and they all share the same repo
  <wallyworld> jam: ok. seems like we need to work out best practice for bzr refugees
  <wallyworld> from what i understand, seems like a shame though it forces you to get the *whole* repo
  <jam> wallyworld: well, you close/archive/rename branches when you don't need them anymore
  <jam> just like in bzr, they are just pointers into the revision graph, so they are "cheap"
  <jam> (gits actual objects are cheaper than bzr's because of extra files bzr keeps around for each branch)
  <wallyworld> but unlike bzr, you get the whole repo and potentially waste lots of disk space if there are lots of blos etc
  <wallyworld> i wonder if that can be avoided
  <jam> wallyworld: bzr pull gets the whole history
  <jam> not sure what you mean about waste
  <wallyworld> for that branch
  <wallyworld> easier to explain verbally :-)
  <jam> wallyworld: so again, usually you would work in the same "working tree" and just pull the branches next to yours
  <wallyworld> right. but i was told if you get a working tree, you also get all other working tree for all other branches in the repo regardless if you want them or not
  <jam> wallyworld: "git clone" defaults to hardlinking the archive
  <natefinch> wallyworld: yes, you do.  But you generally wouldn't check out multiple copies of the same project
  <jam> wallyworld: no other working trees
  <jam> just branch pointers
  <wallyworld> right. but if a repo contains project foo and project bar and you just want foo, you are forced to get bar too
  <wallyworld> i was hoping to avoid that
  <natefinch> wallyworld: yes, well, don't put two projects in the same repo :/
  <wallyworld> ok. seems like many people do though. let's hope we don't
  <jam> wallyworld: sure. though as nate says, they don't generally share repositories for projects that don't share ancestry.
  <jam> wallyworld: there is "git clone --shared"
  <jam> and --local, --bare, --mirror
  <wallyworld> i'm just going y what i've read/seen
  <jam> and shallow clones with --depth
  <jam> and
  <jam>  --separate-git-dir
  <rvba> wallyworld: what was the symptom of the other issue?  "panic: use of closed network connection"?
  <jam> wallyworld: so you can probably do whatever the fsck you want
  <wallyworld> rvba: yes
  <wallyworld> jam: great :-)
  <wallyworld> jam: still trying to grok the best approach to suit my workflow
  <natefinch> jam, wallyworld : except most of the time it's best to just get the whole repo and switch branches as you need to.  That actually works perfectly with Gopath, that assumes one spot on your hard drive for a project
  <jam> wallyworld: though as you mention, clone copies all branches by default, so mixed project repositories will generally cause clone to copy multiple ancestries. In *git's* world view there is just one global ancestry that happens to have disjoint pieces
  <rvba> Oh wait, it crashed after a while
  <rvba> wallyworld: I don't see it with revision 225.  With revision 226, I get that "use of closed network connection" error: http://paste.ubuntu.com/6329291/
  <jam> natefinch: to be fair, the gopath model is particularly bad, so I'm not super keen that another tool does it, too :)
  <wallyworld> rvba: that is interesting indeed
  <jam> natefinch: prior to this, I always had 3 checkouts of a project I was working on. 1 was pristine trunk, 1 was the thing I was actively working on, and 1 was for code review that came back that needed tweaks without wanting to interrupt my current work
  <wallyworld> rvba: there's another way to try and work around Go's braindead http issue. i might try that
  <jam> natefinch: it is nice to be able to just fire up your text editor to see how it was done in trunk
  <natefinch> jam: you can still do that. They're just different branches, and switching between them is super fast in git
  <wallyworld> rvba: so that still leaves the versioning issue :-(
  <jam> natefinch: it is fast in bzr, too, it is *still* nice to not step on what I'm working on to see what it is like in another branch
  <jam> natefinch: note that I said *3* not N for all the branches I work on.
  <rvba> wallyworld: yep.  Let's see if the request looks sane.
  <jam> anyway, stand up time for those who are joining
  <jam> https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig
  <jam> rogpeppe, dimitern, wallyworld, natefinch: ^^
  <jam> mgz: ^^
  <mgz> ta
  <wallyworld> rvba: thanks so much for looking. i'll check back after our standup
  <dimitern> so it is 1h earlier then
  <rvba> wallyworld: np
  <wallyworld> sinzui: not sure if you are online. if you are, i have some info for you
  <rvba> wallyworld: it's the deletion of the affinity group request that failsâ¦
  <wallyworld> rvba: ok. do we do that during bootstrap also?
  <rvba> wallyworld: probably not
  <wallyworld> bootstrap fails with the "same" error
  <wallyworld> well, it did for me and curtis
  <rvba> wallyworld: well, maybe bootstrap does something similar (for some definition of "similar")
  <wallyworld> likely
  <wallyworld> i wonder why that particular api call fails
  <rvba> wallyworld: it would be worth knowing which API call fails exactly when running bootstrap.
  <rvba> Might help us pinpoint the problem.
  <rvba> Because the request seems okay to me, the api header is set.
  <wallyworld> rvba: i'll try and find out. but likely not tonight. too tired
  <wallyworld> rvba: unless ms changed something at their end?
  <wallyworld> wrt versioning?
  <wallyworld> nothing would surprise me
  <rvba> wallyworld: I tried using another version for that request, no dice
  <wallyworld> sigh
  <wallyworld> it shouldn't be this hard :-(
  <jam> natefinch: just wanted to poke, you'll be going of DST in another couple weeks, right? Is that going to impact you for the standup?
  <jam> wallyworld: Replacing the ginger beer with ginger wine leads to a particularly  lethal drink known as a Turbo Dark 'N' Stormy; a favourite tipple of Pembroke College Boat Club (Oxford)
   * wallyworld makes a note to try that tomorrow
  <natefinch> jam: yeah, I was going to bring that up.  The current standup time will soon be 5:45am for me, which is somewhat less than awesome
  <jam> natefinch: sounds pretty awesome for us. You were complaining that your daughter was waking up before you
  <jam> this way we get focused Nate time :)
  <natefinch> jam: ha yeah
  <jam> natefinch: so what time will work for you?
  <natefinch> jam: we could do a half hour earlier, that would be 6:15 my time, whcih is enough time for me to get up and make coffee, which is really all I need before the standup :)  Depends on what works for other people
  <jam> 'earlier' ?
  <natefinch> er later
  <wallyworld> i'll be totally drunk by then :-)
  <jam> wallyworld: not sure if that's a problem :)
  <wallyworld> that's 21:15 for me, but should work
  <wallyworld> jam: yeah, how would you know the difference, right?
  <jam> wallyworld: whmaobaou whaoeu baoeuwhaoeunwhou, might give it away
  <wallyworld> maybe :-)
  <jam> wallyworld: though, I guess when you got tipsy in SFO you didn't seem to mumble
  <wallyworld> i only had a couple of drinks, i'm not a 2 pot screamer :-)
  <wallyworld> hmmm. do you know what a 2 pot screamer is in the US?
  <wallyworld> maybe that's an australian expression
  <wallyworld> a "pot" in australia is a standard 285ml glass of beer
  <natefinch> wallyworld: never heard of a 2 pot screamer
  <wallyworld> maybe it makes sense now that i've explained what a "pot" is ?
  <natefinch> wallyworld: I inferred what it meant from context :)
  <jam> wallyworld: essentially a "cheap date" :)
  <wallyworld> yeah :-)
  <jam> natefinch: so if we chip in to buy you a coffee maker with built in grinder and timer we can keep the current time ? :)
  <natefinch> jam: sure, but I get to pick the model ;)
  <wallyworld> i don't mind if it is later
  <jam> natefinch: I think we can probably move it by 30min. It may mean that I get interrupted more when my son gets home, but I think we can work it out.
  <jam> natefinch: my wife and I had one, the main problem we had was that because the grinder is exposed (to let the grinds into the brewing section), it was *really* loud, and the grinding area needed to be cleaned after every run because the steam would get in there and you end up with soggy grounds
  <natefinch> jam: we can give it a try and change things if it's not working.   5:45 is less of a horrible time for me than most people I think.
  <jam> natefinch: sure, though it is the *daily* standup ,so I'd like to be at a reasonable time as much as possible
  <natefinch> jam: we could also move it even later, though that would probably mean Ian can't go
  <jam> natefinch: so we should definitely do it, you say ? :)
  <natefinch> jam: haha... it would mean about 50% less whining during the standup ;)
  <mgz> he doesn't drink that much wine
  <natefinch> interesting - https://github.com/juju
  <sinzui> Thank you for the email wallyworld
  <hazmat> jam, awesome stuff re scale testing.. have always enjoyed these write ups (+maas, +bzr back in the day)
  <hazmat> one curious bit was the 5s pump on the unit agent api for watch events, was wondering where that came from
  <hazmat> the all watcher client impl has something similiar but thats based on a periodic update and push, is that the same behavior/impl for the agent api?
  <sinzui> Hi devs, we have a critical bug in progress that I want you all to be aware of. Any insight into this issue will be appreciated. I currently think this issue is more than just blocking my 1.16.1 release, I think it is a bug that needs fixing for 1.16.1: https://bugs.launchpad.net/juju-core/+bug/1246320
  <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
  <rogpeppe> lunch
  <rogpeppe> hazmat: all watchers use the same underlying poller (implemented by the state/watcher package)
  <jamespage> sinzui, this might sound like an odd question but the 1.16.1 release will just be juju-core changes right?
  <jamespage> not a re-sync of trunk of deps as well
  <sinzui> jamespage, https://launchpad.net/juju-core/+milestone/1.16.1is just bug fixes that are not entangled with other libraries. The fixes were backported from trunk
  <jamespage> sinzui, goody
  <jamespage> just making sure
  <jamespage> sinzui, dependency management needs to be a key focus for 14.04 if we want to get juju-core into main
  <sinzui> jamespage, understood.
  <jamespage> sinzui, I'm going to take a look at the state of the golang tooling for packaging early in cycle
  <jamespage> if its good enough I'd like to push out dependencies into their own packages
  <sinzui> jamespage, I am considering releases doe gwacl, goose, and gomaasapi. Would you want them as packages in 14.04?
  <natefinch> sinzui, jamespage: I would find it odd to have random Go packages as packages for apt-get.  They're basically useless on their own, and don't represent an actual usable application, unlike juju-core.  But maybe I'm misunderstanding what's being proposed.
  <sinzui> natefinch, it is odd. We tend to bend things like java to debian package management. debian maven for example strives to collect java libs in a tree that is shared for debian package management
  <sinzui> natefinch, I don't know the right path for go, but since it can wrap c libs, it cannot assume all packages are the same
  <jamespage> sinzui, not yet
  <sinzui> PS I had zero success with debian and maven for 13.04 and don't want to try anything like that again
  <arosales> mgz, if you have some time today could you take a look at the merge request @ https://code.launchpad.net/~dstroppa/juju-core/add-joyent-provider/+merge/189360
  <arosales> mgz, or possibly get it to the correct person for review
  <mgz> arosales: sure, that sounds reasonable
  <mgz> ah, whoops, I fixed bug 1243861 but didn't link the bug
  <_mup_> Bug #1243861: juju should add the cloud-archive repository differently <juju-core (Ubuntu):Confirmed> <https://launchpad.net/bugs/1243861>
  <arosales> mgz, thank you sir
  <mgz> ah, it's an ubuntu bug, that's why I missed it
  <sinzui> mgz, we have a fix for that bug going in 1.17 ?
  <mgz> sinzui: it's in 1.16.1 too
  <sinzui> fab.
  <mgz> linking up now
  <mgz> er... or, actually not merged on 1.16 branch yet, just proposed
  <mgz> submitted.
  <mgz> knew I was forgetting some branch as well as the openstack one
  <sinzui> Hi juju dev's I have updated https://bugs.launchpad.net/juju-core/+bug/1246320 . juju 1.14.1 is also broken with the same error. I think azure or streams has changed. I am looking for insight into the root cause
  <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
  <mgz> does seem like an azure change, we discussed it briefly in the standup
  <mgz> I guess the next step is fiddling with the version info gwacl sends and seeing what happens
  <mgz> or just escalating to microsoft and asking what changed
  <arosales> +1 on the importance of sinzui's bug
  <arosales> bug 1246320
  <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
  <mgz> arosales: have you got anyone on the azure end you can bother?
  <arosales> mgz, sure if its a platform issue I can ping microsoft
  <mgz> arosales: given it's failing on 1.14, 1.16, and trunk, when the exact same code was working before, something their side has changed
  <mgz> it may be that we now need to do something different, but it would be good to find out what they changed
  <arosales> mgz, if you can point me to an error message I can pass it their way
  <arosales> error message that would indicate azure is failing as they can still provision instances
  <arosales> be right back
  <mgz> is's basically MissingOrIncorrectVersionHeader, but we may still need to trace exactly what we're sending in the case it fails
  <mgz> hm... I have an idea
  <sinzui> mgz, I discussed azure APIVersion with utlemming in #juju.
  <sinzui> mgz, I just tried a global change of the APIVersion is gwacl by setting the date to today. All I accomplished was making bootstrap fail faster
  <rogpeppe2> dimitern: how's your knowledge of charm upgrade semantics?
  <sinzui> I will try some alternate dates
  <mgz> sinzui: too many jujus. I see that log now. did you also catch the earlier discussion here on it between wallyworld and rvba?
  <sinzui> I read it without full comprehension
  <mgz> well, rvba tried an alternate date and it didn't help
  <sinzui> mgz, fab, I will stop this line of inquiry
  <mgz> I do still worry about the whole on-some-requests-but-not-others aspects
  <mgz> could still be our http stack screwing up I guess, would be nice to wireshark and confirm the exact bits we send for the working vs non-working api calls
  <dimitern> rogpeppe, good
  <rogpeppe> dimitern: i'm trying to upgrade a charm that's in an error state, but it doesn't seem to be upgrading
  <rogpeppe> dimitern: from my reading of the logic in ModeHookError, it should allow upgrading in that case
  <rogpeppe> dimitern: ah, i get it now - i didn't use the upgrade-charm --force flag
  <dimitern> rogpeppe, right!
  <dimitern> rogpeppe, also, there's resolve --retry on the unit that's in an error state
  <rogpeppe> dimitern: yeah, i knew about that
  <rogpeppe> dimitern: i was just confused by the fact that the new charm had been downloaded, but the unit wasn't using it
  <rogpeppe> cool, it all works now.
  <natefinch> rogpeppe, mgz, dimitern:  my instance type constraint: https://codereview.appspot.com/14523052/
  <rogpeppe> natefinch: sorry, i'll have a look in the morning'
  <natefinch> rogpeppe: no problem
  <rogpeppe> natefinch: you might be interested to have a look at http://godoc.org/launchpad.net/juju-utils/cmd/gocharm
  <rogpeppe> natefinch: it's in an early stage - there are a few things that i know will change
  <natefinch> rogpeppe: nice, I'll definitely check it out
  <rogpeppe> natefinch: but the basic structure seems ok, i think
  <rogpeppe> natefinch: you'll also want to look at http://godoc.org/launchpad.net/juju-utils/hook
  <rogpeppe> natefinch: and there's an example charm in launchpad.net/juju-utils/cmd/gocharm/example-charms
   * rogpeppe is now done for the day
  <natefinch> rogpeppe: that's awesome.  Nice work
  <rogpeppe> natefinch: ta
  <rogpeppe> g'night all
  <bac> sinzui: would you have a moment for a hangout?
  <sinzui> I do
  <sinzui> arosales, juju-devs, I am contemplating freezing 1.16.1 and doing a release. I don't know how long it will take to sort out the azure issue, but I would rather do a 1.16.2 tomorrow then delay the maas and lxc fixes
  <arosales> sinzui, and it is looking like the azure issues may be platform dependent
  <arosales> sinzui, thanks for getting the xml rpc capture to utlemming.
  <arosales> sinzui, +1 on 1.16.1 from me and if needed a 1.16.2 for any azure fixes _if_  it is not the platform
  <sinzui> okay. I will proceed with the tests I setup yesterday, then deliver a tarball.
  <mgz> sinzui: going ahead with 1.16.1 now seems reasonable to me
  <thumper> morning folks
  <wallyworld_> sinzui: hi, so just to confirm, someone is talking to microsoft about the azure api issue?
  <sinzui> wallyworld_, I believe utlemming will
  <wallyworld_> ok, great
  <sinzui> He was in was delivering new images for them tomorrow
  <wallyworld_> i'll try and ensure the http connection issue (if i can confirm it for sure) is sorted
  <wallyworld_> btw, i'm almost 100% sure there's no streams issue here at all
  <thumper> finally wrote up loggo http://how-bazaar.blogspot.co.nz/2013/10/loggo-hierarchical-loggers-for-go.html
  <hazmat> thumper, ping..
  <thumper> hazmat: otp now
  <hazmat> thumper, any idea why a machine agent would just ignore containers deployed to it..
  <thumper> got more info?
  <hazmat> nothing in the machine log of note.. and a bunch of containers attached which are pending
  <hazmat> thumper switching channels
  <mramm> thumper: your blog post is now up on hacker news: https://news.ycombinator.com/item?id=6643805
   * thumper -> gym
  <hazmat> jujucore.. there's a panic dump in cloudinit from sodre on #juju
  <hazmat> setting mongodb password
  <hazmat> for posterity http://paste.ubuntu.com/6332775/
  <wallyworld__> hazmat: is there a bug raised?
  <hazmat> wallyworld__, dunno.. wasn't there some size issue with password and mongodb?
  <wallyworld__> seems so, i was just wondering if a bug was raised so we could track it. i'll search and raise one if not done already
  <hazmat> cool
  <sodre> for the record, when that ocurred the admin-secret was set to : 'super-secret'
  <hazmat> solid!
  <wallyworld__> bug 1246517 fwiw
  <_mup_> Bug #1246517: user edited admin-secret causes panic in cloud-init <bootstrap> <juju-core:Triaged> <https://launchpad.net/bugs/1246517>
  <sinzui> thumper, where is the equivalent of the all-machine-log for local-provider environments?
  <sodre> _mup_ even with a long password it still crashes for me.
  <hazmat> sinzui, its in $juj_home
  <wallyworld__> sinzui: btw, juju-core trunk has the --public option now for tools sync. so everything should be in place for metadata generation with mirrors
  <sinzui> okay, thank you wallyworld__
  <wallyworld__> let me know if i can do anything else, or if there are issues etc etc
  <sinzui> wallyworld__, azure did change API yesterday. They believe it is backwards compatible...
  <wallyworld__> not for us it seems?
  <sinzui> our apps would disagree
  <wallyworld__> yep
  <wallyworld__> do we have details?
  <wallyworld__> is there a fix i can work on?
  <bigjools> \o/
  <hazmat> so getting reports of lxc broken on maas in two different envs..
  <hazmat> if anyone can help debug .. see adam_g on #juju/can
  <sodre> hazmat, do you mean trying to add an lxc unit to a MAAS provisioned machine ?
  <bigjools> thumper did all the work for that
  <sinzui> thank you hazmat
  <sinzui> wallyworld__, they need URI, HTTP headers and Request body. my gwacl hack captured the body. I will hack gwacl some more to provide the rest
  <sinzui> wallyworld__, I need to finish 1.16.1 tarball first
  <wallyworld__> or i could help
  <sinzui> I cannot get local provider to work. I don't see any difference between 1.16 and 1.16.1
  <wallyworld__> what symptoms?
  <sinzui> wallyworld__, I will show you the patch
  <hazmat> sinzui, machine log?
  <hazmat> sinzui, lxc-ls output
  <hazmat> sodre, yes
  <hazmat> sodre, does it work for you?
  <sinzui> wallyworld__, http://pastebin.ubuntu.com/6332990/
  <sodre> nope. When I try to add an lxc unit it just hangs.
  <sodre> well, it keeps waiting for them to come up.
  <wallyworld_> sinzui: ok, i can get the extra info and put in bug report
  <sinzui> hazmat, The machine log : http://pastebin.ubuntu.com/6332998/
  <sinzui> hazmat, lxc-ls: curtis-local-machine-1  curtis-local-machine-2  lts-p
  <sinzui> hazmat, also, I did do an lxc-destroy for the two machines from previous tests because destroy-environment left them behind in /var/lib/lxc
  <hazmat> sinzui, its been working for me.. bbiam
  <hazmat> sinzui, lxc-destroy directly should be avoided if possible.. local provider leaaves some turds in /etc/lxc/autostart and /var/lib/juju otherwise
  <hazmat> not sure if it bookeeps enough to be smart about that
  <hazmat> sinzui, so lxc-ls --fancy shows them as running?
  <hazmat> sinzui, i assume there stuck in pending in status?
  <hazmat> sinzui, so if their running from --fancy.. log into one and pastebin /var/log/cloudinit-output.log
  #juju-dev 2013-10-31
  <sinzui> hazmat, yes --fancy shows them running
  <sinzui> hazmat, yes pending status, which is all I have ever seen since 1.14.0
  <hazmat> hmm
  <hazmat> that's a good start
  <hazmat> sinzui, jorge and marco had this issue, thumper had to do surgery on them.. but tbd if its needed here
   * hazmat waits for cloudinit output
  <sinzui> hazmat, cloud-init: http://pastebin.ubuntu.com/6333073/
  <hazmat> cool.. the source of fail
  <sinzui> which agrees with
  <sinzui> curtis-local-machine-1  RUNNING  -     -     YES
  <hazmat> sinzui, so let's try a clean tear down..
  <sinzui> I wonder if I need to locate abentley's bug that explained his two step clear lxc cache
  <hazmat> sinzui, issue there is no cloudinit data made it to the container
  <hazmat> sinzui, juju destroy-environment
  <sinzui> done
  <hazmat> sinzui, anything in ls /etc/lxc/auto ?
  <sinzui> no
  <hazmat> sinzui, how about ls /var/lib/juju/containers/
  <sinzui> hazmat, empty
  <hazmat> sinzui, hmm.. and ls /var/lib/juju/removed-containers/
  <hazmat> sinzui, this is trunk or 1.17 ?
  <hazmat> er. 1.16
  <sinzui> hazmat, 1.16.1 release candidate...the one that is supposed to fix lxc
  <hazmat> ah.. one of those
  <sinzui> hazmat, Lots in removed containers
  <sinzui> curtis-local-machine-1  curtis-me-machine-1    curtis-me-machine-1.2  curtis-me-machine-1.4
  <sinzui> curtis-local-machine-2  curtis-me-machine-1.1  curtis-me-machine-1.3  curtis-me-machine-2
  <hazmat> sinzui go ahead and kill them.
  <hazmat> and move  your $JUJU_HOME/$local_provider_name out of the way
  <sinzui> hazmat, done
  <hazmat> and that should be pristine for a new bootstrap
  <hazmat> sinzui, if your bootstraping.. with 1.16.1 tools you may need --upload-tools
  <hazmat> if there not published
  <sinzui> I think that last step is key. I have not done that today
  <hazmat> sinzui, yeah.. --debug on bootstrap and pastebin for posterity if needed
  <sinzui> hazmat, nothing different from my previous tests...so far http://pastebin.ubuntu.com/6333122/
  <hazmat> sinzui, can you set juju set-env  logging-config="<root>=DEBUG;juju.provisioner=TRACE"
  <hazmat> and then deploy something..
  <hazmat> sinzui, btw re logs $JUJU_HOME/$local-env-name/machine-0.log has the logs of interest
  <sinzui> yeah, that is what I am tailing
  <hazmat> oh.. cool
  <thumper> sinzui: no all machines log for local at this stage
  <hazmat> sinzui, so .. i get roughly the same content you did
  <hazmat> 2013-10-31 00:26:17 INFO juju.provisioner provisioner_task.go:367 started machine 1 as instance kapil-local-machine-1 with hardware <nil>
  <thumper> so, what's the summary of the current issues?
  <sinzui> hazmat, 3 running machines without net
  <hazmat> thumper, local provider seems to work on trunk.. sinzui has it failing for him 1.16.1
  <hazmat> thumper, somewhat more critical is that it doesn't seem to work on maas as a container at all atm
  <sinzui> hazmat, all three services/machines are pending
  <hazmat> sinzui, you can watch them do upstart and lxc-create via pstree
   * thumper sighs...
  <hazmat> thumper, the containers on maas are critical for some ODS stuff
   * thumper nods
  <thumper> let's get that working first then
  <hazmat> thumper, i'm going to give it a go with trunk on the maas i think..
  <hazmat> just to see if that helps at all. plus to have a source compile i can instrument.. since the logging here seems to be nil
  <thumper> hazmat: do you have any instructions for a virtual maas setup
  <hazmat> sinzui, cool, is cloudinit done from pstree?
  <thumper> so we can test locally?
  <sinzui> abentley reported this bug. I don't think it is related because I start my lts container every day
  <sinzui> https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1236490
  <_mup_> Bug #1236490: Container has no networking <amd64> <apparmor> <apport-bug> <saucy> <linux (Ubuntu):Fix Released> <lxc (Ubuntu):Invalid> <https://launchpad.net/bugs/1236490>
  <hazmat> thumper, you mean did i attend the maas training ;-) ?
  <hazmat> no actually..
   * bigjools bans the maas training from existence
  <thumper> sinzui: hangout? I can talk through some things
  <hazmat> sinzui, can you pastebin lxc-ls --fancy
  <hazmat> not sure what you mean by no networking
  <sinzui> hazmat, yes it is. I can tell when my fan stops actually
  <hazmat> cloudinit is injected here via file
  <hazmat> sinzui, wow.. is that spinning rust?
  <hazmat> sinzui,  or ssd?
  <sinzui> ssd
   * hazmat runs a dozen containers normally
  <hazmat> only 2-4 for juju though
  <hazmat> i don't even notice them
  <sinzui> I have my fan tuned to come on often...its a mac and they run hot
  <hazmat> gotcha
  <sinzui> hazmat, no improvement http://pastebin.ubuntu.com/6333175/
  <hazmat> that's intriguing
  <hazmat> sinzui, can you go to /var/lib/juju/containers
  <thumper> there should be console.log in there for each machine
  <hazmat> and pastebin the console.log
  <thumper> that is the output of running the cloudinit
  <hazmat> although in this case i think we want the container.log
  <thumper> sinzui: also pastebin ifconfig
  <hazmat> the netns stuff should have happened pre lxc
  <hazmat> er. pre cloudinit
  <sinzui> hazmat, console.log http://pastebin.ubuntu.com/6333189/
  <sinzui> thumper, my ifconfig or one from a container?
  <thumper> sinzui: from the host
  <hazmat> sinzui, can you pastebin the container.log post ifconfig ..
  <hazmat> that should detail why the no netns
  <sinzui> thumper, ifconfig http://pastebin.ubuntu.com/6333193/
  <hazmat> huh
  <hazmat> you've got three veth devices there.
  <hazmat> for your three containers
  <hazmat> sinzui, is dnsmasq running ?
  <thumper> sinzui: how about the lxc.conf from the /var/lib/juju/containers/<firstmachine>/
  <hazmat> you should see something like... 116       1190  0.0  0.0  28184   804 ?        S    Oct15   0:04 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative
  <hazmat> er.. ignoring the preamble.. that's from ps aux | grep dnsmasq
  <sinzui> hazmat, yes http://pastebin.ubuntu.com/6333198/
  <hazmat> sinzui, the container.log should have the netns come up....
  <sinzui> hazmat, the lxc.conf from -1 http://pastebin.ubuntu.com/6333199/
   * hazmat leaves to thumper's capable hands
  <thumper> ok, this needs help from an lxc person
  <thumper> as it is outside the remit of juju
  <thumper> the container didn't get networking
  <thumper> prior to all the juju shit
  <sinzui> thumper, fab, as I think too
  <thumper> sinzui: if you bring up a machine manually, does it get an ip address?
  <sinzui> while abentley's bug is not identical, I will try the work around
  <sinzui> thumper, do you mean bring one down than bring it back up again?
  <thumper> sinzui: what do you get with "ls -l /var/cache/lxc" ?
  <thumper> actually
  <thumper> sinzui: ls -lh /var/cache/lxc/cloud-precise/
  <sinzui> thumper,
  <sinzui> drwxr-xr-x 2 root root 4096 Jul  1 18:29 cloud-precise
  <sinzui> drwxr-xr-x 3 root root 4096 Jan  9  2013 precise
  <sinzui> thumper, -rw-r--r-- 1 root root 221M Jun 24 03:07 ubuntu-12.04.2-server-cloudimg-amd64-root.tar.gz
  <sinzui> -rw-r--r-- 1 root root 206M Dec 17  2012 ubuntu-12.04-server-cloudimg-amd64-root.tar.gz
  <thumper> ok, that's it
  <thumper> ubuntu-12.04-server-cloudimg-amd64-root.tar.gz is too old
  <thumper> and the refresh mechanism isn't working
  <thumper> this is the problem marco had
  <thumper> and jorge
  <thumper> smoser is aware
  <thumper> sinzui: delete the file
  <hazmat> sinzui, can you post the console.log
  <thumper> destroy the environment
  <hazmat> er.. container.log
  <hazmat> sinzui, ^
  <thumper> and start again
  <hazmat> before destroying
  <smoser> thats a known issue.
  <smoser> yeah, remove that file.
  <thumper> lxc will download a fresh copy of the cloud image
  <hazmat> oh.. that one
  <thumper> and it "should" work
  <hazmat> i forget about that.. i ran into that as well
  <hazmat> smoser, so what causes that?
  <hazmat> smoser, with the new lxc cloud template / cloud img download there's no dot versioning around point releases..
  <hazmat> but never really understood why the old container wouldn't netns
  <thumper> sinzui: still there?
  <smoser> "new lxc cloud template" ?
  <sinzui> hazmat, I have a copy of the container log. anything else before I start purging
  <smoser> there is no such thing as "dot"
  <smoser> ever.
  <smoser> 12.04.3 is no different than 12.04
  <smoser> the cloud images do not distinguish them in name. they did at one point and that was a bug.
  <hazmat> smoser, it is though from a cached file sense
  <smoser> that is independent of .2 verus .3
  <hazmat> previously it did.. and it worked better for caching and invalidation purposes when the point release incremented
  <smoser> previously it was broken on the server side in naming
  <smoser> which broke people who expect to 'wget http://path/to/some/known/current/release/file.img'
  <smoser> https://bugs.launchpad.net/ubuntu/+bug/1220366
  <_mup_> Bug #1220366: cloud-images have inconsistent filenames in 12.04.3 <bot-comment> <cloud-images> <cloud-images-build> <Ubuntu:Fix Released by utlemming> <https://launchpad.net/bugs/1220366>
  <sinzui> thumper, I am bootstrapping again
  <hazmat> ic
  <hazmat> that should fix sinzui's issue then
  <smoser> the cloud template needs to be smarter.
  <smoser> but we really would rather make a lxc wrapper that was smarter.
  <smoser> ie, hallyn doesn't think such knowledge should live in lxc itself.
  <hazmat> smoser, ala the kvm front end.. or ala docker front end?
  <smoser> uvtool-lxc is the plan in my head.
  <smoser> which is also ideally where the "--use-the-fastest" cloning argument goes.
  <sinzui> My fan has started
  <smoser> :)
  <hazmat> smoser, the default behavior with -s works for me generally.. ie autodetect btrfs ;-).. but i'm curious about the docker lvm thin provisioning on sparse file dev pool
  <sinzui> whoa! there it is. Installed
  <hazmat> sidnei just got the that merged into lxc-clone (re lvm-thin) support
  <hazmat> sinzui, do we have tools published for 1.16.1 or tarball?
  <smoser> hazmat, i think there were issues with that default behavior.
  <smoser> are you sure it even remains ?
  <smoser> and it wouldn't select unionfs
   * hazmat checks
  <sinzui> Thank you rocket scientists! hazmat thumper smoser. I really appreciate your patience
  <thumper> sinzui: awesome
  <thumper> that's one fixed
  <thumper> and that fault should definitely be logged on the local provider troubleshooting page <- jcastro, marcoceppi
  <thumper> now to look at maas
  <bigjools> o/
  <bigjools> I just got pinged about this
  <thumper> hazmat: what versions were you playing with?
  <bigjools> narrowed it down any yet?
  <thumper> bigjools: not started looking yet
  <hazmat> thumper, 1.16.0 for maas
  <bigjools> thumper: I guess I should fire up my maas server? :)
  <thumper> bigjools: gimmie time to shower (haven't since back from the gym yet), and we can have an initial call prior to working out debugging plan
  <thumper> bigjools: ack
  <sinzui> hazmat, yes, my testing tools are in place in aws and hp and canonistack. I place them in the testing/ subdir and point the tools url to them: http://juju-dist.s3.amazonaws.com/
  <bigjools> thumper: roger mcdodger
  <sinzui> hazmat, and I will officially make the tarball in about 10 minutes
  <hazmat> smoser, it requires explicit lxc-clone -s  but yeah.. it works without specing btrfs directly
  <hazmat> du -hs /var/lib/lxc/container-name -> 7mb
  <sinzui> hazmat, and you can see all the scripts I am using at https://code.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2
  <sinzui> you can make your own deb without any juju parts installed
  <wallyworld_> bigjools: how long does an juju azure bootstrap instance normally take to spin up?
  <smoser> wallyworld_, ~ 5 minutes i think
  <bigjools> wallyworld_: from memory not at all quick, I think it was like 5-10 m
  <wallyworld_> ok, ta. just bootstrapped an and waiting for juju status
  <wallyworld_> s/just/have been
  <marcoceppi> thumper: can you summarize what should be logged?
  <thumper> marcoceppi: the symptom was that the machines were pending, and lxc-ls --fancy showed the machines started but with no ipv4 address
  <thumper> marcoceppi: ls -lh /var/cache/lxc/cloud-precise showed ubuntu-12.04-server-cloudimg-amd64-root.tar.gz to be over 4 months old
  <thumper> marcoceppi: delete that file, destroy the environment and try again
  <thumper> a new lxc cloud image will be downloaded and all should be good
  <thumper> bigjools: hangout?
  <thumper> marcoceppi: does that all make sense?
  <sinzui> thumper, marcoceppi It makes sense to me. marcoceppi  can interview me tomorrow or he/nick can invite me to try updating the docs
   * thumper nods
  <thumper> thanks sinzui
  <bigjools> thumper: calling you....
  <thumper> which me?
  <bigjools> the head
  <bigjools> wrong one?
  <bigjools> this is why I don';t have two g+ profiles
  <marcoceppi> thumper: yeah, that's what happened to me
  <marcoceppi> already recorded
  <wallyworld_> bigjools: do you know the magic command to ssh into a bootstrap node on azure? i have the ip address of the machine
  <bigjools> juju ssh 0 :)
  <wallyworld_> bigjools: doesn't work. times out
  <wallyworld_> bootstrap has failed
  <wallyworld_> but i have the ip address of the bootstrap node
  <bigjools> ummmm in that case you're probably screwed
  <bigjools> it will be firewalled
  <wallyworld_> well, when i say bootstrap failed, it completes but juju status fails
  <bigjools> ok
  <wallyworld_> and i want to find out why
  <bigjools> can you telnet port 22?
  <wallyworld_> i'll try
  <wallyworld_> yes
  <wallyworld_> when i try ssh directly, it fails with a permission denied error (publickey)
  <bigjools> so it's up but hasn't added the key
  <wallyworld_> yeah, seems so
  <sinzui> wallyworld_, with azure, I wait 30 minutes for the machine to really be up
  <bigjools> which means bootstrap was not as successful as you might think
  <wallyworld_> sinzui: really? oh joy
  <bigjools> !
  <wallyworld_> wtf takes so long
  <bigjools> oh - yes the crazy provisioner
  <bigjools> you can see in the MS dashboard
  <wallyworld_> so before when juju status timed out and i destoyed my env, i should have just  waited
  <sinzui> wallyworld_, on canonistack, we get TLS timeouts while the bootstrap node is coming up, but azure is silent. I have seen the machine come up in 20 minutes, but 30 is realistic
  <wallyworld_> dashboard? you have a link?
  <wallyworld_> sinzui: ok. i think i've fixed the azure incompatibility but am trying to make sure
  <bigjools> wallyworld_: well not exactly you would need my password.  let me look at it
  <wallyworld_> ok
  <wallyworld_> bigjools: i bet i can guess it
  <wallyworld_> aussieaussieaussieoioioi
  <bigjools> wallyworld_: "ianisacocktrumpet"
  <wallyworld_> that would have been my next guess
  <bigjools> wallyworld_: in fact it was so secure I can't remember it :)
  <wallyworld_> ha ha ha
  <bigjools> wallyworld_: get your own credentials
  <wallyworld_> bigjools: they're in the mail apparently
  <wallyworld_> i only needed yours cause of this azure cockup and the time pressure to fix it
  <bigjools> yeah np
  <sinzui> I encrypted my creds and put them in the CI machine so that my team could build better testing
  <wallyworld_> i wonder what else besides juju msft broke with their change?
  <bigjools> wallyworld_: btw baldrick left a hidden present that the mower found
  <wallyworld_> \o/ :-D
  <bigjools> you can imagine the result
  <wallyworld_> did it splat all over you /me asks hopefully?
  <bigjools> >:(
  <sinzui> Your dog is named baldrick?
  <wallyworld_> yep :-D
  <wallyworld_> he has a cunning plan
   * sinzui was just telling his son about Ebenezer black adder 5 minutes ago.
  <bigjools> wallyworld_: I see two VMs running
   * wallyworld_ is finding it hard to type cause he is laughing so hard to bigjools
  <bigjools> one of them from juju the other I think for a gwacl run
  <wallyworld_> bigjools: ah the gwacl one needs to be shutdown. i thought i did
   * bigjools considers putting baldrick's present in wallyworld_'s mailbox
  <wallyworld_> sinzui: blackadder is one of my favourite tv shows
  <wallyworld_> bigjools: how can you, it's all vapourised all over you, right?
  <bigjools> there's some left
  <wallyworld_> next time get jen to mow the lawn after baldrick has visited :-D
  <bigjools> wallyworld_: so the cpu graph shows it finishing a heavy load at 11:45
  <wallyworld_> the juju one?
  <wallyworld_> what's the ip address?
  <bigjools> 137.135.11.14
  <wallyworld_> yep, that's the one juju status keeps polling mongo for trying to do a status
  <wallyworld_> :-(
  <wallyworld_> i really need to get the logs off that sucker
  <bigjools> wallyworld_: 17070 and 37017 ports are open
  <bigjools> that's it
  <wallyworld_> 37017 is the mgo port
   * bigjools wonders if it's the non-ssl mgo
  <wallyworld_> and it says conection refused
  <wallyworld_> ah
  <wallyworld_> that could be it
   * bigjools gets lunch while wallyworld_ fixes it
   * wallyworld_ doesn't know how to fix that one
  <bigjools> deploy with saucy?
  <wallyworld_> ah ok
  <wallyworld_> good idea
  <bigjools> it might have failed to add the cloud archive
  <wallyworld_> yeah, sounds likely
  <bigjools> although doesn't explain why you can't ssh in
  <wallyworld_> hopefully saucy will work
  <bigjools> heard a lot of swearing coming from the living room earlier - jen found jake pulling keys off her laptop again... ROFL
   * bigjools lunches
  <wallyworld_> lol
  <thumper> sinzui: still around?
  <sinzui> I am
  <thumper> sinzui: I have a critical bug
  <thumper> sinzui: for 1.16
  <thumper> sinzui: it is the lxc and maas problem
  <thumper> that they are using for ods
  <thumper> demo
  <thumper> always last minute
  <thumper> it is a regression introduced when the provisioner moved to the api
  <sinzui> I just cut 1.16.1, but plan a 1.16.2 for azure.
  <thumper> but no one noticed
  <sinzui> thumper, Do I need to take down 1.16.1
  <thumper> no
  <thumper> I don't think so
  <thumper> because they can use a custom release
  <thumper> I think
  <thumper> but we may have a 1.16.2 real soon now
  <thumper> if they do need a release
  <thumper> what's the timeframe for 1.16.2?
  <thumper> bigjools: I have a plan
  <sinzui> I can make it tomorrow immediately after 1.16.1 is needed. I hope for an azure fix of couse
   * thumper nods
  <thumper> ok
  <thumper> let me file this bug and get to work
   * sinzui creates milestone
  <wallyworld_> sinzui: azure fix will be commited to gwacl today i hope
  <sinzui> wallyworld_, excellent. Testing will be quick since it is limited to two providers
  <wallyworld_> i'm doing a local test now
  <sinzui> Did I mention abentley got rudimentary CI running on HP. we can run about 200 upgrade and bootstrap tests a day
  <wallyworld_> ran into the precise mongo thing i think
  <wallyworld_> trying on saucy
  <thumper> wallyworld_: are you chasing the azure issue?
  <thumper> wallyworld_: is there a bug?
  <wallyworld_> yes. the issue is with msft, not us
  <wallyworld_> we need to change gwacl cause they broke the api
  <wallyworld_> i'm testing now
  <thumper> :)
  <thumper> wallyworld_: can you target the bug to 1.16.2?
  <wallyworld_> but azure is sloooooow
  <wallyworld_> thumper: sure. there's 2 of them - a public and a private
   * thumper nods
  <thumper> bigjools: lp:1246556 for the juju maas bug
  <thumper> which it seems mup can't understand, bug 1246556
  <_mup_> Bug #1246556: lxc containers broken with maas <api> <maas-provider> <juju-core:Triaged by thumper> <https://launchpad.net/bugs/1246556>
   * thumper moved some code around, all tests still pass
  <thumper> I find this mildly disturbing
  <wallyworld_> bigjools: not sure how many azure instances i have running - i think 2. a bootstrap using saucy still doesn't allow juju status to connect sadly
  <wallyworld_> thumper: i have 2 small azure mp's for the 1.16.2 release if you have a moment sometime https://code.launchpad.net/~wallyworld/gwacl/azure-management-api-change https://code.launchpad.net/~wallyworld/gwacl/fix-request-eof-2/+merge/193346
   * thumper is in deep thought
  <thumper> sorry
  <wallyworld_> np
  <wallyworld_> bigjools: what is the price for 2 small gwacl reviews?
  <wallyworld_> needed for juju 1.16.2 release tomorrow
  <thumper> bigjools: bugger...
  <thumper> bigjools: it seems like the quick fix won't work very well at al
  <thumper> may have to fix it the right way
  <bigjools> wallyworld_: I can look for you, after all you came bearing gifts yesterday
  <wallyworld_> indeed i did
  <wallyworld_> they are only small
  <wallyworld_> branches
  <bigjools> thumper: great architecture FTW
  <bigjools> 466 lines
  <bigjools> is small in juju world?
  <wallyworld_> bigjools: sadly i have a saucy node running now which also disallows connections to mongo
  <bigjools> wallyworld_: is the firewall endpoint open?
  <wallyworld_> not sure. what port is that do you know?
  <wallyworld_> i thought you said before 37017 was open?
  <wallyworld_> which is the port the client is sing to contact mongo
  <bigjools> ok
  <bigjools> wallyworld_: hang on I need to make call first
  <wallyworld_> ok
  <bigjools> wallyworld_: first comment - pease remove the factored version numbers
  <wallyworld_> why?
  <bigjools> I previously had someone stop doing that - each api request is separately versioned and if you factor it you will introduce subtle  bugs
  <wallyworld_> it kinda sucks having them all copied and pasted in the code
  <bigjools> yes - but it needs to be done
  <wallyworld_> ok then
  <bigjools> I rooted out at least 3 bugs last time I unfactored it
  <wallyworld_> sigh
  <bigjools> I know
  <wallyworld_> how the fook do we keep track of all the apis
  <wallyworld_> sounds like a nightmare
  <bigjools> wallyworld_: fwiw the version should be associated with the request struct ideally
  <bigjools> they change versions when the format changes
  <bigjools> so they are very closely tied
  <wallyworld_> ok. for now though, i'll just go back to how it was
  <wallyworld_> and do a search and replace of string
  <bigjools> wallyworld_: for example if you update the refactored version you would change all the calls at once, and introduce potentially subtle sematic changes
  <wallyworld_> i thought they would all change together
  <wallyworld_> as a group
  <bigjools> IME they dont
  <wallyworld_> :-(
  <bigjools> it might say they do....
  <bigjools> but I call BS
  <wallyworld_> fair enough. it is msft after all
  <bigjools> you might want to mention this in the code
  <wallyworld_> ok
  <thumper> arg
  <thumper> stabby stabby
   * thumper is trying to unpick something
  <thumper> but others are sewing things tighter together
  <bigjools> wallyworld_: I bailed on the other one
  <wallyworld_> bigjools: i wanted to avoid a random custom data so i could hard code the expected response
  <bigjools> wallyworld_: why?
  <wallyworld_> same reason i like to use strings in tests
  <bigjools> the test demonstrates that it copes with random input
  <bigjools> you should always use random data where the data itself doesn't matter
  <wallyworld_> using the same api call in a test though as the logic sort of negates the test
  <bigjools> otherwise it says to the test reader that you're crafting a particular scenario
  <bigjools> feel free to ignore it, you have my +1 already
  <wallyworld_> ok, i'll think about it
  <wallyworld_> thanks
  <wallyworld_> why bail on other one?
  <bigjools> because it's deep Go innards that I know nothing about
  <wallyworld_> ok
  <bigjools> you're the Go expert :)
  <wallyworld_> i used same technique as in goose etc
  <wallyworld_> you reckon?
  <bigjools> self-approve?
  <wallyworld_> might do
  <bigjools> do you need all you instances on my azure account?
  <bigjools> your
  <wallyworld_> bigjools: the saucy one would be good. but i need to be able to ssh in to see the cloud init log
  <bigjools> you need a custom image with the backdoor
  <wallyworld_> ugh ok
  <bigjools> you have to build one and upload it to private storage and tell the api to use it when booting
  <bigjools> I forgot how though :(
  <wallyworld_> i never knew how
  <thumper> jam3: ping
  <wallyworld_> in the first place
  <bigjools> there's a config item for a custom kernel
  <bigjools> image I mean
  <jam> thumper: pong
  <thumper> jam: hangout? I have a critical bug to discuss
  <wallyworld_> bigjools: and you can't ssh in at all and grab the cloud init log?
  <jam> thumper: sure, I have to head out in about 20min
  <thumper> jam: ack
  <thumper> jam: https://plus.google.com/hangouts/_/72cpjufs8mm0i0htkjiuch72s8?hl=en
  <jam> I'm getting "call ended because of server error" give me a sec
   * thumper nods
  <jam> thumper: https://plus.google.com/hangouts/_/72cpjqbfsmf42rpnbt2o36cgk0
  <jam> see if this one works
  <jam> If I follow your link, I got the "new" hangout layout
  <jam> thumper: I saw you show up for 1 second
  <thumper> jam: I have new, you have old
  <thumper> they don't like each other
  <thumper> server error
  <thumper> skype?
  <jam> thumper: know of any way to trigger the right one?
  <thumper> mumble
  <thumper> I don't care
  <jam> thumper: I'm on skype, and you're in my friends list
  <jam> but not showing as online
   * thumper starts skype
  <thumper> jam: https://bugs.launchpad.net/juju-core/+bug/1246556
  <_mup_> Bug #1246556: lxc containers broken with maas <api> <maas-provider> <juju-core:Triaged by thumper> <https://launchpad.net/bugs/1246556>
  <bigjools> wallyworld_: no, unless cloud-init finishes putting the ssh key in your're buggered
  <wallyworld_> \o/
  <wallyworld_> maybe it should do that first up if it doesn't already
  <bigjools> wallyworld_: hence you need the backdoor.  Having said that - there is an api field for adding a user and p/w
  <bigjools> but we don't use it as it's mental
  <wallyworld_> got a hint where i look save me searching?
  <wallyworld_> i might do it to debug
  <bigjools> hang on
  <bigjools> wallyworld_: in LinuxProvisioningConfiguration you set a Username and Password.  It's in the same place Customdata goes
  <wallyworld_> ok, ta
  <bigjools> wallyworld_: line 545 in provider/azure/environ.go
  <wallyworld_> great
  <bigjools> it was a mandatory field so we randomised it
  <bigjools> yay security
  <wallyworld_> lol serious? mandatory?
  <wallyworld_> wtf
  <bigjools> this is msft
   * wallyworld_ sighs *again*
  <thumper> jam:  lp:~thumper/juju-core/maas-lxc
  <sodre> juju-devs: is anyone still up ?
  <wallyworld_> sodre: some of us are here
  <sodre> hi wallyworld_: I traced the reason for that earlier panic on juju bootstrap
  <wallyworld_> oh cool
  <wallyworld_> admin-secret issue i think?
  <sodre> I've updated the bug report on launchpad. But I was wondering if you could explain to me something related to it.
  <sodre> yeah, it had nothing to do with the admin-secret.
  <wallyworld_> i'll read the bug
  <sodre> The issue is that jujud tries to get the provider-state through swift without authentication.
  <wallyworld_> it should use creds when it tries to get it
  <wallyworld_> i think from memory
  <wallyworld_> let me check
  <sodre> from the client side, (juju) connects to swift using goose.
  <sodre> but the bootstrap level seems to use (Go 1.1 package http)
  <jam> sodre, wallyworld_: provider-state needs to be in a container that is world readable because all of the clients download stuff from the container via wget/direct http access
  <jam> when we set up the bucket ourselves, we set the .r:.rlistings *
  <sodre> for somereason that is not being set on on my version
  <jam> sodre: you're on Havana ?
  <sodre> yes
  <wallyworld_> jam: not necessarily the conatiner itself but the file
  <jam> wallyworld_: we need the container, we put all the charm data, tool data, etc in there
  <wallyworld_> i think
  <jam> and all needs to be downloadable from plain wget
  <jam> we don't need it on EC2 because we can generate Signed URLs, but we don't have support for that on Openstack
  <wallyworld_> and doesn't wget work so long as the file itself is readable?
  <sodre> right now, my way around it has been to put tools in the juju-dist and set that world readable.
  <sodre> but then this provider-state issue came up.
  <jam> wallyworld_: as in, we put a bunch of stuff in the container, and *each* needs to be world readable. It is a fair point that you could just have each file have the ACL, but we need all of them anyway
  <wallyworld_> jam: sure. but now in trunk, public storage instances are gone from the providers
  <wallyworld_> azure, maas never had them anyway
  <jam> wallyworld_: "private" storage *must* be public for Openstack
  <jam> wallyworld_: so if your patch which got rid of the PublicStorage api also removed setting .r:.rlistings for the other Storage, then you broke openstack deployments
  <wallyworld_> i didn't tinker with the other Storage to my kmowledge
  <jam> wallyworld_: we *don't* want to give our Provider credentials to the agents running on non machine-0, so they have to have credentials-free access to download the charm blobs from storage
  <jam> wallyworld_: "rlisting" doesn't exist in the source tree
  <wallyworld_> jam: right. so i just checked, the "private" storage on openstack does have rlistings perms
  <jam> unless that is in goose
  <wallyworld_> jam: it's a const in goose
  <jam> wallyworld_: k
  <wallyworld_> swift.PublicRead
  <wallyworld_> see line 514 of provider.go
  <wallyworld_> in openstack package
  <jam> wallyworld_: yep
  <jam> I found ti
  <jam> sodre: can you "swift stat" your container to see what the ACL is ?
   * wallyworld_ has to go get kid from school
  <sodre> I need to bootstrap again, because I've set it by hand.
  <sodre> give me one se.d
  <sodre> also, I'm on 1.16, not trunk. fyi
  <sodre> .r:*
  <sodre> humm... strange....
  <sodre> i need a new bucket name... one sec.
  <sodre> jenvs from hell...
  <sodre> the read acl is empty http://paste.ubuntu.com/6334137/
  <sodre> jam: any ideas if that can be fixed in 1.16.1 ? or 1.17 ?
  <wallyworld_> sodre: did you create that control bucket by hand?
  <sodre> yes
  <sodre> I got it to work as follows
  <wallyworld_> sodre: because juju will not set read acl if it exists already
  <sodre> I first swift post juju-control-bucket --read-acl .r:*
  <sodre> no... no..
  <sodre> if the bucket does not exist, it is still not setting it to .r:*
  <wallyworld_> really? the code in that area hasn't changed. it uses PublicRead = ACL(".r:*,.rlistings")
  <sodre> maybe it is the , .rlistings ?
  <sodre> because it is not setting it to read mode right now
  <sodre> ... it could be a radosgw issue again..
  <wallyworld_> could a havana thing maybe?
  <sodre> let me try the following.
  <sodre> I have a bucket right now with read ACL set to empty.
  <sodre> swift post pet.sodre.juju-control-02 --read-acl .rlistings
  <sodre> the read acl is still empty.
  <sodre> should it have changed ?
  <wallyworld_> um. not sure. i don't use the swift tool for that sort of thing too often
  <wallyworld_> cause juju does it for me :-)
  <sodre> are you using radosgw as well ?
  <wallyworld_> i've tested juju on hpcloud and our own internal openstacj deployment using folsom and grizzly
  <sodre> two variables: It could be Havana, or it could be RadosGW
  <wallyworld_> i'm testing now our our openstack
  <wallyworld_> sodre: so a seift post does set the read acl for me
  <wallyworld_> ie it is non empty after a swift post --read-acl ....
  <sodre> can you paste the readacl line for me.
  <wallyworld_> sure
  <wallyworld_>  Read ACL: .rlistings
  <sodre> humm..
  <sodre> strange!
  <wallyworld_> and i think this is on havana
  <sodre> what is .rlistings for anyways ?
  <wallyworld_> it allows unauthed clients to list the container
  <wallyworld_> the contents thereof
  <sodre> I see.
  <sodre> It seems that Rados does not support rlistings
  <sodre> shouldn't a plain .r:* suffice ?
  <wallyworld_> well that sucks if true
  <wallyworld_> don't think so - certain operations do need to list the files in a container
  <wallyworld_> but i can't tell you the specifics of the top of my head
  <sodre> http://wiki.ceph.com/01Planning/02Blueprints/Dumpling/Better_Swift_Compatability_for_Radosgw
  <wallyworld_> ah
  <wallyworld_> well, that sorta explains it :-(
  <wallyworld_> it does say .rlisting has something done
  <sodre> there as a patch in the wild, but it seems they did not want to apply it.
  <sodre> according to http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg13829.html
  <sodre> all that is needed is to set a READ permission to  see it via S3
  <sodre> I imagine unauthenticated access.
  <sodre> wallyworld_: I've updated the bug report one more time.... what is the easiest way for me to modify the juju-core code?
  <wallyworld_> what do you want to change?
  <sodre> I would like to see why it is not honoring the .r:*
  <sodre> when working with RadosGW
  <wallyworld_> right now, you need to use bzr to get a copy of the source code off launchpad
  <wallyworld_> go get works too
  <wallyworld_> have you ever done that?
  <sodre> yeah, I did it quickly today. I am not sure how to ``debug'' the code, ie.. gdb did not really help
  <sodre> is there a way to step through, or are debug statements the standard ?
  <wallyworld_> lol
  <wallyworld_> Go has very limited tooling support for debugging
  <sodre> i.c.
  <wallyworld_> in my experience
  <wallyworld_> i think you can use gdb
  <wallyworld_> but i have not done that myself
  <wallyworld_> i'm used to modern IDEs
  <sodre> sure I can use that as well, will it work in eclipse ?
  <wallyworld_> perhaps. I use Intellij where there is a decent Go plugin but it does lack debug support sadly
  <sodre> juju-core will be the first "Go" code I ever read
  <wallyworld_> it's similar syntax to C - not too hard to read
  <sodre> alright... so I don't spend too much time digging... in which file should I be looking for these rlistings ?
  <wallyworld_> for openstack, there's two main files of interest....
  <wallyworld_> provider/openstack/provider.go <--- this one sets up the storage instance for an openstack environment
  <wallyworld_> it creates it with the ACL I pasted above
  <wallyworld_> and provider/openstack/storage.go <--- this one creates the container (if required) and reads/writes/lists files therein
  <wallyworld_> your main interest will be in storage.go
  <sodre> got it
  <wallyworld_> the Storage interface is quite simple - it has List(), Put(), Get() etc methods
  <wallyworld_> i normally just use lots of fmt.Println() statements to debug
  <wallyworld_> thrre's also debug logging
  <wallyworld_> if you run a command like bootstrap with --debug, it will print useful info also
  <sodre> yeah, --debug is my friend :)
  <sodre> it seems the code went to goose
  <sodre> i'll spend some time with it.
  <sodre> thanks a lot man!
  <wallyworld_> np. do ask if you have more questions
  <wallyworld_> yes, goose is the lib used by juju to talk to openstack
  <sodre> thanks. I should probably go to sleep ... its 4am here
  <sodre> the good thing is that there is a work around
  <wallyworld_> wow. yes. go to bed!
  <wallyworld_> where are you?
  <sodre> tz= EST
  <sodre> I live in the DC area
  <wallyworld_> ah ok. i'm in australia
  <wallyworld_> so almost drinking time here :-)
  <sodre> nice :)
  <wallyworld_> thanks for all your hard work!
  <sodre> I'm trying to deploy OpenStack at work
  <sodre> I need to show them a demo ...
  <sodre> so... here is an issue
  <sodre> I am running my compiled version of juju
  <sodre> how do I bootstrap my system ?
  <sodre> it can't find 1.17
  <wallyworld_> you need to use --upload-tools
  <wallyworld_> if you are running from source
  <sodre> tsk tsk tsk... IU need to sleep
  <sodre> :)
  <sodre> s/IU/Im
  <wallyworld_> yeah tomorrow!
  <sodre> btw...
  <sodre> 2013-10-31 08:05:44 INFO juju.environs.sync sync.go:103 listing target bucket
  <sodre> 2013-10-31 08:05:44 DEBUG juju.environs.tools storage.go:35 reading v1.* tools
  <sodre> 2013-10-31 08:05:45 ERROR juju supercommand.go:282 failed to list contents of container: pet.sodre.juju-control-02
  <sodre> issue *listing contents*
  <wallyworld_> right, that is consistent with not having rlisting permissions
  <wallyworld_> juju needs to list the tools available to find the correct ones to use
  <sodre> this is a bad bug in RadosGW
  <sodre> alright.. ttyl
  <rogpeppe> mornin' all
  <rvba> Hi wallyworld_, did you figure out what the issue with the Azure provider was?
  <wallyworld_> rvba: hi, yes. msft screwed us over
  <rvba> wallyworld_: nice :/
  <wallyworld_> they changed the api in an incompatible way
  <rvba> Great.
  <wallyworld_> tl;dr; we needed to change the apiversion passed to various management calls
  <rvba> So much for having API versions and all that jazz.
  <wallyworld_> yeah
  <wallyworld_> and we also needed to tweak the run utility to base64 encode the userdata
  <rvba> All right, thanks for the update.
  <wallyworld_> i'm landing some stuff now. thanks for the input yesterday
  <wallyworld_> just had trick or tweaters at my door
  <wallyworld_> treaters lol
  <mgz> mornin'
   * thumper hopes jam can run the meeting tonight as he is hacking now he's back
  <natefinch> morning all
  <jam> fwereade: welcome back
  <jam> team meeting: https://plus.google.com/hangouts/_/calendar/bWFyay5yYW1tLWNocmlzdGVuc2VuQGNhbm9uaWNhbC5jb20.09gvki7lhmlucq76s2d0lns804
  <jam> fwereade_: ^^
  <jam> axw: ^^
  <fwereade_> jam, I am trying
  <jam> sure, we've been having trouble with new vs old G+ interfaces
  <axw> my machine froze, joining now
  <fwereade_> jam, it's weird, it is literally *just* the hangout that's not working
  <jam> fwereade_: extra joy in that it was working with just you and I until more people joined
  <fwereade_> jam, yeah, indeed
  <fwereade_> jam, there's a crazy storm out here but I doubt its effect is selective enough to be causing this
  <jam> fwereade_: is dimiter around today?
  <fwereade_> jam, he is, but he was out earlier; he expected to make it back for the meeting though
  <rogpeppe> damn, did i miss the team meeting today?
  <rogpeppe> jam: sorry about that, i totally forgot
  <natefinch> rogpeppe: yeah but you didn't miss much
  <rogpeppe> jam: i'd have joined if you'd pinged me...
  <rogpeppe> natefinch: ok, that's good
  <natefinch> rogpeppe: thumper was just talking about how much he loves having loggo on github
  <rogpeppe> natefinch: ironically?
  <thumper> ?!?
  <rogpeppe> thumper: you really *do* love having loggo on github?
  <thumper> heh
   * thumper goes back to work
  <natefinch> rogpeppe: I wasn't being serious :)
  <rogpeppe> natefinch: it sounded like you weren't - just checking
  <natefinch> rogpeppe: actually I think the only real complaint was having his username in the import url.  He says it feels too unprofessional
  <rogpeppe> natefinch: yeah, that's actually part of why i chose launchpad for some projects
  <rogpeppe> natefinch: the user name doesn't seem like that important a part of the project
  <natefinch> rogpeppe: yeah, I wish there was a different way to do it... you can do something like a project team and have the project under the team.
  <rogpeppe> natefinch: yeah
  <thumper> can you have a team of one?
  <sinzui> yes
  <sinzui> You can have a team of 0 in Lp
  <thumper> sinzui: I was referring to github
  <thumper> jam: I think I may have a complete fix with tests
  <natefinch> thumper: yes, you can have a team of 1 on github
  <thumper> natefinch: so I could create an org called loggo?
  <thumper> is it worth it?
  <sinzui> thumper, github is similar to Lp, no one is in the team when it is created, you add members.
  <natefinch> thumper: up to you... you'd have github.com/loggo/loggo
  <thumper> meh
  <thumper> except they call it an organisation
  <mgz> make the team ~loggo :)
  <thumper> heh
  <mgz> ACCEPT THE TILDE
  <thumper> probably an invalid character
  <natefinch> yeah it's just alphanumeric and dashes
   * thumper twiddles fingers while the local lxc downloads and the amazon one spins up
   * thumper proposes while he waits
  <thumper> ok... local provider still works with my change
   * thumper waits for ec2
  <natefinch> why working so late, thumper?\
  <thumper> natefinch: critical bug around lxc and maas
  <thumper> natefinch: they want it for the ODS demo
  <thumper> and sinzui is waiting on it to cut 1.16.2
  <natefinch> thumper: ahh, bummer
  <thumper> sinzui: care to land a branch that increments the version number on lp:juju-core/1.16
  <sinzui> thumper, oops, I will do that, thank your for reminding me
  <natefinch> sinzui: please make sure you increment scripts/win-installer/setup.iss as well.  It needs to get updated for the windows installer.
  <natefinch> (it's just a text file, the version number is at the top, you'll see it)
  <sinzui> natefinch, oh. I've got no notes on that. Have you been doing it each time you run it?
  <natefinch> sinzui: yeah, I've just been modifying it when I create the installer, but obviously that's not a good habit to keep up
  <natefinch> sinzui: I'm considering just making script that'll update all the right spots.
  <thumper> \o/
  <thumper> works on ec2
  <thumper> have ubuntu deployed into 1/lxc/0
  <thumper> jam, fwereade_: https://codereview.appspot.com/20220043/
  <thumper> and dimitern if he wants
  <thumper> this is to fix lxc on maas for the demo
  <thumper> makes it so we don't get an environ config for lxc provisioners
  <dimitern> thumper, looking
  <thumper> local provider works
  <thumper> and ec2 was able to provision a container
  <thumper> and deploy the ubuntu charm into it
  <thumper> so we know it at least works on something
  <sinzui> I wish I knew how to make DeployLocalSuite.TestDeployNumUnits pass on my computer
  <thumper> I really want to hand this off as it is almost 1am
  <dimitern> thumper, reviewed
  <thumper> dimitern: ta
  <thumper> sinzui: about to land my 1.16 branch
  <thumper> sinzui: wallyworld_ landed the azure fixes already
  <sinzui> I saw
  <thumper> sinzui: once mine is in, you should be good for 1.16.2
  <sinzui> rock
  <sinzui> Maybe I wont purse 1.16.1. It has not built yet
  <thumper> sinzui: however I'm about to go to bed
  <thumper> hopefully all lands ok
  <thumper> how long does it normally take?
  <thumper> and who can see if it is in progress?
  <sinzui> thumper, jamespages has the juju release scripted. He builds the new package and send it off to Ubuntu and the PPA builders in on go.
  <sinzui> He emails me when he has run the script
  <thumper> sinzui: I was really referring to the landing bot
  <sinzui> 15 minutes
  <thumper> I think I'll wait just to be sure
  <sinzui> natefinch, Is this right https://codereview.appspot.com/20230043 ?
  <natefinch> sinzui: great.  LGTM'd
  <sinzui> jamespage, are you about?
  <natefinch> fwereade_: do we have a list of the client commands that need to be API-ified?  I figured I'd knock one or two out
  <fwereade_> natefinch, we have a list of the ones that don't, and there are 2 of them
  <natefinch> fwereade_: haha ok
  <natefinch> fwereade_: what uses the API now, so I can see how it should be set up?
  <natefinch> fwereade_: nevermind, looks like add unit does  (I thought I remembered that one did)
  <fwereade_> natefinch, I think "get" does too
  <natefinch> fwereade_: figured I'd start on status, unless you have another suggestion
  <fwereade_> natefinch, yeah, let me chat to you about it after meeting
  <natefinch> fwereade_: sure.  I'm going to be AFK in about a half hour (for about an hour)
  <jam> natefinch: status is going to be much bigger than some of the other ones, and might involve some checking to make sure we are getting the passwords set on first connect, etc.
  <jam> something like "set" would be much easier if you want to get your feet wet
  <natefinch> jam: that's fine, I can do set to start :)
  <mattyw> the env vars that get set in a running hook - like: CHARM_DIR and JUJU_UNIT_NAME - where abouts are they actually set in the code?
  <fwereade_> mattyw, uniter/context.go
  <mattyw> fwereade_, cool thanks, think I just found it
  <rogpeppe> fwereade_: do you know any way of configuring the local provider so that it doesn't download a whole precise image when a precise charm is started?
  <rogpeppe> fwereade_: (at least, i *think* that's what it's taking ages to do - i'm not sure if there's any way of telling at all)
  <fwereade_> rogpeppe, I in't think we grabbed the whole thing every time
  <rogpeppe> fwereade_: i'm not sure how to tell what it's doing
  <rogpeppe> fwereade_: will we always grab the whole thing in a newly bootstrapped local env?
  <fwereade_> rogpeppe, I believe that is currently the case, but I have not looked at the code in question myself
  <rogpeppe> fwereade_: ah, sorry, i thought you were the one that reviewed it
  <rogpeppe> fwereade_: so no way to avoid that first download then. hmm.
  <rogpeppe> fwereade_: another small question: is there a good place for a charm to stash local state that *doesn't* get managed under git? for example, a unix-domain socket or something that should not be rolled back on failure?
  <fwereade_> rogpeppe, there's no canonical location, no
  <rogpeppe> fwereade_: i guess $CHARM_DIR/../charm-state might work ok, although it's not great.
  <fwereade_> rogpeppe, eww? name it after the charm outside the juju data-dir I would think?
  <rogpeppe> fwereade_: or just make up a name in /var/lib which includes the env uuid and the unit name
  <fwereade_> rogpeppe, +1
  <mramm> hey all: thumper's blog post about his logging library for Go is now up on hacker news: https://news.ycombinator.com/item?id=6643805 please upvote it if you think it is interesting, and feel free to comment on hacker news if you think you have something to add.
  <natefinch> mramm: if I could get to hacker news, I'd upvote it
  <adam_g> what is the proper way to upgrade an environment using a custom built juju and jujud binary pair? someone showed me last week but i've forgotten
  <mgz> adam_g: the magic is simply to have the juju you run to do the upgrade have the matched jujud as a sibling in the directory
  <mgz> this is very magic, but hey go hey go go
  <adam_g> thats what i thought
  <adam_g> adding the --upload-tools flag seems to skip going to s3
  <mramm> natefinch: problems with hackernews? thumper had them yesterday too. :(
  <natefinch> mramm: looking online, it's a DNS propagation issue
  <sodre_zzz>     juju-devs: has anyone seen this type of error with Juju & RadosGW
  <sodre_zzz> caused by: failed unmarshaling the response body: bootstrap-verify
  <mgz> sodre_zzz: can you manually fetch that object from the provider storage and see what it contains?
  <mgz> ...what provider even is that?
  <sodre> OpenStack, but Swift is served by ceph-radosgw
  <mgz> so, `swift list` to get CONTAINER, then `swift get CONTAINER bootstrap-verify`
  <sodre> alright... one sec.
  <hazmat> wallyworld_, wasn't there some facility you mentioned about auto upload-tools?
  <sodre> ahhh.. the error is different!
  <sodre> mgz: it says  juju was unable to list the contents of the container.
  <mgz> sodre: looking in ~/.juju/environments/ENV.jenv where ENV is the name of your environment will also tell you if you find the 'control-bucket' key at the bottom
  <sodre> Let me pastebinit
  <sodre> wasn't there a but where juju is expecting to see a .json object when listing contents in swift?
  <sodre> s/but/bug/
  <mgz> sodre: we might not be parsing the error response correctly
  <mgz> so, you're seeing a follow-on error rather than the underlying cause
  <mgz> sodre: can you file a bug against juju-core for this please?
  <sodre> will do, but can you take a look at the pastebin first ?
  <mgz> I need to transfer back home now, so would like to pick up later
  <mgz> sodre: sure, fast :)
  <sodre> http://paste.ubuntu.com/6336731/
  <sodre> this is with Havana, fyi.
  <mgz> sodre: yeah, looks like we're just expecting json and getting plain text
  <sodre> yeap, the error is in goose
  <mgz> there's a bug for this (or something very related) against goose already
  <sodre> yeap, that's what I thought.
  <mgz> if you can find that and add more details, that would be ace
  <sodre> i'll search for that.
  <sodre> thanks mgz
  <mgz> I've already got that bug on my list to look at
  <fwereade_> bbl
  <sodre> jam: are you around >?
  <sodre> I have quick goose patch/bug
  <natefinch> sodre: it's pretty late for Jam, almost 10pm.  Anything that I could help you with?
  <sodre> sure
  <sodre> the current version of goose/swift.go
  <sodre> it requests a list of entries in a container but forgets to set the format to json
  <sodre> the patch is very short,  http://paste.ubuntu.com/6336923/
  <sodre> natefinch, what do you think ?
  <natefinch> sodre: looks good, except you need to gofmt your source.  You have two spaces where the standard format is a tab.
  <sodre> got it, how is that done ? I am new to go
  <natefinch> go fmt <filename>  it'll rewrite the file
  <sodre> http://paste.ubuntu.com/6336944/, where should I send this to?
  <natefinch> so, in theory, you should branch lp:goose, make your change, and then propose the branch for merging to the lp:goose trunk
  <sodre> got it
  <natefinch> sodre: if you have credentials etc on launchpad, it's not too hard
  <sodre> I am new to Opensource collaboration...
  <sodre> I have an account on lp. but I've never contributed anything other than bug reporrts.
  <sodre> can you quickly walk me through it or point me to docs with the steps.
  <natefinch> sodre: I'm pretty new to the launchpad process too.  Let me see if I can find the steps somewhere
  <sodre> okay.
  <natefinch> sodre: ok, so presuming you started off by doing a bzr branch lp:goose to get your local copy.... you can do this to commit and propose your change (from the root directory of goose on your local drive):
  <natefinch> bzr commit -m 'commit comment'
  <natefinch> bzr push lp:~sodre/goose/<branchname>
  <natefinch> bzr lp-propose lp:goose
  <sodre> cool
  <sodre> I'm filing the bug report as well.
  <natefinch> awesome
  <natefinch> sodre: thanks for the bug and for the fix.  It's really a huge help even for little fixes like this.
  <natefinch> brb
  <sodre> np. I am just trying to get OS running here at work.
  <natefinch> TIL: when it looks like everyone else on freenode has quit at the same time... probably it's you that has actually quit
  <natefinch> sodre: I missed anything you might have said after you said you were just trying to get OS to work.
  <sodre> I didn't say anything :)
  <sodre> I just came across the bug because I need to get OpenStack running where I work.
  <sinzui> natefinch, will you have time today to create a a juju 1.16.2 windows package and file an Rt to get it signed?
  <natefinch> sinzui: sure will.   Hopefully this time the RT will actually be acted upon
  <sinzui> natefinch, send me the rt number and I will track it. I am happy for a delay since I never know when the builders will complete
  <natefinch> sinzui: ok
  <natefinch> sinzui: you probably will get an email about it, but here's the link: https://rt.admin.canonical.com/Ticket/Display.html?id=65618
  <sinzui> thank you natefinch. I will add that to my task list for 1.16.2
  <natefinch> sinzui: no, thank you.  I just want to make sure the windows installer stays available at the same version as the other platforms
  <sinzui> This one has to get to the users since they are the group that most likely will use juju on azure.
  <natefinch> morning thumper
  <thumper> morning natefinch
   * thumper feels a little knackered
  <natefinch> seems like you were just here :)
  <thumper> :)
  <thumper> yeah
  <thumper> I know that feeling
  <thumper> however I also have three kids still in bed
  <thumper> instead of at school
  <thumper> the event we went to last night was a lot later than I expected
  <thumper> didn't get home until about 22:30
  <natefinch> thumper: doh
  <thumper> which is way late for the youngest two
  <thumper> if I knew it was going to be that late, I would have organised a babysitter
   * thumper shrugs
  <thumper> oh well
  <natefinch> how old are your kids?
  <thumper> 8, 10 and 12
  <thumper> it was a "world of wearable arts" show for the eldest
  <natefinch> sounds like fun
  <thumper> it was quite good, but went on and on
  <thumper> you can really tell which outfits the kids did themselves, and which parents helped on
  <thumper> :)
  <natefinch> haha yeah....
  <jam> thumper: I had some comments on your proposal, they aren't critical but it might be good to get them in if sinzui is going to be doing a 1.16.2 soon
  <sinzui> jam, the tarball was sent
  <jam> well then, I guess it doesn't matter :)
  <sinzui> I might be convinced to make another if there are zero downloads of it
  <jam> sinzui: I'm not a big fan of reusing numbers regardless
  <thumper> jam: I may not worry too much for the 1.16 branch, but may take them into account on the move to trunk
  <sinzui> me neither. This is case where the one packers we are delivering too is traveling
  <thumper> sinzui: how dare he?
  <thumper> travel when we need him
  <thumper> geez
  <sinzui> SPoF
  <natefinch> busfactor violation
  <thumper> true that
  <jam> sinzui: I *thought* that when he set up the super special PPA he did give you access. At least, I listed a short list of names
  <jam> with you on it
  <sinzui> I don't see any new PPA on my list...
  <sinzui> oh, let me check teams
  <natefinch> thumper: if you get time today, I have a couple branches that could use some reviewing:  ec2 instance constraints: https://codereview.appspot.com/14523052/   modification to destroy-environment to require environment name: https://codereview.appspot.com/14502070/
  <jam> I don't know if that team actually got set up, but we did discuss it
  <thumper> natefinch: ok, I'll add to the list :)
  <sinzui> jam, thank you and your elephantine memory. I was added to a  team  8 days ago
   * sinzui starts scripting
  <thumper> natefinch: so this is "juju destroy-environment amazon"
  <natefinch> thumper: correct
  <thumper> cool
  <thumper> juju destroy-environment production
  <thumper> oops
  <natefinch> that's basically the only change from current behavior
  <natefinch> I did add an exclamation point to the warning message, too.
  <thumper> :)
  <thumper> nice
  <natefinch> forgot there was a tweak to scanning the input... the code that was there before was actually completely borked. It happened to work, but also returned an error we were ignoring
  <arosales> bcsaller, do you have some time to join us @ https://plus.google.com/hangouts/_/calendar/YW50b25pby5yb3NhbGVzQGNhbm9uaWNhbC5jb20.tj9cngmc3p25r5nvbdthk5up8c
  <sodre> sinzui: have you already fixed 1209003
  <sinzui> sodre, no, no work has been done on it yet
  <sinzui> sodre, sorry, work has started on it and iI need to target it
  <sodre> okay. I've placed the patch on a branch for review. It is one-line.
  <sodre> I found the bug through a different route, so I did not notice it was duplicate of your bug until a few minutes ago.
  <thumper> sinzui: good news, adam_g confirmed my work on maas, it makes lxc containers work (FSVO work)
  <sinzui> I hit the mysql+apparmor bug while testing local provider today. I almost stalled the releases. I am glad I remembered the bug
  <thumper> sinzui: what's that bug?
  <sinzui> thumper, https://bugs.launchpad.net/juju-core/+bug/1236994
  <_mup_> Bug #1236994: Mysql doesn't deploy on local provider due to apparmor <juju-core:Incomplete> <mysql (Juju Charms Collection):New> <https://launchpad.net/bugs/1236994>
  <thumper> wha?
  <thumper> it used to work, what broke?
  <sinzui> thumper, I think the issue is intermittent. I think I deployed that last night when I confirmed my local provider worked
  <jamespage> sinzui, I am now - I guess you need me to push 1.16.2 everywhere right?
  <natefinch> I'm outta here. night all
  <thumper> night natefinch
  <jamespage> sinzui, don't expect armhf for quantal and raring btw
  <jamespage> we are missing the dpkg magic fix in those series right now
  <sinzui> quantal is dead to me
  <sinzui> maybe we should stop releasing quantal packages. I think users have had 6 months to get to raring
  <jamespage> sinzui, not really - raring expires in 2 months
  <jamespage> :-)
  <jamespage> 9 month support these days
  <jamespage> sinzui, I'd prefer we keep publishing - I still have to test using juju-core for those series
  <sinzui> yeah, but I think juju-core has tried to give users an extra 3 months.
  <jamespage> but just not fuss about armhf
  <jamespage> sinzui, where do you want to pull trusty binaries from? I've not uploaded to ppa for that series?
  <jamespage> I can do - the distro package will supercede anyone actually running on trusty once it lands
  <thumper> sinzui, jamespage: FWIW +1 on not caring about quantal and raring armhf
  <sinzui> jamespage, this is the script we use for releases and CI. We pull from 3 archives, including Ubuntu: http://pastebin.ubuntu.com/6337941/
  <sinzui> jamespage, I think trusty will work. I need to switch to Lp API to get packages from the new archive.
  <jamespage> sinzui, should do
  <jamespage> I'll leave it off the backports like last cycle then
  <sinzui> sorry lines 197  start where I define the archives to search
  <jamespage> sinzui, OK - its nearly baked - everything is built - just waiting for LP to publish to ppa.launchpad.net
  <sinzui> fab. I am rushing to finsh my script while getting children ready for Halloween
  <jamespage> sinzui, oh fun
  <jamespage> sinzui, bear in mind that http://archive.ubuntu.com/ubuntu/pool/universe/j/juju-core/ won't get the armhf bits
  <sinzui> ugh
  <jamespage> sinzui, trying to remember where they appear
  <jamespage> its ports.ubuntu.com or something
  <jamespage> sinzui, http://ports.ubuntu.com/pool/universe/j/juju-core/
  <sinzui> jamespage, will we do this for the final release of trust?
  <jamespage> sinzui, sorry - do what?
   * jamespage thinks his brain is not quite working right
  <sinzui> will armhf always be in ports for trust?
  <jamespage> sinzui, yes
  <jamespage> I think so
  <jamespage> sinzui, might be easier to just pull everything from the PPA
  <sinzui> My mind has left the building. I look forward to a walk in the dark with people dressed weirder that me
  <sinzui> Do my eyes deceive me, I think everything is int eh staging ppa
  <jamespage> sinzui, yes - its all done
  <sodre> sinzui: should I change #1209003 to Fix Commited ? I have placed a pull request already.
  <_mup_> Bug #1209003: juju bootstrap fails with openstack provider (failed unmarshaling the response body) <openstack> <Go OpenStack Exchange:In Progress by psodre> <juju-core:Triaged> <https://launchpad.net/bugs/1209003>
  <thumper> man...
  <thumper> I'm normally a bit drained on Fridays due to the late meetings
  <thumper> but actually coding after the meetings to fix the critical bug as knocked me out
   * thumper turns on the coffee machine
  <jamespage> sinzui, I'm dropping for today
  <jamespage> sinzui, if you need me todo anything tomorrow am my time ping me a mail
  <sinzui> Thank you for your time jamespage
  <jamespage> sinzui, np - ditto
  <bigjools> thumper: a good coffee fixes everything
   * thumper looks for wallyworld_
  <wallyworld_> yes?
  <thumper> hey
  <wallyworld_> ho
  <thumper> got time to chat?
  <wallyworld_> ok
  <thumper> https://plus.google.com/hangouts/_/72cpimmkjidp1clo6ipgc9gg90?hl=en
  <adam_g> is bootstrapping an environment /w 1.16.2-precise-amd64 expected to pull 1.16.0 tools or 1.16.2 down from S3?
  <thumper> adam_g: I believe it matches on major and minor
  <thumper> adam_g: to be sure --upload-tools
  <adam_g> thumper, seemed to have only matched major. ill roll with --upload-tools for now
  <jcastro> http://askubuntu.com/questions/369263/juju-bootstrap-fails-with-temporary-failure-in-name-resolution-using-amazon-aw
  <jcastro> did we break a bucket?
  #juju-dev 2013-11-01
  <wallyworld__> jcastro: that bucket is his control bucket, created on demand at bootstrap time; it's not a public bucket which we could break
  <wallyworld__> maybe ec2 was being particularly slow at that time - sadly stuff can take time to propagate through after an action is performed from what I understand
  <wallyworld__> juju does wait for ec2 to "catch up"; maybe it didn't wait long enough this time. or maybe there was another error which went unnoticed in the output
  <sinzui> thumper-afk, that mysql on lxc bug is very erratic. I just deployed it twice on lxc successfully.  My successful runs are between the hours of 0 an 6 UTC
  <adam_g> hmph
  <adam_g> trying to use 2 different juju environments in one maas (two different maas users/keys)
  <adam_g> http://paste.ubuntu.com/6338725/
  <adam_g> one fails to start
  <wallyworld__> adam_g: 1.16 has a bug in this area - 1.16.2 is being packaged now and will allow maas to better support juju with multiple users. but i'm not sure if the symptoms you are seeing exactly relate to the problem as i've not run into the issue myself
  <bigjools> looks different to me
  <adam_g> wallyworld__, ssory, should have mentioned. im using 1.16.2
  <thumper> adam_g: that's weird, not seen that before
  <wallyworld__> adam_g: in that case, my maas foo is not good enough. maybe bigjools  can help
  <adam_g> tried re-bootstrapping again, this time with only one environment. was the same thing
  <adam_g> trying again on another cluster
  <sodre> adam_g: what is exactly the issue you're facing?
  <bigjools> it doesn;t look anything to do with maas to me, there's a mongo connection error
  <sinzui> adam_g, tools are in canonistack and aws, your tests might be different now
  <adam_g> sodre, with two separate maas users, each with their own juju environment, only one bootstraps
  <adam_g> 2013-07-15 00:35:38 DEBUG juju.state open.go:88 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
  <adam_g> 2013-07-15 00:35:38 ERROR juju.state open.go:93 TLS handshake failed: x509: certificate has expired or is not yet valid
  <sodre> is one of them *not admin* ?
  <adam_g> sodre, neither of them are
  <sodre> one sec.
  <adam_g> hmm. ive gotten further on another maas cluster (with the same versions of both juju and maas)
  <sodre> can you see if the workaround in #1245092 works for you ?
  <wallyworld__> sodre: you can revert your rev 110 to remove the json fix using a cherry pick - bzr merge -r 110..109
  <_mup_> Bug #1245092: environ-provisioner fails to start <admin> <bootstrap> <Go MAAS API Library:Triaged> <https://launchpad.net/bugs/1245092>
  <adam_g> sodre, make both users admin?
  <sodre> wallyworld__: okay I'll do that. but at the moment I can't test with my live deployment without the json fix :(
  <sodre> adam_g: or at least one of them and see if that one bootstraps.
  <wallyworld__> you can add the json fix when you tst but just not commit it
  <sinzui> We need to support sync or sum checking with gwacl to upload tools
  <adam_g> sodre, fwiw, i was able to bootstrap 2 environments with non-admin users on another cluster
  <wallyworld__> sinzui: you saying tools upload is unreliable on Azure?
  <sodre> adam_g: okay... then maybe that is a different bug.
  <sinzui> wallyworld__, Iam saying I have been uploading tools for 20 minutes, most of them are already in azure.
  <wallyworld__> ah right
  <sinzui> bigjools, Do to accidents in team organisation, I can create series in charms. Does your tarmac charm really need to be in saucy? Would you be willing to skip to trusty?
  <wallyworld__> i noticed yesterday Azure is really, really sloooooooow
  <sinzui> wallyworld__, tools check sums will change we regenerated, so checksum is a requirement
  <wallyworld__> yeah. we calculate checksums anyway, so makes sense to short circuit upload if possible
  <bigjools> sinzui: it is series-agnostic
  <bigjools> I pushed it to precise for now
  <thumper> axw: morning
  <axw> thumper: ahoy
  <thumper> axw: I'd like to point you at https://bugs.launchpad.net/juju-core/+bug/1246905
  <_mup_> Bug #1246905: status for manual provider hangs <manual-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1246905>
  <axw> looking
  <thumper> axw: I've not dug into it any more than my comment says
  <axw> okey dokey. I'll dig in. thanks
  <axw> thumper: if you have a moment later, I replied to the thread about interfaces... keen to hear if you have any suggestions, as I know you went through something similar not too long ago
  <thumper> axw: heh, you mean that world of hurt?
  <axw> can't remember if you had to deal with this particular problem
  <axw> heh :)
  <adam_g> bigjools, 2013-07-15 00:56:15 ERROR juju.state open.go:93 TLS handshake failed: x509: certificate has expired or is not yet valid  <- any idea?
  <bigjools> adam_g: none at all, sorry.  this looks like a connection problem to mongo, which is nothing to do with maas
  <bigjools> we tested that multi-user worked already so maybe an env issue for you?
  <adam_g> bigjools, yeah, looks lthat way. multi-user stuff is working fine in another environment
  <bigjools> sorry I can't help more :(
  <adam_g> cls
  <adam_g> what  is reponsible for /var/lib/juju/server.pem ?
   * thumper -> school run
  <wallyworld__> adam_g: that is generated by cloud init based on the ca-private-key in the environment settings as far as i know
  <wallyworld__> or ca-cert perhaps
  <sinzui> Damn. I cannot upload public tools to hp
  <sinzui> I can see my test from this afternoon, but now my creds don't work
   * sinzui uploads tools using the gui
  <sinzui> Oh something has cursed me. sync-tools fails, swift fails, js errors using Hp's gui
   * sinzui drinks more
  <adam_g> wallyworld__, cloud-config contains: http://paste.ubuntu.com/6338969/
  <adam_g> mongo is running: usr/bin/mongod --auth --dbpath=/var/lib/juju/db --sslOnNormalPorts --sslPEMKeyFile /var/lib/juju/server.pem --sslPEMKeyPassword xxxxxxx --bind_ip 0.0.0.0 --port 37017 --noprealloc --syslog --smallfiles
  <adam_g> but
  <adam_g> Oct 31 20:21:47 ewing mongod.37017[14138]: Thu Oct 31 20:21:47.025 [conn2326] end connection 10.246.8.3:37271 (0 connections now open)
  <adam_g> Oct 31 20:21:47 ewing mongod.37017[14138]: Thu Oct 31 20:21:47.204 [initandlisten] connection accepted from 127.0.0.1:35146 #2327 (1 connection now open)
  <adam_g> Oct 31 20:21:47 ewing mongod.37017[14138]: Thu Oct 31 20:21:47.205 [conn2327] ERROR: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
  <adam_g> (^ from syslog)
  <adam_g> and
  <adam_g> 2013-10-31 20:22:09 ERROR juju.state open.go:93 TLS handshake failed: x509: certificate has expired or is not yet valid
  <adam_g> cloud-init-output
  <wallyworld__> adam_g: sadly i'm not very familiar with issues pertaining to certificate generation etc.  i have no idea how to debug your issue. if you are around in about 4 hours, the folks from europe will come online and will be able to help a lot more
  <wallyworld__> the mechanics all seem ok, but the content must be wrong and i don't know why
  <sinzui> I think Hp Cloud is in flames
  <sinzui> I cannot eve talk to my deployed env, or create a new env
  <sodre> wallyworld: when you have time, can you help me with the cherry picking. I am getting an error saying revision 110 does not exist
  <wallyworld__> sodre: are you working from the same branch you pushed to lp?
  <wallyworld__>  110. By Patrick Carlos 8 hours ago
  <wallyworld__>     Closes #1246827. GOOSE asks for swift.list to be in json format.
  <_mup_> Bug #1246827: swift List expects JSON output but does not pass format=json in http request <Go OpenStack Exchange:New> <https://launchpad.net/bugs/1246827>
  <sodre> yeah. that is what happened.
  <wallyworld__> in any case, so long as your current copy of the branch only has the fixes for that branch that's all that matters
  <thumper> wallyworld__: you firefighting?
  <wallyworld__> thumper: no, just helping on irc
  <wallyworld__> and doing a review
  <thumper> wallyworld__: work in progress that I'm considering landing ... https://code.launchpad.net/~thumper/juju-core/kvm-container-interface/+merge/193533
  <thumper> start of the kvm bits
  <sodre> okay, maybe I need to make a new branch and check-pick from there.
  <thumper> work I had in a pipeline
  <wallyworld__> thumper: that's a lot of green :-)
  <thumper> sure is
  <thumper> and here you just thought that I sat on my thumb all day
  <wallyworld__> sodre: no need to start again i don't think - the diff looks ok actually, so you must have removed the code
  <wallyworld__> thumper: well......
  <wallyworld__> no comment
  <sodre> yeah, got it :(
  <sodre> :)
  <sodre> the code needs testing with pure Swift like canonistack or hp cloud, just to be safe.
  <wallyworld__> sodre: i tested with canonistack and it's ok
  <wallyworld__> after i made a change
  <wallyworld__> but the unit tests need fixing also
  <sodre> cool. I tested with radosgw and it works as well.
  <wallyworld__> thumper: just started; so for lxc, there's a golxc lib. no gokvm equivalent? also, container/interface.go contains "kvm" in the docstrings
  <thumper> hmm... yes no gokvm
  <thumper> I though we'd just wrap it in the container/kvm bit
  <thumper> it isn't complicated
  <wallyworld__> on the surface, it would be good to implement this consistently
  <sodre> walyworld__: I fixed the unit test. and I've added the StatusNoContent  to requestdata.  let me know if you need anything else.
  <thumper> much simpler interface than lxc
  <wallyworld__> ok
  <wallyworld__> sodre: will do, thanks
  <thumper> the container/lxc or container/kvm is the juju interface to the containers
  <thumper> so in that respect, it is consistent
  <wallyworld__> ok. will read more
  <sinzui> does anyone have an hp cloud envs that they can get the status of now
  <thumper> I'm not going to add a new top level project for the hell of it
  <thumper> not yet, anyway
  <sinzui> oh brilliant, I think someone took away my access to hp while I was deploying
  <thumper> haha
  <sinzui> I am in the console and watching the feature disappear
  <sinzui> CI is DEAD
   * thumper calls it a day a little earlier than normal
  <thumper> given the late night antics
  <thumper> have a good weekend everyone
  <wallyworld__> sodre: because you have this : "-params.Add("format", "json")" in the diff, it looks like someone has already added that change in and you would now be removing it. so add it back toyour branch and re-push
  <sodre> :)
  <wallyworld__> sinzui: well that suck donkey balls :-(
  <wallyworld__> sinzui: want me to try and log in?
  <sinzui> yeah. because someone not me may have just broken all juju deploys. that is to say I had to jack hammer the tools into place and I cannot verofy they work
  <sinzui> wallyworld__, If you bootstrap on HP, do you get 1.16.2
  <wallyworld__> sinzui: i'll look, just a sec
  <sinzui> oh, fuck I still need to copy those packages to the public PPA
  <wallyworld__> sodre: you want to remove 50	+ fmt.Sprintf("%s", bodydata)
  <sodre> ohhh... sorry
  <wallyworld__> np :-)
  <sodre> what do I do with bodydata ?
  <wallyworld__> sodre: well, i this case i think you can ignore it. this is just a test mock and we don't implement container acl, so we just want to return an ok response
  <sodre> alright I'll take away the entire ioutiol.ReadAll
  <wallyworld__> sinzui: my wallyworld login on hp cloud still have access to the accounts for the public bucket and also another account with compute resources etc
  <wallyworld__> sodre: what is the content of the body?
  <wallyworld__> sodre: maybe we want to just assert that looks sensiblw
  <wallyworld__> sinzui: did you want to use that for anything till your access can be sortedout?
  <sodre> wallyworld__: I think it is just the header with X-Container-Read
  <sinzui> wallyworld__, I want confirmation that a bootstrap gets the tools I uploaded via the gui
  <wallyworld__> sinzui: ok, checking now
  <sinzui> wallyworld__, have you deployed using the juju-public-tools region?
  <wallyworld__> sinzui: yes, the required url is baked into juju for 1.16
  <wallyworld__> so no need for tools-url or anything else
  <sinzui> hmm. I was told not to do that
  <wallyworld__> you were told wrong :-)
  <wallyworld__> well, i believe so
  <sinzui> Hey the console lets me see juju-scale test again and I can see my three machines are still up
  <wallyworld__> sinzui: was a reason given?the whole idea of hpcloud in 1.16 was to remove the need for public-bucket-url and tools-url ie make it config free. and that's also in the release notes
  <sinzui> wallyworld__, hey I am in, I have access from the command line again
  <wallyworld__> \o/
  <sinzui> log says it selected 1.16.2 for upgrade
  <wallyworld__> great
  <sinzui> so maybe someone was messing with me or HP  got paralytic for 2 hours
  <sinzui> \o/ upgrade is perfect
  <wallyworld__> whoohoo
  <sinzui> I guess in theory, I could run the script to republish the tools. It will tell me if I did something wrong
  <sodre> wallyworld__: I pushed the code to lp:~psodre/goose/radosgw-acl
  <sodre> now I need to fix the swift-jason branch I just messed up.
  <wallyworld__> sodre: i see what may have happened - i didn't notice the prereq branch. can you please resubmit the mp and leave out the prereq. you can choose to keep history so existing comments are retained
  <wallyworld__> for some reason, the preview diff is now messed up also. hopefully resubmit will help
  <sodre> okay...
  <wallyworld__> sodre: in case you don't know, use the Resubmit proposal link
  <sodre> ahhh
  <sodre> I really need to get used to lp
  <wallyworld__> it's good once you know all the features
  <sodre> sorry, where is the resubmit
  <wallyworld__> top right
  <sodre> on the bug page ?
  <wallyworld__> no, the mp page
  <sodre> got it.
  <sodre> so no prereq.
  <wallyworld__> yep
  <wallyworld__> that's better
  <wallyworld__> except the json line is still there
  <sodre> argh.
  <sodre> how do I see that ?
  <wallyworld__> look t the preview diff
  <sodre> okay okay...
  <wallyworld__> on the mp page
  <wallyworld__> i'd say you added it to get stuff working
  <sodre> I can take it out.
  <wallyworld__> sodre: normally it would be ok to leave in, but because someone else is supposed to have a fix in progress, it's best not to overlap
  <sodre> I am the one assigned the json bug
  <wallyworld__> oh?
  <sodre> yeah, sinzui set it to me.
  <wallyworld__> ok. it was discussed yesterday that someone in our team would start on it. but since you have the bug, go for it
  <wallyworld__> link the bug to your branch
  <wallyworld__> so now you have 2 bugs linked
  <sodre> I had a branch just for that bug
  <sodre> it is already linked.
  <sodre> maybe the best would be to approve/merge that first.
  <sodre> then merge the radosgw-acl
  <wallyworld__> sure. that's the more "correct" way
  <wallyworld__> although for one line... :-)
  <sodre> i know
  <sodre> okay... let me just link the entire branch to both bugs.
  <wallyworld__> since you have the branch, propose it and i'lllook
  <wallyworld__> argh. whatever works :-)
  <sodre> alright its proposed and linked to the bug
  <wallyworld__> sodre: i've approved, thanks. next step is to set the commit message (usually copy the description) and mark as approved. then the landing bot will take over
  <sodre> okay. is that in the Merge Proposal page as well ?
  <sodre> cool
  <wallyworld__> yep
  <wallyworld__> sodre: update the descption to reference both bugs also :-)
  <sodre> is it just a matter of writing #XXXX ?
  <sodre> or is there a special markup ?
  <wallyworld__> If you write bug XXXXX it will be hyperlinked. I think lp:XXXX works also
  <wallyworld__> it's been a while so i can't recall with 100% certainty
  <sodre> alright. all done.
  <wallyworld__> right, so bug #XXXworks also :-)
  <wallyworld__> sodre: i've marked a approved at the top of the mp. it should land in trunk in about 5 minutes
  <sodre> :) that is so coo
  <sodre> cool
  <sodre> once that is done, can I delete the branches ?
  <wallyworld__> you can delete them off your local disk yes. but not on launchpad
  <wallyworld__> you can always get them off lp again if needed
  <sodre> okay.
  <wallyworld__> sodre: merged already :-)
  <wallyworld__> you fixed two bugs. thank you
  <sodre> \o/
  <wallyworld__> now i'll update the dependencies in juju-core so your work gets included in the next release
  <sodre> I see. Do these changes ever make it back to saucy ?
  <wallyworld__> long story. they will go into juju 1.18 (next release). that will be available via a ppa. i don't believe we are allowed to put 1.18 in saucy backports
  <sodre> I see. I know that we had a release today, 1.16.2 ( or something) is that available through PPA as well ?
  <wallyworld__> yes
  <wallyworld__> if not now, then very soon
  <wallyworld__> release smoke testing happening as we speak
  <sodre> alright, I'll just tell people at work to update their juju version.
  <wallyworld__> if they want your fix, they will need to grab trunk and compile themselves
  <wallyworld__> we will be releasing 1.17 as a beta to iron out any issues prior to 1.18
  <sodre> Got it,  compiling juju with go is fairly straightforward as well. so it should not be an issue.
  <wallyworld__> sodre: just got the release email for 1.16.2. ppa is at https://launchpad.net/~juju/+archive/stable
  <sodre> thanks. I'll add that. I was looking forward to the MAAS/lxc fix
  <rogpeppe1> mornin' all
  <axw> morning
  <rogpeppe> axw: you might be interested to have a look at some stuff i've been doing to make it easy (hopefully) to write charms in Go: http://godoc.org/launchpad.net/juju-utils/cmd/gocharm
  <axw> rogpeppe: looks cool :)
  <rogpeppe> axw: thanks
  <axw> I like the thought of having go-gettable "charm helpers"
  <axw> makes them feel a bit more first class
  <rogpeppe> axw: did you see launchpad.net/juju-utils/charmbits/httpserver ?
  <axw> rogpeppe: didn't get that far yet, but I did see it in hte imports
  <rogpeppe> axw: i'm trying to come up with a scheme that lets people build charms modularly
  <rogpeppe> axw: it seems to be working ok so far, although i'm slightly concerned that people might find the abstractions hard to deal with
  <axw> rogpeppe: so that charmbits/httpserver package is used in a charm that is also the application?
  <axw> I mean.. service
  <rogpeppe> axw: yes
  <rogpeppe> axw: the idea is that you can write an entirely self-contained charm in Go if you want
  <axw> yup
  <rogpeppe> axw: the awkward piece being you want to be able to tell upstart to run a piece of your charm as a service and you don't really want to make another binary for that
  <axw> yeah I was just thinking about that.. you haven't got to that part yet? I don't see anything to do with running the binary outside of a hook
  <rogpeppe> axw: actually that bit is done
  <axw> ah yeah
  <axw> I see now.
  <rogpeppe> axw: see http://bazaar.launchpad.net/+branch/juju-utils/view/head:/charmbits/httpserver/server.go#L112
  <rogpeppe> axw: and hook.Context.RegisterCommand
  <rogpeppe> axw: there's a standup in 7 minutes if you fancy coming along BTW
  <axw> rogpeppe: sure, wife's out so I may as well
  <axw> ta
  <fwereade_> mgzh, meeting
  <mgzh> ta
  <rogpeppe> frickin' new hangouts
  <rogpeppe> natefinch: ping
   * fwereade_ has to go to the shop while it's open, bbiab
  <sodre> !on-call
  <rogpeppe>  i like the fact that i can reboot and my local juju environment is still running just fine
  <rogpeppe> i like the fact that i can reboot and my local juju environment is still running just fine
   * rogpeppe gets some lunch
  <natefinch> rogpeppe: let me know when you're back from lunch'
  <abentley> sinzui: When mysql dies for you, does mysql/1 go into an error state?
  <sinzui> abentley, install error
  <abentley> sinzui: For me, it just went to 'error', not 'install-error', so I guess it's because Jenkins only has mem=2G.
  <sinzui> oh
  <natefinch> fwereade_: you around?
  <fwereade_> natefinch, hey dude, sort of
  <fwereade_> natefinch, explaining juju to anold colleague
  <natefinch> fwereade_: heh... possibly quick question (if not, that's ok).  The server API for set was already written, with part of the implementation having this comment:
  <natefinch> / parseSettingsCompatible parses setting strings in a way that is
  <natefinch> / compatible with the behavior before this CL based on the issue
  <natefinch> / http://pad.lv/1194945. Until then setting an option to an empty
  <natefinch> / string caused it to reset to the default value. We now allow
  <natefinch> / empty strings as actual values, but we want to preserve the API
  <natefinch> / behavior.
  <natefinch> fwereade_: this makes the API implementation retain the behavior of setting a value to "" causing the value to get unset.
  <natefinch> fwereade_: The comment makes it seem like that's on purpose... but that just totally breaks the expected functionality of set if we always go through the API, so I'm not really sure what the correct course of action is.
  <fwereade_> natefinch, I think we need parallel implementations for now, that's just a compatibility thing for the gui
  <natefinch> fwereade_: oh, the GUI, ok
  <fwereade_> natefinch, we want to express the cli functionality now
  <fwereade_> natefinch, and *that's* a job for aslightly neater ServiceSet call
  <fwereade_> natefinch, so we have halfachanceofoneday being able toset a bunch ofservice attributes transactionally
  <fwereade_> natefinch, eg, set charm and config at the same time
  <natefinch> fwereade_: so we  need two - the existing one for the GUI and a new one for the CLI
  <fwereade_> natefinch, yes please -- ask frankban if heeverstarted on the newserviceset thing
  <natefinch> fwereade_: cool, thanks for clearing that up
  <rogpeppe> natefinch: ping
  <natefinch> rogpeppe: howdy
  <rogpeppe> natefinch: fancy a chat about HA?
  <natefinch> rogpeppe: definitely
  <rogpeppe> natefinch: we could try G+ again - i've rebooted, so it *may* be ok again
  <natefinch> rogpeppe: might as well give it a second change
  <natefinch> chance
  <rogpeppe> natefinch: https://plus.google.com/hangouts/_/calendar/bWFyay5yYW1tLWNocmlzdGVuc2VuQGNhbm9uaWNhbC5jb20.09gvki7lhmlucq76s2d0lns804?authuser=1
  <fwereade_> bbl
  <sinzui> rogpeppe, CI is failing for juju-core. One reason relates lxc-provisoner has no credentials for the user attribute. The test is not using lxc though. We are starting a caonistack instance. Also note that juju 1.16.0 client got the 1.16.2 agent. Do you have any insights https://pastebin.canonical.com/99724/
  <rogpeppe> sinzui: on a call currently - will get back to you
  <rogpeppe> need to reboot again :-(
  <sinzui> rogpeppe, I can elaborate about the error. I don't think the juju version is the issue because we see the same messages when using 1.16.2. The issue may be that the juju client is in a precise lxc bootstrapping precise on canonistack.
  <sinzui> we don't see this issue running saucy clients/client not in lxc
  <rogpeppe> sinzui: looking now
  <rogpeppe> sinzui: what is the CI actually trying to do?
  <rogpeppe> sinzui: that error message is coming from the openstack provider
  <sinzui> rogpeppe, It is verifying I can deploy a simple stack to canonistack
  <sinzui> rogpeppe, jenkins in canonistack created an lxc to build juju-core tip, then from in the lxc, began bootstrapping canonistack, hp, and aws envs
  <rogpeppe> sinzui: so the bootstrap worked ok?
  <sinzui> rogpeppe, no, bootstrap returned an error
  <rogpeppe> sinzui: hmm, weird, then how come the bootstrap node is working ok?
  <rogpeppe> sinzui: what error did bootstrap return?
  <sinzui> abentley, what error did the bootstrap return for CI of canonistack?
  <rogpeppe> sinzui: so the bootstrap returned an error, but it successfully started an instance running the machine agent? from the error returned, that seems quite odd, as the error should have stopped it bootstrapping before it got anywhere near that stage.
  <sinzui> rogpeppe, me network/ssh is busted. I cannot sshuttle this hour. I think I need to reboot to get into my 3rd test of this scenaro
  <abentley> sinzui, rogpeppe: bootstrap didn't error, but after bootstrap, nothing worked.  Not "status", not "deploy".  It just hangs.
  <abentley> (And occasionally emits ERROR TLS handshake failed: EOF)
  <rogpeppe> abentley: ah, that makes more sense
  <rogpeppe> abentley: could you paste (privately) the contents of the .jenv file for the environment? (elide private keys if you care)
  <abentley> rogpeppe: https://pastebin.canonical.com/99730/
  <abentley> rogpeppe: I can also add your public ssh key to the bootstrap node so you can log in, if you like.
  <rogpeppe> abentley: that would be useful, thanks
  <sinzui> abentley, rogpeppe my third test did eventually let me in. all is good. I will go downgrade to 1.16.0 again
  <abentley> rogpeppe: You should be able to ssh to ubuntu@10.55.60.105 now.
  <rogpeppe> abentley: thanks. does that still have the problem?
  <abentley> rogpeppe: Yes.
  <abentley> sinzui: I am experiencing this issue with saucy.
  <sinzui> abentley, I am not...
  <sinzui> abentley, I downgraded to 1.16.0 on precise in an lxc. I deployed and got the 1.16.2 agent. I has just started. I am now deploying services
  <abentley> sinzui: This issue is not really related to juju CI.  I'm trying to create a jenkins node where I will eventually do CI.  I'm doing this because the existing jenkins node has 2G and that may not be enough.
  <abentley> But this is really just juju bootstrap not leading to to a working environment on canonistack.
  <sinzui> abentley, ha ha, i just hit the max security groups/instances again
  <abentley> sinzui: That was fast!
  <sinzui> abentley, I will tear this down. I don't think we need it since I misunderstood the scenario
  <sinzui> abentley, or maybe no...I think missed the step where I go the the lxc. my machine is now angry that I have an out of date package installed
  <sinzui> also. I need to work from OS X for sometime today. Once again I botched the homebrew pull request. I will never use githubs edit feature again. I will work with the real code and real tests
  <sinzui> now apt is angry. I think I need to do one thing at a time to achieve something
  <rogpeppe> abentley: i can't connect to that machine
  <abentley> rogpeppe: Does it refuse to authenticate you or can you not even connect to it?
  <rogpeppe> abentley: connection timed out
  <abentley> rogpeppe: do you have your .ssh/config set up to allow you to ssh into canonistack machines?
  <abentley> rogpeppe: I just sshed in successfully.
  <rogpeppe> abentley: ah, probably not
  <abentley> rogpeppe: the crucial thing is to have chinstrap act as a proxy.
  <rogpeppe> abentley: sample config?
  <abentley> rogpeppe: https://pastebin.canonical.com/99741/
  <rogpeppe> abentley: bingo!
   * sinzui interesting. githubs dimensions aren't broken in safari. Just chromium
  <rogpeppe> abentley: is the environment working now?
  <rogpeppe> abentley: i think i know what your problem was
  <abentley> rogpeppe: Yes, it's working now.
  <rogpeppe> abentley: ha
  <rogpeppe> abentley: so there's an easy fix:
  <rogpeppe> abentley: run juju status
  <abentley> rogpeppe: But juju status doesn't work.
  <abentley> rogpeppe: I does now, but it didn't.
  <abentley> s/I does/It does/
  <rogpeppe> abentley: really? it worked for me when i copied your jenv file to your instance and ran juju status
  <rogpeppe> abentley: and that caused the secrets to be pushed to the environment which caused everything to start working
  <abentley> <abentley> sinzui, rogpeppe: bootstrap didn't error, but after bootstrap, nothing worked.  Not "status", not "deploy".  It just hangs.
  <abentley> (And occasionally emits ERROR TLS handshake failed: EOF)
  <rogpeppe> abentley: hmm, i wonder if that's an addressing problem
  <rogpeppe> abentley: where were you trying to run juju status from?
  <abentley> rogpeppe: IME, that's what you get when the bootstrap node isn't responsing to API requests even at the http level.
  <abentley> rogpeppe: I was running status from this laptop I'm typing on.
  <rogpeppe> abentley: the bootstrap node was responding to API requests - i have a feeling your juju client just had no connectivity to the bootstrap node
  <rogpeppe> abentley: but... you say it works now?
  <abentley> rogpeppe: IME when there's no connectivity, nothing happens at all, not " ERROR TLS handshake failed: EOF"
  <abentley> rogpeppe: I did restart sshuttle just now.
  <rogpeppe> abentley: i wonder if was an issue with that
  <abentley> rogpeppe: I guess it's possible.  I took the "ERROR TLS handshake failed: EOF" messages as an indication that the sshuttle tunnel was working.
  <rogpeppe> abentley: basically, the API server seemed to be running just fine, which makes me think that the problem was elsewhere.
  <rogpeppe> abentley: if the tunnel was playing up, you might easily get an error like that (which is from the TLS negotiation taking place *within* the ssh tunnel)
  <abentley> rogpeppe: When I restarted, I switch to using 10.55.60.105 itself as the tunnel endpoint.  I had previously been using 10.55.32.48.  Normally, that's not a problem, but these machines are in different regions.
  <abentley> 10.55.60.105 is lcy01, 10.55.32.48 is lcy02
  <rogpeppe> abentley: ah, so that was the problem?
  <abentley> rogpeppe: Yes, I had the same symptoms when I switched back to 10.55.32.48
  <rogpeppe> abentley: great.
  <rogpeppe> abentley: if you reported a bug, i guess we can mark it as invalid...
  <rogpeppe> abentley: we should probably fix things so that the provisioner doesn't constantly fall over while waiting for a valid environment, i guess
  <rogpeppe> abentley: actually, that should be fixed in trunk already
  <abentley> rogpeppe: I didn't file a bug, but the whole episode only increases the importance of bug #1224057 to me.
  <_mup_> Bug #1224057: Juju needs more than SSH access <addressability> <canonistack> <improvement> <ssh> <juju-core:Triaged> <https://launchpad.net/bugs/1224057>
  <rogpeppe> fwereade_: current status of my HA discussion with nate: https://docs.google.com/a/canonical.com/document/d/1hGa2PNz6JyGFc7FjA-gX2_UnXRUGAZbS8O9TiS51bl8/edit
  <rogpeppe> fwereade_: you'll probably want to bikeshed a bit on names etc
  <abentley> sinzui: Has there been a bug reported that 1.16.0 deploys agent 1.16.2?
   * sinzui no
   * sinzui abentley: I know version mismatch is discussed in bugs, but I don't see a specific bug about the mismatch
  <rogpeppe> right, that's me done
  <rogpeppe> happy weekends all
  <abentley> rogpeppe: thanks!
  <abentley> sinzui: Bug #1247232
  <_mup_> Bug #1247232: Juju 1.16.0 deploys agent version 1.16.2 <juju-core:Triaged> <https://launchpad.net/bugs/1247232>
  <sinzui> thank you abentley
  <hazmat> jam, how do we trigger the support for untrusted certs?
  <jam> hazmat: set "ssl-hostname-verification: false" in the environment config.
  <hazmat> ah.. found it. ssl-hostname-verification
  <hazmat> jam, thanks
  <jam> rogpeppe1: if you're around https://codereview.appspot.com/20940043
  <jam> It turns out that Uniter didn't share the common code that you fixed to use the cached addresses
  <rogpeppe1> jam: oops. looking.
  <jam> rogpeppe1: not really your fault, the code *should* have been shared
  <abentley> sinzui: I am seeing a bug deploying wordpress on the local provider (on precise): https://pastebin.canonical.com/99751/
  <rogpeppe1> jam: reviewed
  <jam> rogpeppe1: it doesn't include CACert because Uniter doesn't use it
  <jam> Uniter grabs APIAddresses to pass to the hook context
  <jam> but it doesn't give them the cert
  <rogpeppe1> jam: hmm, maybe it should
  <rogpeppe1> jam: i think we *should* pass the cert to the hook context (i think there's an existing issue about that)
  <rogpeppe1> jam: so i think it's worth providing in the API so we can easily do that later
  <rogpeppe1> jam: but don't bother if it's a hassle
  <jam> rogpeppe1: when it comes to that point, then we can easily move the function from Addresser to APIAddresser, I'd like to keep the change focused
  <rogpeppe1> jam: ok
  <jam> rogpeppe1: I'm not worried (particularly much) about the memory overhead. Just that every place that used to include Addresser now has to grow 2 lines to do the same thing.
  <rogpeppe1> jam: there are only two such places AFAICS
  <abentley> sinzui: from cgroup-lite.log: https://pastebin.canonical.com/99753/
  <rogpeppe1> jam: i kinda think it would make both of those places more readable (more explicit that they make both state and API addresses available)
  <rogpeppe1> jam: but i don't mind much - i just had to think quite hard when reading the code in common
  <jam> rogpeppe1: do you think they should be renamed to StateAddresser and APIAddresser then ?
  <abentley> sinzui: The problem appears to be cgroups-lite being upgraded.  If I remove the package manually, the installs complete successfuly.
  <rogpeppe1> jam: that would be inclination, yes
  <rogpeppe1> s/be/be my/
  <abentley> sinzui: It affected both mysql and wordpress.
  <abentley> sinzui: I suspect removing apt-get upgrade from the install scripts would also fix it.
  <sinzui> abentley: I am still swimming in the ramifications of this news
  <sinzui> abentley: Does this relate to cloud-archive?
  <abentley> sinzui: I don't know if it's packaged there.
  <abentley> sinzui: The cloud archive is an apt source in the lxc containers.
  <sinzui> I see
  <abentley> sinzui: I do not see groups-lite in http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/c/
   * sinzui yep, that is where I am
  <abentley> s/groups-lite/cgroups-lite/
  <sinzui> I wonder if a dep in the archive uses it though
  <abentley> sinzui: I bet it's installed in the cloud images, but not an up-to-date version.
  <abentley> sinzui: It was attempting to upgrade it, not to install it.
  #juju-dev 2013-11-03
  <davecheney> wowo, google compute engine is really janky
  <davecheney> shithouse first user experience
  <davecheney> Project '{{deleted.name}}' is now scheduled to be deleted after {{deleted.date|date:'short'}
  <davecheney> ^ awesome
  <davecheney> Error
  <davecheney> API rate limit exceeded. Rate limit may take several minutes to update if Google Compute Engine has just been enabled, or if this is the first time you use Google Compute Engine.
  <davecheney> this is not a consumer service
  <davecheney> Creating instance "bootstrap", this may take a few minutes. Do not refresh or navigate away from this page.
  <davecheney> wow, thanks 90's
  <davecheney> % juju switch "null"
  <davecheney> ERROR "null" is not a name of an existing defined environment
  <davecheney> ^ the docs say to do this
  <davecheney> ignore
  <davecheney> copy and paste error
  <davecheney> https://bugs.launchpad.net/juju-core/+bug/1247688
  <_mup_> Bug #1247688: manual provisioning fails on debian wheezy host <juju-core:New> <https://launchpad.net/bugs/1247688>
  <davecheney> W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/raring-backports/multiverse/source/Sources  404  Not Found [IP: 202.158.214.106 80]
  <davecheney> what the f is going on with the mirrors this morning
  <davecheney> http://paste.ubuntu.com/6355925/
  <davecheney> bootstrap is fucked, AGIAN
  <davecheney> has anyone been able to use saucy ?
  <davecheney> i can't bootstrap any saucy series charms
  <wallyworld_> davecheney: the metadata you have uploaded for tools does not contain tools for saucy
  <wallyworld_> you may need to re-generate and upload
  <davecheney> well shit
  <davecheney> wallyworld_: i did bootstrap --upload-tools
  <wallyworld_> what was the client?
  <wallyworld_> series
  <wallyworld_> the metadata has
  <wallyworld_>             "products": [
  <wallyworld_>                 "com.ubuntu.juju:12.04:amd64",
  <wallyworld_>                 "com.ubuntu.juju:13.04:amd64"
  <wallyworld_>             ]
  <wallyworld_> so raring and precise
  <davecheney> raring client
  <wallyworld_> right
  <wallyworld_> so a araring client doesn't upload tools for saucy
  <wallyworld_> there is a work around
  <davecheney> wallyworld_: thanks, i'll just deploy local:raring/ubuntu
  <wallyworld_> ok. sorry about confusion
  <davecheney> nah, it's ok
  <davecheney> this is only a problem in the dev version
  <wallyworld_> yeah
  <davecheney> AWWW FUCK
  <davecheney> now my provisioner is broken because it can't ever get past that fauly saucy machine to provision my raring machine
  #juju-dev 2014-10-27
  <fwereade> wallyworld, anastasiamac: if I wake up tomorrow with even a cursory pre-review of http://reviews.vapour.ws/r/263/diff/ I will be most grateful
   * fwereade bed right now
  <wallyworld> ok, will do
  <fwereade> <3
  <wallyworld> good night :-)
  <anastasiamac> fwereade: k :-)
  <bigjools> hey folks, what's the quickest you've seen an LXC come up, given its image is already cached?
  <stokachu> bigjools: 32 seconds
  <bigjools> stokachu: precisely?  lol
  <stokachu> bigjools: yea ive got a counter
  <bigjools> that's fairly swift, thanks
  <stokachu> thats using ubuntu-cloud template btw
  <stokachu> regular ubuntu is slightly longer
  <bigjools> we're trying to work out if it's a problem when juju uses maas to make LXCs if the DNS entry takes up to a minute to appear
  <stokachu> and with lxc-clone enabled
  <stokachu> bigjools: im actually testing juju+maas 1.7 ill let you know if i see anything like that
  <bigjools> stokachu: well this feature is not present yetr
  <stokachu> ah ok
  <bigjools> we're working on it now, but trying to get a feel for what will be a problem
  <bigjools> we're having to make some compromises
  <stokachu> cool, understood
  <bigjools> https://code.launchpad.net/~gmb/maas/add-ptr-records-bug-1382190/+merge/239493 if you're interested
  <stokachu> nice will take a look
  <bigjools> stokachu: wait, are you working on your Sunday?!
  <stokachu> bigjools: yea :( trying to get our unmanaged installer ready for tuesday's lds release
  <stokachu> so we're testing daily maas 1.7 right now
  <bigjools> stokachu: ah ok. LXCs don't get DNS until that branch lands :)
  <stokachu> good to know, fortunately we're using KVM for our bootstrap node in maas
  <stokachu> so it works well
  <bigjools> stokachu: ok so what would be good to know is if a delay getting the DNS entry up after the LXC appears will be a problem in practice
  <stokachu> bigjools: ok cool, i subscribed the team to that bug and added a note on our tasklist to keep an eye on that
  <bigjools> great thanks
  <stokachu> np
  <anastasiamac> axw, wallyworld: isLoopback is great idea but it does not work for explicit "localhost"...
  <axw> anastasiamac: no, you'd still need a string comparison for that
  <axw> you're just cutting out the comparison to 127.0.0.1 and ::1
  <anastasiamac> axw: k... m with u :-)
  <anastasiamac> axw: it looks like it does not cater for ip where hostport is only ::1 either...
  <anastasiamac> axw: to be fair, I suspect that it's url.parse that does not cater for this case...
  <anastasiamac> axw: just how it does not cater for malformed urls...
  <axw> anastasiamac: ah, no, you're right, SplitHostPort expects there to be a port... in that case if you get an error you could tack on a ":0" and try again. starting to get a little convoluted...
  <anastasiamac> axw: hmm... do u think that mayb regular experession could work neater? it will abbrv8 the if- and is easier to modify to cater for "funny" cases...
  <axw> anastasiamac: actually, is the port even optional? I'm not sure there is a default port for apt proxies?
  <anastasiamac> axw: from apt.conf man page... "  http
  <anastasiamac>        HTTP URIs; http::Proxy is the default http proxy to use. It is in
  <anastasiamac>        the standard form of http://[[user][:pass]@]host[:port]/. Per host
  <anastasiamac>        proxies can also be specified by using the form http::Proxy::<host>
  <anastasiamac>        with the special keyword DIRECT meaning to use no proxies. If no
  <anastasiamac>        one of the above settings is specified, http_proxy environment
  <anastasiamac>        variable will be used."
  <axw> ok
  <anastasiamac> axw: it looks like port could be optional... unless square brakcets have other meaninng...
  <axw> nope, that would mean optiona
  <axw> l
  <anastasiamac> axw: ;-(
  <anastasiamac> i'll do a regexp for now... we can alsways neat it up once url.parse and net.IP.isLoopback cater for every possibility... :-)
  <axw> okey dokey
  <anastasiamac> axw: i do prefer to call on framework rathern than rely on regexp
  <anastasiamac> axw: thnx for pointing me in the right (i.e. proper) direction! :-)
  <axw> nps
  <bigjools> folks, when juju destroys an environment, does it keep issuing commands to destroy individual machines if it later notices that one is not in the state it expected (ie getting destroyed)?  This might be provider-specific, in which case I'm talking about maas.
  <bigjools> axw: you worked on the maas provider recently :) --^
  <axw> bigjools: I'll check what it does
  <bigjools> cheers
  <axw> I think it issues a single release call
  <axw> bigjools: http://pad.lv/1319016
  <axw> it does a single release API call
  <axw> bigjools: why do you ask?
  <bigjools> axw: because https://bugs.launchpad.net/maas/+bug/1381619
  <mup> Bug #1381619: Failed to destroy-environment when node is in commissioning or new state <cloud-installer> <oil> <juju-core:Triaged> <MAAS:Triaged> <https://launchpad.net/bugs/1381619>
  <bigjools> axw: ignore the bug title, look at what greg posted
  <bigjools> I'll need to explain some background perhaps:
  <bigjools> we added some more states. It used to go from ALLOCATED straight to READY as soon as an API call was issued to release the node, however now it goes via a RELEASING state
  <bigjools> but because this broke some API users, we folded RELEASING into ALLOCATED so that API users never see it
  <bigjools> I wondered if juju was issuing a release call, and still seeing ALLOCATED when in reality it's RELEASING, so juju issues another release?
  <axw> bigjools: ah ok. nope, we don't interpret the error or retry at the moment
  <bigjools> oh ,weird then
  <axw> good to know for the bug I mentioned tho...
  <bigjools> yeah
  <axw> I'll paste that in there
  <bigjools> axw: how does it know when the env is destroyed?  Or does it just fire and forget?
  <axw> bigjools: it'll do do a release of all nodes, and check if it passed/failed based on HTTP status code only
  <bigjools> ok
  <bigjools> hmmm
  <bigjools> that bug makes no sense then, unless people were mannually titting about with things first
  <axw> bigjools: hmm. actually, Bootstrap will do a release of the bootstrap node if it manages to acquire it and then something goes wrong
  <axw> then when the env is destroyed, it'll try to release it again
  <bigjools> ah
  <axw> I *think* anyway
  <bigjools> axw: so releasing twice?
  <axw> why did the instance stop showing up tho? i.e. why "ERROR bootstrap failed: refreshing addresses: no instances found
  <axw> Stopping instance..."
  <bigjools> NFI what was going on there
  <bigjools> it's a shame people don't report what actions they were taking, and just post log snippets
  <axw> bigjools: actually no, that error message greg pasted is from the first attempt to release
  <axw> there may or may not be a second attempt, depending on what MAAS returns to Juju when it tries to list nodes with agent-name later, but I don't think it's relevant
  <bigjools> axw: looking at the event log, it did fail while coming up
  <bigjools> well
  <bigjools> something caused bootstrap to release the node while it was deploying
  <axw> (the second attempt would occur after that error is listed)
  <bigjools> I asked greg to file another bug with more info
  <axw> wallyworld: can you please take a look at http://reviews.vapour.ws/r/137/diff/2-3/
  <wallyworld> sure, justing finishing a review, will look real soon
  <axw> there's a drive by fix to ssh/run_test.go, which failed when I tried to land my change once
  <axw> yep no rush
  <axw> the main change is that I've made the metadata commands use environs.New, rather than environs.Prepare; i.e. they require an existing env now
  <wallyworld> axw: i think the changes are ok - any env should already be existing when using those metadata commands
  <axw> wallyworld: thanks. utils/ssh is used by testing, so can't use the existing ShortWait/LongWait
  <wallyworld> ah ok
   * wallyworld -> get more coffee, it's an emergency
  <axw> wallyworld: I've removed myself from that credential bug and moved it back to the planning lane. azure and joyent still need doing.
  <wallyworld> axw: ok, np
  <mattyw> morning everyone
  <wallyworld> axw: if you get a chance before your EOD, cloud you look at http://reviews.vapour.ws/r/264/ ? it's a large number of file but 99% of the changes are mechanical - the tools stream needs to be added as a parameter to many test methods
  <axw> wallyworld: ok
  <wallyworld> ty, if you don't get to it, that;s fine
  <axw> nah I need a break from storage, will look now
  <voidspace> morning all
  <lazyPower> o/ morning core devs
  <wallyworld> axw: thanks for review :-) i prefer to use string literals in tests to cause breakage when things change, so that tests are thought about
  <jam> morning voidspace
  <jam> voidspace: did you see: https://github.com/juju/juju/pull/962 wallyworld wants to cache your lookup, which I think we were talking about doing a different way
  <jam> ah, read it wrong, you proposed it :)
  <jam> the email comes as "no-reply" and then is responded by "ian" so the mail preview always gives me the wrong impression
   * wallyworld doesn't want to cache the lookup - the pr implements a cache and i had a query about it
  <jam> wallyworld: we can't change vpc at all for the lifetime of an environment
  <axw> wallyworld: ok
  <jam> wallyworld: instances in 1 vpc can't communicate via private network to instances in another vpc
  <jam> wallyworld: so the "lifetime of juju process" is a misnomer
  <jam> it is "lifetime of a juju environment"
  <wallyworld> jam: ok, thanks. the pr said the vpc *could* change
  <wallyworld> it was unclear if that would be in the lifetime of an env or not
  <jam> wallyworld: first thoughts were that it could, further thinking realized "no we can't actually support that"
  <wallyworld> ok, np. i think then my only concern (from memory) is that a muctex should be used
  <wallyworld> mutex
  <voidspace> yeah, my mistake in the description
  <voidspace> the vpc changing would be "bad" for an environment - but it isn't likely to happen during the lifetime of a process anyway
   * TheMue wonders about the Google Calendar. It shows the correct time now after switching back from DST, but the alerts are for a false time?!?!?!
  <voidspace> wallyworld: I'll add the mutex
  <wallyworld> voidspace: ok, thanks, that then conforms to existing practice
  <voidspace> wallyworld: that's in code already committed - it was just missed in the review of that code
  <voidspace> easy to add it now
  <wallyworld> voidspace: np. i also had a question about one of the tests
  <voidspace> wallyworld: yes, I just answered that
  <voidspace> wallyworld: the test is the same because I want to *prove* that cached value is used
  <voidspace> wallyworld: so even after changing what the api would return I expect to get the previous value back
  <wallyworld> ah, let me take another look
  <voidspace> wallyworld: proving that a second api call wasn't made
  <voidspace> wallyworld: that's the intent
  <wallyworld> voidspace: i may be dumb, but from what i can tell, the code could still make an api call and return those result values and make the test pass. the more correct approach would be to mock out the api call and put in an assert that it is not called
  <jam> TheMue: you didn't get your reminder for our 1:1? I didn't see you around, I should have pinged, I guess
  <voidspace> wallyworld: if it made an api call it would get the opposite result
  <voidspace> wallyworld: if you remove the caching and run the test it fails
  <voidspace> wallyworld: I did check that
  <voidspace> wallyworld: also, with the mutex
  <wallyworld> voidspace: ok, then i was being dumb :-)
  <TheMue> jam: I got it, for in 18 minutes. And later I thought that it would conflict with our standup.
  <voidspace> wallyworld: there is a mutex in the call to e.ec2()
  <voidspace> wallyworld: the common pattern in ec2.go where e.ec2() is used seems to be *not* to use an extra mutex
  <TheMue> jam: but no I got the alert for the standup, also one hour too late.
  <wallyworld> voidspace: ok, np. i missed that nuance
  <jam> TheMue: are these email alarms or popup ones?
  <TheMue> jam: email
  <voidspace> wallyworld: the mutex in use seems to be a config mutex - preventing the ec2 configuration changing whilst we're building the ec2 instance
  <voidspace> although it doesn't prevent it changing whilst we're using it :-/
  <wallyworld> voidspace: yeah, that was my worry
  <wallyworld> voidspace: anyway, that's a separate issue it seems, so i marked it lgtm
  <voidspace> wallyworld: locking every use would serialise all our api calls though
  <voidspace> wallyworld: awesome, thanks
  <wallyworld> np, thanks for putting up with my questions
  <voidspace> heh :-)
  <TheMue> jam: but simply won't care for them in the future now as I now it :D and will be at standup at the correct time
  <voidspace> wallyworld: I'll change the PR description before merging, so the comment about default-vpc is correct
  <wallyworld> awesome, ty
  <fwereade> wallyworld, thanks for the review
  <fwereade> wallyworld, lots of that stuff is preexisting, but now someone's complained about them I can muster up the emotinal strength to work on some of them a bit ;p
  <wallyworld> fwereade: np, it looks great but as i said, i am unfamiliar with the finer detail
  <fwereade> wallyworld, it needs more steps before it's really approaching properly sane tbh
  <wallyworld> fwereade: yeah, i figured as much, hard to tell what was new vs moved
  <fwereade> wallyworld, that's the trouble
  <wallyworld> eat an elephant one bite at a time
  <fwereade> wallyworld, exactly
  <fwereade> wallyworld, and I've only really been going on this path for a week and it's already feeling noticeably better
  <wallyworld> yep :-)
  <fwereade> wallyworld, actions needs a bunch of work too
  <wallyworld> if it all works, i reckon commit what you've done so far
  <wallyworld> so it doesn't bit rot
  <fwereade> wallyworld, well, it's worth another pass to fix your suggestions
  <wallyworld> ok
  <wallyworld> they were not mandatory
  <wallyworld> if the code was existing
  <wallyworld> but would be nice to see them fixed sometime
  <fwereade> wallyworld, yeah, I reserve the right to ignore the ones that turn out to be rabbit holes
  <fwereade> wallyworld, at least for this branch
  <wallyworld> yep, np there
  <jam> TheMue: standup ?
  <fwereade> wallyworld, but I am very firmly trying to steer a happy course between fix-only-what's-needed and FIX-ALL-THE-THINGS, cos the former leads where we are today and the latter stops me doing anything directly
  <wallyworld> fwereade: +100
  <TheMue> jam: coming, my clock say :59 ;)
  <fwereade> wallyworld, just pushed a couple of updates, would you give it a quick once-over? I didn't fix the yucky docstring because whenever I start trying to edit that file I start pulling on loose threads everywhere and get sucked in
  <wallyworld> sure
  <wallyworld> fwereade: changes look sound
  <fwereade> wallyworld, awesome
  <fwereade> wallyworld, is that a ship-it, then? :)
  <wallyworld> fwereade: if you are happy, yeah :-) as i said, take my +1 with a little skepticism :-)
  <wallyworld> but i like the direction it is heading very much
  <fwereade> wallyworld, cool
  <fwereade> wallyworld, I'm making a big effort to avoid changing functionality
  <fwereade> wallyworld, not that it always works, I just realised that I broke relation settings caching in one of the previous branches, so that's th enext one before I can fix actions
  <wallyworld> yep, tests will tell us :-)
  <hazmat> something in 1.21alpha2 seems to have broken the allwatcher
  <hazmat> its no longer reporting all the services in an environment
  <perrito666> morning
  <natefinch> morning
  <mattyw> tasdomas, as OCR could you take a look at this http://reviews.vapour.ws/r/265/?
   * perrito666 tries to convince his ubuntu that it can upgrade to a new version
  <voidspace> jam: so I haven't yet found the neutron api for assigning a private p address
  <voidspace> jam: one example I saw created a port to reserve the address
  <voidspace> http://blog.felipe-alfaro.com/2014/05/09/fixed-ip-addresses-with-openstack-neutron-for-tenant-networks/
  <voidspace> I'm not sure a port is quite what we want
  <voidspace> https://wiki.openstack.org/wiki/Neutron/APIv2-specification#Create_Port
  <voidspace> The APIv2 spec doesn't seem to have an alternative api though
  <voidspace> I'll look and see if dimiter has written about this in the networking spec
  <jam> voidspace: I'm back in our 1:1 hangout
  <voidspace> jam: ok
  <jam> I found "UpdatePort" which I think is what we want
  <voidspace> jam: except the addresses you provide overwrite the existing ones
  <jam> voidspace: yeah
  <voidspace> jam: so effectively that would be manual management
  <jam> I don't see a way to "request a new one"
  <jam> voidspace: I'm seriously considering manual management
  <jam> voidspace: as that gives us a token
  <jam> "this IP address that I'm assigning is going to go to this LXC"
  <perrito666> I would really appreciate a review on this http://reviews.vapour.ws/r/238/ looks big but its not
  <tasdomas> mattyw, reviewed your PR - looks good
  <mattyw> tasdomas, thanks very much
  <TheMue> Hmm, did something changed on runTransaction()? My so far working migration now aborts since I merged master.
  <TheMue> Sadly it doesn't says why it is aborting.
  <fwereade> perrito666, LGTM with trivials
  <fwereade> perrito666, good change, thanks
  <perrito666> fwereade: tx for the review
  <fwereade> jam, can I send you into #juju-gui to have a talk with frankban about something networky-looking please?
   * perrito666 visists a friend company for coworking for the day and he gets his space bar fixed
  <fwereade> jam, no public networks from the AllWatcher: http://pastebin.ubuntu.com/8703311/ (although I see a localhost with public scope..?)
  <frankban> fwereade: yeah, but no ipv4 address with public scope
  <fwereade> frankban, indeed, but localhost doesn't helpmost people very much ;p
  <frankban> fwereade: yeah, and this breaks local quickstart usage
  <fwereade> frankban, I'm sorry I can't give you my full attention right now -- dimitern may be on a swap day? otherwise jam or TheMue are most likely to have something relevant to say about networking
  <fwereade> frankban, it sounds like a critical bug regardless, please go ahead and report it
  <frankban> fwereade: this seems to be part of a more general problem with the mega-watchers: hazmat filed bug 1386143
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Confirmed> <https://launchpad.net/bugs/1386143>
  <fwereade> frankban, ok, cool, would you add it to that bug then (assuming status gives you what you want, but the watcher is broken?)
  <frankban> fwereade: sure, and thanks
  <fwereade> waigani, heads up: ^^ I think you/menn0/thumper have been in state lately?
  <jw4> fwereade: morning; were you saying you were going to do a bunch of refactoring in actions?
  <fwereade> jw4, have you been seeing the changes going by in uniter?
  <fwereade> jw4, it's an extension of that really -- trying to move context/runner related stuff out of uniter
  <jw4> fwereade, ah, you mean the hook context stuff?  I haven't caught up the last 24 hours yet :)
  <jw4> fwereade: sure  I remember now
  <fwereade> jw4, in particular, I think the validation stuff needs to move into the context package
  <jw4> fwereade: kk
  <fwereade> jw4, and the RunActions stuff needs at least *some* explicit testing beyond "RunAction fails on a non-action context"
  <fwereade> jw4, but, I'm getting there
  <fwereade> jw4, I'm 90% focused on structural stuff and testability
  <jw4> fwereade: you're on a tear... did you rest this weekend at all?
  <fwereade> jw4, I just managed to ignore everything else and focus on code for the week :)
  <fwereade> jw4, the weekend was mostly relaxed
  <jw4> fwereade: lol... good
  <fwereade> jw4, once I've got a load of the uniter stuff out of the way I should be able to rework the modes into an explicit hook queue
  <fwereade> jw4, it's what I "should" be doing right now but I need to clear the underbrush first iyswim
  <jw4> fwereade: yep, yep
  <jam> frankban: is this on EC2? And do you have default-vpc available there?
  <jam> 172.* looks suspiciously like the cloud-local addresses EC2 likes to give out
  <jam> ah, I guess this was a local environ
  <jam> so not ec2
  <jam> voidspace: since you're on at this time, do you have a chance to look at the networking part of bug #1386143 ?
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Confirmed> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <frankban> jam: the addresses problem is on a local env, yes
  <jam> frankban: so I think the issue is that the 10.* address *is* the public address on local
  <frankban> jam: exactly, and it was previously
  <voidspace> jam: sure
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1386204
  <sinzui> natefinch, jam, can you find someone to fix a test that fails when we increment to alpha3? bug 1386204
  <mup> Bug #1386204:  metadataSuite.TestAsJSONBuffer fails when version is alpha3 <ci> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1386204>
  <mgz> eric wrote the test, but should be pretty simple for anyone to fix up
  <jam> voidspace: so I'm guessing how we determine public vs private cahnged,
  <voidspace> jam: ok
  <voidspace> *looks* like it should be easy...
  <voidspace> a backup test
  <voidspace> jam: ah, I thought you were referring to 1386204
  <voidspace> sinzui: jam: 1386204 we can fix by formatting the current version into the test data
  <voidspace> currently it's hardcoded and will fail every time the version changes
  <voidspace> sinzui: jam: I'll propose a branch for that
  <sinzui> thank you voidspace
  <sinzui> voidspace, you might want to merge https://github.com/juju/juju/pull/970 to get the version change too.
  <perrito666> natefinch: ericsnow wwitzel3 getting there
  <voidspace> sinzui: I have a branch that works with trunk (trivial) and also passes with the version changed manually
  <voidspace> sinzui: I'll just propose it separately
  <sinzui> thank you voidspace
  <voidspace> https://github.com/juju/juju/pull/971/
  <voidspace> There's no reviewboard url yet
  <voidspace> natefinch: care for a (hopefully) easy review?
  <mgz> voidspace: seems fine to me
  <voidspace> mgz: cool, will this merge or do I need to force?
   * voidspace is grabbing coffee
  <mgz> it is the blocking bug, so you just need to mark as fixes-NNN
  <perrito666> ericsnow: natefinch wwitzel3 the call got unusable
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1386204 1385289
  <sinzui> natefinch, jam I have escalated Bug #1385289 because the last 8 devel revisions tested failed because upgrades consistently timeout on all providers...including stable and devel maas
  <mup> Bug #1385289: local storage migration is very slow <ci> <regression> <test-failure> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1385289>
  <natefinch> perrito666: np
  <voidspace> ericsnow: I didn't see a reviewboard review created for this PR
  <voidspace> ericsnow: https://github.com/juju/juju/pull/971
  <ericsnow> voidspace: checking
  <ericsnow> voidspace: looks like github is rate-limiting the requests
  <ericsnow> voidspace: I'll see if I can work around that better (may not happen immediately)
  <voidspace> ericsnow: does it just drop them you think?
  <voidspace> ericsnow: ok, np
  <ericsnow> voidspace: I'm thinking it drops requests ("API rate limit exceeded ...")
  <ericsnow> voidspace: I should be able to work around that though
  <ericsnow> voidspace: did it not create a review request at all?
  <voidspace> ericsnow: as far as I can tell not
  <voidspace> sinzui: how can tell which revision was used for the 1.21alpha2 release?
  <voidspace> sinzui: looking for the version number change in the commit history?
  <ericsnow> voidspace: bummer (I would have expected it to make one)
  <ericsnow> voidspace: in the meantime you can use rbt post
  <voidspace> ericsnow: no need on this occasion, but thanks
  <ericsnow> voidspace: k
  <sinzui> voidspace, https://github.com/juju/juju/releases
  <sinzui> voidspace, reports.vapour.ws will one day republish that ^ info
  <voidspace> sinzui: ah, cool - thanks
  <voidspace> I checked my tags in my local clone, but I don't have them all
  <voidspace> maybe this is a git thing
  <voidspace> I have the major release tags, but not the alphas
  <voidspace> well, minor release tags
  <voidspace> ah
  <voidspace> git pull upstream master  --tags
  <voidspace> now I have them
  <voidspace> sinzui: my fix for issue 1386204 landed
  <voidspace> sinzui: I've marked it as fix committed
  <sinzui> voidspace, rock
  <TheMue> Gna, first needed to find why PR failed (overlapping change regarding DocID) and now the fixes block it. :)
  <voidspace> Is anyone else looking at bug 1386143
  <voidspace> https://bugs.launchpad.net/juju-core/+bug/1386143
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Triaged> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <voidspace> frankban: ping
  <voidspace> now do I get the megawatcher output?
  <voidspace> Bug report above states "Probably a side effect of this regression: on local environments, the mega-watcher for machines no longer reports the ipv4 address in the public scope"
  <frankban> voidspace: pong
  <voidspace> frankban: hey
  <voidspace> frankban: see above
  <voidspace> frankban: I'd like repro instructions for getting the mega-watcher log
  <voidspace> frankban: as you reported for bug 1386143
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Triaged> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <frankban> voidspace: ok
  <frankban> voidspace: do you have quickstart installed? otherwise I can quickly write a script to see the mega-watcher output
  <voidspace> frankban: I don't have it installed no
  <voidspace> frankban: I'm happy to install it if it helps
  <voidspace> is it just from juju-plugins?
  <frankban> voidspace: no, ok I am adding dupe instraction re quickstart to the bug
  <voidspace> frankban: cool, thanks
  <voidspace> Ok, I now have juju-quickstart installed
  <voidspace> frankban: so, running juju-quickstart dies with:
  <voidspace> http://pastebin.ubuntu.com/8706138/
  <voidspace> frankban: is this the error you're getting?
  <voidspace> I'd still like to be able to see the megawatcher output directly
  <frankban> voidspace: no, this is a very old version of quickstart
  <voidspace> frankban: hah, ok
  <voidspace> I need to add the right ppa then
  <voidspace> I assumed I had it already
  <voidspace> frankban: is the version in the juju-stable ppa correct?
  <frankban> voidspace: yeah, added instructions for that too in https://bugs.launchpad.net/juju-core/+bug/1386143
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Triaged> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <voidspace> or should I get from source
  <voidspace> frankban: thanks
  <frankban> voidspace: did not mention it, but the current version in the PPA is juju quickstart v1.4.4
  <frankban> voidspace: FWIW sometimes I use this quick and dirty script to observe the juju megawatcher: http://pastebin.ubuntu.com/8706233/
  <voidspace> frankban: awesome, thanks
   * perrito666 just now realizes that the build is blocked
  <voidspace> frankban: so I've relinked the /usr/bin/juju* binaries to my built ones (so I can use juju-quickstart) and I'm doing git bisect
  <voidspace> see if I can tell where this broke
  <voidspace> frankban: as the bug is reported against alpha2 I'm assuming alpha 1 is known good
  <frankban> voidspace: I am not sure about that
  <frankban> voidspace: FYI the env var JUJU can be used in quickstart to override the JUJU binaty location, so that you don't have to relink the distro juju
  <frankban> binary even
  <voidspace> frankban: hah
  <voidspace> frankban: that would be better...
  <voidspace> frankban: so we're not sure that 1.21 alpha 1 is good
  <voidspace> frankban: so I need to go back and find the last good release and bisect from there
  <frankban> voidspace: I did not test that
  <voidspace> frankban: ok, thanks
  <tvansteenburgh> voidspace: i've been using deployer with alpha1 since it came out w/ no problem
  <frankban> np, thank you for looking at that
  <tvansteenburgh> voidspace: (fwiw), i think this broke in alpha2
  <voidspace> tvansteenburgh: cool, thanks
  <voidspace> tvansteenburgh: I'm mid-bisect so stopping to double check would be "inconvenient"
  <voidspace> although I'll probably check on my laptop anyway
  <voidspace> wasting time on a bisect that can never work would be *worse*
  <frankban> voidspace: I suppose you already noticed that, but FYI the addresses bug (quickstart) is only part of the bigger problem (and we assumed to be related but we are not really sure), the bigger problem being entities (e.g. services) not listed in the mega-watcher responses
  <frankban> voidspace: if you are tackling also that, then my watch script can be useful while bisecting, e.g. bisect, bootstrap, deploy one or two services, run the script to see the watcher response reflect the status, etc.
  <hazmat> tvansteenburgh, alpha2 is the breaker
  <hazmat> alpha1 is fine
  <hazmat> tvansteenburgh, this has nothing to do with deployer per se
  <hazmat> tvansteenburgh, this is just core having broken the all|mega-watcher
  <tvansteenburgh> hazmat: understood, i was just trying to help voidspace limit the breadth of his bisect :)
  <hazmat> right
  <hazmat> voidspace, you have a script already?
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs:  1385289
  <hazmat> the issue is that you'll need to cross grade your api servers to match your bisect or bootstrap per.. ick
  <voidspace> hazmat: no, but it's not a comlex set of steps
  <voidspace> hazmat: I'll destroy-environment and go install ./... in between each bisect
  <voidspace> hazmat: (had to stop - restarting now)
  <voidspace> hazmat: it's building a local environment, so not complex
  <voidspace> hazmat: I'll probably script it
  <natefinch> perrito666: your restore worker is LGTM
  <perrito666> natefinch: tx, I just got power back
  <perrito666> apparently 40Â°C exceeds what the power converters can handle
  <ericsnow> what would cause "state changing too quickly; try again soon" when running a transaction?
  <wwitzel3> ericsnow: not sure, the only place that error comes from is the txn transactionRunner.Run
  <perrito666> ericsnow: wwitzel3 most likely too many retries to run a tx.Op with no success
  <ericsnow> wwitzel3: yeah, and I'm not exactly sure what I did to change the existing behavior
  <wwitzel3> ericsnow: and the body of that is just a for loop that used nrRetries (set to 3) .. so after 3 attempts, the if none of the return paths in the for loop hit, you get that error.
  <ericsnow> perrito666: what does the retries?
  <cmars> ericsnow, that error is often (though not always) a bug in the logic for state changes.
  <ericsnow> cmars: I'm introducing a separate DB just for backups and running backups-related queries against that
  <cmars> ericsnow, a transaction that fails to maintain the assert condition for the duration of the operation will error out that way
  <ericsnow> cmars: I'll look into that
  <cmars> ericsnow, for example your operation negated the assert condition, that'd be a logic bug
  <cmars> ericsnow, could also be a concurrency issue, in which you're requiring conditions that fail to converge when you'd expect
  <fwereade> cmars, hey, did you come to a conclusion about service-owner?
  <cmars> fwereade, service-owner.. the context escapes me atm
  <fwereade> cmars, we added it to hook contexts as part of the initial abortive metrics spike
  <fwereade> cmars, AFAIK nobody uses it
  <fwereade> cmars, I'd really like to just delete it and hope nobody notices
  <fwereade> cmars, also, would you confirm that we'll be dropping the allow-add-metrics-only-in-some-hooks? because I'd like to drop it right now ;p
  <thumper> natefinch: hey
  <natefinch> thumper: yo coming
  <natefinch> wwitzel3: lxc meeting?
  <perrito666> rick_h_: how do you remove the tool bar form gvim?
  <natefinch> perrito666: install emacs ;)
  <katco> did someone say emacs?
  <natefinch> heh
  <perrito666> natefinch: sadly I only have 10 fingers in my hand
  <katco> no worries, they sell foot pedals now!
  <menn0> perrito666, katco: I use Emacs with Evil which reduces the need for non-standard appendages
  <menn0> https://gitorious.org/evil/pages/Home
  <menn0> it also requires that your brain has been rewired for Vim bindings but mine already was :)
  <menn0> it's a really well written extension anyway
  <katco> menn0: i've used that when vim ppl try to help me out at my keyboard
  <katco> menn0: i hear good things
  <natefinch> evidenly redhat is branching out.  My daughter played with some of these at school: http://2.bp.blogspot.com/-3c1hIpxVvms/T5CvbwxJV7I/AAAAAAAABZ0/xDYY4yOkm9E/s1600/easter+scentos.jpg
  <waigani> thumper/menn0 what should I test for when a machine has not instanceid and there is no instanceData doc?
  <menn0> waigani: I think so. The test should ensure that a warning is logged.
  <waigani> menn0: okay, cheers
  <menn0> waigani, thumper: bug 1386143 could be due to migration changes ...
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Triaged> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <alexisb> waigani, hey there
  <alexisb> I need to chat with mramm before we meet, mind if I am a few minutes late?
  <thumper> menn0: hmm... ya think?
  <menn0> waigani, thumper: in fact I think we already fixed that bug but it didn't make alpha2. this is probably due to the doubled up env UUID prefixing in the allwatcher.
  <thumper> ah...
  <fwereade> wallyworld, ping
  <thumper> sinzui: what is the process for broken alphas?
  <waigani> alexisb: when are we meeting?
  <menn0> thumper: the fix is part of the machines colleciton change
  <thumper> sinzui: do we just try the next alpha?
   * thumper nod
  <thumper> s
  <menn0> thumper: shall I dig in to this a bit?
  <wallyworld> fwereade: am in a meeting, will ping soon
  <thumper> menn0: yeah, have a quick dig and see if the machine fixes did fix it
  <cmars> wallyworld, ping me as well when you're done
  <menn0> thumper: k
  <waigani> alexisb: just ping we when you're ready
  <fwereade> wallyworld, cool, talking to cmars about the interactions between metrics and status
  <fwereade> wallyworld, rough eta?
  <fwereade> thumper, menn0, waigani: o/
  <thumper> o/
  <wallyworld> fwereade: 30 mins?
  <waigani> fwereade: hey
  <fwereade> thumper, menn0, waigani: hazmat had a bug about AllWatcher not notifying about important stuff; frankban had probable confirmation; can I rest assured that you guys are loking into  it/on it?
  <waigani> fwereade: sounds like the above bug?
  <thumper> fwereade: yeah, menn0 is looking into it now
  <fwereade> waigani, menn0, thumper: yay, you're already on it, I am redundant :)
  <sinzui> thumper, we fix them in the next alpha. We want to release alpha3 this week
  <fwereade> wallyworld, cmars: please talk when you're both free
  <fwereade> wallyworld, cmars: will try to be back in half an hour, if not I think you have enough to talk about
   * fwereade goes to play some lego star wars
  <alexisb> waigani, sent you a reschedule
  <waigani> alexisb: sweet, thanks
  <waigani> menn0: http://reviews.vapour.ws/r/177/
  <menn0> waigani: I'm going to be a while... looking at this regression
  <waigani> menn0: all good, I'll move onto sequences
  <menn0> waigani: sounds good. that's not going to rely on the instancedata branch
  <waigani> menn0: I'll put my initials next to what I'm working on in the MESS doc
  <menn0> waigani: good idea
  <menn0> waigani: I'll do the same
  <menn0> waigani: remember to add LK cards as you go as well
  <waigani> menn0: sigh, I'll add those now. I added your username in the MESS doc
  <menn0> waigani: cheers
  <voidspac_> menn0: ping
  <menn0> voidspac_: hey
  <voidspac_> menn0: hi, how's it going?
  <voidspac_> menn0: are you working on 1386143?
  <menn0> voidspac_: not too bad
  <menn0> voidspac_: I am indeed
  <menn0> voidspac_: I suspect it's already been fixed by Jesse and I
  <voidspac_> menn0: I started to look at it - my intention was to do a git bisect
  <ericsnow> voidspac_: thanks for taking care of that backup metadata test :)
  <menn0> voidspac_: we noticed the problem and fixed it but the branch didn't land before alpha2 was fixed
  <voidspac_> menn0: but following the repro instructions on the bug I couldn't get juju-quickstart working even with 1.20!
  <voidspac_> menn0: hah
  <menn0> voidspac_: s/fixed/cut/
  <voidspac_> menn0: what is the problem?
  <voidspac_> ericsnow: no problem, easy fix
  <menn0> voidspac_: a buggy change introduced as part of env UUID migrations
  <voidspac_> menn0: ah right, can I look at your branch?
  <voidspac_> menn0: I started looking through the diff between 1.21 alpha 1 and alpha 2
  <voidspac_> it's about 8000 lines or something...
  <menn0> if it's what I think it is then the fix is already merged.
  <menn0> voidspac_: it'll be part of 69c60b6fb293a7e6ea3a32d6365cf5eccd257fa5
  <voidspac_> menn0: doesn't look like anything landed on trunk
  <menn0> voidspac_: the bottom of megawatcher.go
  <voidspac_> menn0: ah, right - so trunk should already be unbroken
  <menn0> voidspac_: the docID method was adding the environment UUID onto ids that already had it
  <voidspac_> menn0: I'll try it with the repro instructions
  <menn0> voidspac_: I'm doing the repro with master now
  <voidspac_> menn0: cool
  <menn0> voidspac_: looking better so far. there's unit deltas coming out of the allwatcher with master where there wasnt with alpha2
  <rick_h_> perrito666: https://github.com/mitechie/pyvim/blob/master/.vimrc#L116
  <menn0> voidspac_: quickstart works with master so it looks like the problem is fixed \o/
  <voidspac_> menn0: awesome, great news
  <wallyworld> fwereade: sorry, free now, too late?
  <menn0> voidspac_: I'll try one more thing to confirm and then will update the ticket
  <wallyworld> cmars: hi, did we need to talk?
  <fwereade> wallyworld, cmars: heyhey
  <cmars> wallyworld, yeah, wanted to catch up on status stuff
  <wallyworld> sure
  <cmars> and other things
  <wallyworld> hangout?
  <fwereade> one of you statr one please :)
  <wallyworld> we can use this one https://plus.google.com/hangouts/_/canonical.com/ian-tim
  <fwereade> wallyworld, joined
  <wallyworld> fwereade: cmars posted one aove
  <wallyworld> https://plus.google.com/hangouts/_/gqzen7cwgsho4s6fxwqrgoc7dma?hl=en
  <menn0> voidspac_, thumper, waigani: regarding bug 1386143... quickstart and the GUI are definitely working with master. still doing a few more checks though.
  <mup> Bug #1386143: 1.21 alpha 2 broke watch api, no longer reports all services <api> <regression> <juju-core:Triaged by menno.smits> <juju-gui:Triaged> <juju-quickstart:Triaged> <https://launchpad.net/bugs/1386143>
  <waigani> menn0: good to hear!
  <thumper> menn0:  good
  <thumper> thanks
  <waigani> menn0/thumper manual upgrade testing, deploy wordpress and mysql before upgrade?
  <waigani> anything else?
  <menn0> waigani: I have a script which also sets up some subordinate units
  <menn0> waigani: I think I've sent it to you before
  <waigani> menn0: yeah I just remembered that, looking for it now...
  <menn0> waigani: that's what I've been using
  <waigani> thumper/menn0: http://reviews.vapour.ws/r/269/
  <perrito666> katco: but tell me, does you editor clean and refreshes like mine? https://dl.dropboxusercontent.com/u/7254758/Photo%2027-10-14%2019%2017%2054.jpg
  <perrito666> and no, I did not go to te supermarket just for the sake of this discussion
  <waigani> perrito666: lol
  <rick_h_> menn0: <3 glad to hear about bugs already fixed
  #juju-dev 2014-10-28
  <cmars> contrib Casey Marshall
  <cmars> (that was for mup_)
  <wallyworld> thumper: i disagree that bug 1385289 should block landings. stuff isn't broken, it's slower
  <mup_> Bug #1385289: local storage migration is very slow <ci> <regression> <test-failure> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1385289>
  <wallyworld> it should be critical for sure
  <davecheney> $ go version
  <davecheney> go version devel +211c346906d0 Tue Oct 28 09:56:33 2014 +1100 linux/power64le
  <thumper> wallyworld: agreed
  <davecheney> ^ real go, on power
  <wallyworld> \o/
  <thumper> nice
  <davecheney> thumper: fixed a few bugs there today
  <thumper> \o/
  <davecheney> including some stuff that makes me wonder how the guys at google were testing this
  <davecheney> https://code.google.com/p/go/source/detail?r=cf3c9803a3d2116a9621fb1e9b2226a8fa3d5e10
   * thumper -> physio
  <davecheney> for example
  <waigani> menn0: you about?
  <menn0> waigani: yep
  <waigani> menn0: i've hit a tricky one, can we talk it over?
  <waigani> menn0: standup channel?
  <menn0> yep and yep
  <perrito666> sorry did you say that the critical bug was fixed?
  <waigani> menn0: should we add a check for the EnvUUID as a generic test for all the upgrade steps?
  <menn0> waigani: it already does :)
  <menn0> see checkAddEnvUUIDToCollection
  <waigani> menn0: ? how did the tests pass then?
  <menn0> that checks that document has the env-uuid field
  <menn0> but it doesn't check that the struct has it added
  <menn0> there's no generic way to do that
  <menn0> well... I maybe there is but it won't be pretty
  <waigani> reflection and assuming the name is EnvUUID?
  <waigani> yeah, it would be ugly
  <waigani> I'm happy to let it go
  <menn0> I think the best bet is to check in the test function after checkAddEnvUUIDToCollection has done its thing
  <menn0> so for instanceData, check EnvUUID where MachineId and Status are checked
  <rick_h_> wallyworld_: I'm going to check with frankban on adding this as a new bug but fyi on https://bugs.launchpad.net/juju-core/+bug/1336843 as it's a fix in alpha2 that has a follow up effect.
  <mup_> Bug #1336843: bootstrap without a jenv destroys an existing environment  <bootstrap> <juju-core:Fix Released by cox-katherine-e> <https://launchpad.net/bugs/1336843>
  <wallyworld_> rick_h_: no worries, file a bug if needed and katco can look into it
  <rick_h_> wallyworld_: cool
  <wallyworld_> rick_h_: if there's an issue, we'll fix asap
  <wallyworld_> thanks for letting us know
  <davecheney> thumper: $ juju version
  <wallyworld_> axw: long story wrt SyncBuiltTools, maybe my explanation was crap. i took out the stream param and wrote to a "custom" directory/stream when building the tools but it failed and i traced through the code to figure out what was happening. but the logic that syncs expects the tools in the source datasource to be in the same stream as the destination, since it needs to read from the public cloud that way. hard to explain,
  <wallyworld_> happy to discuss if needed
  <davecheney> 1.21-alpha3-trusty-power64le
  <thumper> davecheney: awesome to see
  <waigani> menn0: http://reviews.vapour.ws/r/269/
   * menn0 looks
  <davecheney> ok, now back to what I was supposed to be doing
  <axw> wallyworld_: SyncBuiltTools needs a stream param; I was just saying --upload-tools should set an obviously arbitrary value
  <wallyworld_> axw: trouble is that an arbitrary value breaks the processing, but i'll take another look to be sure
  <menn0> waigani: done
  <waigani> sweet, thanks menno
  <wallyworld_> since upload-tools calls a method which copies from source to dest, and that copy needs to have same stream for source and dest, not an arbitrary one for source
  <axw> wallyworld_: upload-tools does not do that anymore.
  <axw> look at uploadTools in cmd/juju/upgradejuju.go; it opens the file and passes it to UploadTools
  <wallyworld_> some infrastructure did in the tests
  <wallyworld_> i'll take another look to be sure,but a whole bunch of sync tools tests failed
  <wallyworld_> if the source stream != dest stream
  <wallyworld_> and sync tools calls that same shared function as upload tools ses from emory
  <waigani> menn0: testing for EnvUUID field, we currently are not doing it for any of the collections. Shouldn't be for all or none?
  <menn0> waigani: probably
  <menn0> waigani: your call
  <menn0> waigani: the most important thing I guess is that the field is there in the DB, which we are testing
  <menn0> waigani: it's trivial to add the field to the struct later if we've forgotten
  <waigani> menn0: yep, I'll leave it as is
  <axw> wallyworld_: http://paste.ubuntu.com/8712745/    <- all tests pass in cmd/juju with this, and upgrade-juju works as expected. haven't tested bootstrap, but should be the same.
  <wallyworld_> axw: trouble is, when i made that change, most tests in environs.sync fail
  <axw> ?? how could a change in cmd/juju affect environs/sync?
  <wallyworld_> oh, i see what your saying
  <wallyworld_> you're
  <wallyworld_> i had experimented with not having the streams parameter at all
  <wallyworld_> and just using "custom" all the time
  <axw> right, the param is needed for *other* cases, just no in this and the bootstrap case
  <axw> not*
  <wallyworld_> right ok, fair point. sorry i misunderstood you
  <axw> nps
  <wallyworld_> i'll change that when i do the index file fix
  <axw> it's a fairly minor thing, just wanted to make sure we both were on the same page
  <wallyworld_> yeah, np. thanks for pushing the issue
  <waigani> menn0: the instanceData upgrade step. To add a post-env UUID, based on the machine _id have to either: assume machine _id is pre-env and pass it through st.docID() (in which case, the upgrade step has to come before machine upgrade), or assume the machine _id is post-env (in which case, the upgrade step has to come after the machine upgrade)
  <waigani> menn0: i.e. I don't see how we can make the step not care about order
  <menn0> waigani: or you could detect if the machine is pre or post env UUID upgrade
  <menn0> waigani: by checking the env-uuid field
  <waigani> menn0: or that
  <waigani> menn0: okay, will do
  <menn0> waigani: if that seems too messy, then pick and ordering and add a comment in the list of steps explaining that the ordering is important and why
  <waigani> menn0: I didn't suggest it as it seemed a bit messy, I'll see how bad it looks and fall back to a comment
  <waigani> menn0: I also forgot about just checking the EnvUUID - thought we'd have to regex to _id
  <menn0> waigani: cool. yeah, just see how it looks.
  <axw> wallyworld_: what's the simplest way to create tools+metadata from source?
  <axw> I only want the built tools in the metadata I create
  <wallyworld_> axw: there's no easy way as such. if you have a tarball, you can easily generate metadata, but as far as i know, there's no easy way to get a tarball from source
  <axw> wallyworld_: that's what I though - thanks
  <wallyworld_> axw: what are you trying to do?
  <axw> just want to live test the upgrade bug
  <axw> I've got a unit test
  <axw> I'll bootstrap local and pull the tarball out of there
  <wallyworld_> yep, that's what i do
  <wallyworld_> maybe we need a maketarball plugin
  <waigani> menn0: so we search for instanceData docs with the machineID - pre or post env depending on whether or not the instanceData upgrade step has been run
  <waigani> menn0: I'm thinking order just matters: migrate 1. instanceid, 2. machines envUUID 3. instanceData envUUID
  <menn0> menn0: that's probably less messy isn't it?
  <menn0> menn0: just write a comment explaining it and be done with it
  <waigani> menn0: sgtm!
  <waigani> menn0: http://reviews.vapour.ws/r/177/
  <waigani> menn0: http://reviews.vapour.ws/r/269/
   * menn0 is looking
  <menn0> waigani: done
  <waigani> menn0: cheers
  <waigani> thumper: http://reviews.vapour.ws/r/269/
  <waigani> thumper: http://reviews.vapour.ws/r/177/
  <thumper> ack
  <wallyworld_> thumper: remind me, we're not meant to run apt-get update for local provider are we? i thought we'd check for dependencies like cpu-checker and complain if they weren't there, but not just install shit onto a user's host http://pastebin.ubuntu.com/8714033/
  <davecheney> thumper: menn0 https://github.com/juju/juju/pull/976
  <davecheney> i haven't created an rbt review for this yet
  <davecheney> i want to get a sense for how gross you think this is
  <davecheney> (I think it's 50% gross)
  <thumper> davecheney: rbt gets magically created for you now
  <thumper> wallyworld_: yes we are supposed to run update by default for local
  <thumper> wallyworld_: we should be able to configure it off for those who know what they are doing
  <wallyworld_> thumper: ok, ta
  <thumper> and have done an update recently themselves
  <wallyworld_> thumper: that was on my hst
  <wallyworld_> host
  <wallyworld_> not the lxc container
  <thumper> oh...
  <thumper> no, we install shit
  <wallyworld_> that log from when i bootstrapped
  <thumper> which is just another reason to fix the local provider
  <wallyworld_> yeah :-(
  <thumper> so that machine zero is a container too
  <wallyworld_> i thought we'd check but not install
  <thumper> I don't remember that
  <wallyworld_> maybe i'm making it up
  <wallyworld_> jam: i have an lxc question - you suggested in a bug comment setting a http/apt/whatever proxy to http://10.0.3.1:8000 instead of http://localhost:8000 where 10.0.3.1 is the lxcbr0 bridging address. i have set up a little http server on the host and can wget to it fine from the host, but inside the lxc container, wget can't get out due to connection refused. could be a firewall issue between container and host? do you
  <wallyworld_> have any suggestion?
  <jam> wallyworld_: so you put an http server on your host machine, and then are unable to access 10.0.3.1:PORT from lxc, is that correct?
  <jam> wallyworld_: did you start your HTTP server before starting LXC?
  <jam> Is it binding to "*" (0.0.0.0) or is it binding to 127.* or to an explicit IP address?
  <wallyworld_> jam: yes, but can access localhost:PORT from host
  <jam> wallyworld_: from the *host* can you access 10.0.3.1:PORT
  <wallyworld_> yes
  <wallyworld_> from my PC
  <wallyworld_> oh
  <wallyworld_> sec, let me check
  <jam> wallyworld_: my first thought is that your service isn't bound to all networks
  <wallyworld_> jam: yeah, can't connect to 10.0.3.1 from host. i was just trying to validate the replacing localhost with the bridge address actualy worked
  <wallyworld_> maybe there's no need to do that
  <jam> wallyworld_: so are you just doing something like python SimpleHTTPServer ?
  <wallyworld_> the http proxy, apt proxy etc are all set to <bridge addr> as required
  <jam> Are you handing it an address to bind to ?
  <jam> Rather than "0.0.0.0" ?
  <jam> wallyworld_: it might certainly be a real life issue, as having your HTTP proxy set up may not bind it to the LXC bridge address if it comes up first.
  <wallyworld_> jam: ah, i see the issue, the little go prog is binding to localhost
  <wallyworld_> http.ListenAndServe("localhost:8000", nil)
  <jam> wallyworld_: right, I think you can change that to ":8000' or maybe "0.0.0.0:8000"
  <wallyworld_> jam: i assumed that would cover all connections into the physical host
  <jam> wallyworld_: nope, specifying localhost restricts it to 127.* as it is how you avoid making something publi
  <wallyworld_> oh, i'll try that, ty :-)
  <wallyworld_> jam: you are awesome, that worked, ty
  <jam> wallyworld_: np
  <menn0> davecheney: looking now
  <menn0> davecheney: done
  <stokachu> anyone notice when doing a juju bootstrap with a maas provider that when it runs its apt-get update/upgrade it returns a non zero status?
  <stokachu> even though its chugging right along
  <axw> stokachu: apt-get update/upgrade fails, but bootstrap continues on? what version of juju is that?
  <stokachu> 1.20.10
  <stokachu> the upgrade doesn't fail though
  <axw> can't say I've seen that. we do "set -e" ... can't see any obvious reason why that'd happen
  <stokachu> it returns the same 124 status code
  <davecheney> stokachu: exit 124 is timeout
  <davecheney> that is coming from deep the bowels of apt
  <stokachu> ok
  <davecheney> stokachu: did you paste the bootstrap log ?
  <stokachu> nah im doing another deploy
  <stokachu> ill paste once it fails again
  <davecheney> kk
  <davecheney> The following packages have unmet dependencies: gdb : Depends: libpython3.4 (>= 3.4~b1) but it is not going to be installed
  <davecheney> kill me
  <tasdomas> morning
  <tasdomas> ashipika was trying to land his PR and got this response from the buildbot: http://juju-ci.vapour.ws:8080/job/github-merge-juju/1029/console
  <tasdomas> any ideas as to what may be causing this?
  <ashipika> and this one is interesting, as well http://juju-ci.vapour.ws:8080/job/github-merge-juju/1028/console
  <davecheney> my tasdomas mongodb shat itself on startup
  <davecheney> that is why it cannot complete the TLS handshake
  <tasdomas> davecheney, should we retry the merge?
  <davecheney> yes
  <davecheney> it'll pass eventually
  <tasdomas> davecheney, ok - ashipika's first attempt failed when godeps failed to retrieve launchpad.net/tomb
  <davecheney> yup
  <davecheney> in that case
  <davecheney> launchpad shat itself
  <ashipika> :D
  <ashipika> shit is definitely transitive..
  <davecheney> ashipika: we checkout all the build deps on each build
  <davecheney> makes the build suceptable to other systems that are weakly available
  <ashipika> davecheney: no problem, should i just keep retrying them until the build succeeds?
  <davecheney> ashipika: yup
  <davecheney> both of those failures are transiant
  <ashipika> davecheney: rogerroger
  <davecheney> sorry for the inconveneince
  <ashipika> davecheney: no, no, i completely  understand.. thanks for the info
  <davecheney> ashipika: np
  <dimitern> morning all
  <jam> morning dimitern
  <dimitern> hey, jam!
  <mattyw> morning all
  <mattyw> dimitern, morning
  <TheMue> morning
  <voidspace> morning all
  <jam> dimitern: I'm not sure if you saw the calendar update, but we moved the standup time by 45 minutes to see if it would work better for everyone (you may not be back yet, I guess)
  <jam> TheMue: voidspace: just a poke that it is soon since I know the calendar is changed
   * fwereade out for a bit
  <jam> hey fwereade
  <jam> have fun :)
  <voidspace> jam: omw
  <voidspace> jam: hmmm... firefox is in a redirect loop
  <voidspace> restarting browser time...
  <jam> voidspace: sounds exciting
  <jam> where will you end up next ? :)
  <voidspace> oh it is
  <voidspace> hah
  <voidspace> always an adventure
  <dimitern> voidspace, jam, TheMue, hey guys, I've just came back - omw for the standup
  <voidspace> dimitern: hey
  <perrito666> morning
  <voidspace> perrito666: morning
  <mattyw> dimitern, after the standup could you take a look at this? http://reviews.vapour.ws/r/265/
  <dimitern> mattyw, sure
  <mattyw> dimitern, thanks very much - it's the fix for the add-unit bug you found last week
  <dimitern> mattyw, sweet!
  <perrito666> is anyone using exuberant-ctags?
  <mattyw> perrito666, I am
  <perrito666> mattyw: and that with vi?
  <mattyw> perrito666, of course (are there other editors ;) )
  <perrito666> mattyw: I am getting an empty tagbar with go
  <perrito666> yet, if I ctags -x on a go file I get results
  <mattyw> perrito666, you use tagbar? wow - not used that in a loooong time.
  <mattyw> perrito666, which ctags are you using for go?
  <perrito666> mattyw: care to share what are you using?
  <perrito666> 5.9-svn
  <mattyw> perrito666, https://github.com/jstemmer/gotags
  <perrito666> mattyw: I am a bit wary of using go get as a package manager, since I tend to nuke my /bin in go
  <mattyw> perrito666, I don't update it enough to worry about package management - just build it then move it somewhere you don't nuke
  <mattyw> perrito666, do you use godef?
  <mattyw> perrito666, and by extension vim-godef?
  <perrito666> mattyw: I dont, what is it?
  <mattyw> perrito666, https://github.com/dgryski/vim-godef
  <perrito666> sweeeet
  <wwitzel3> fwereade: so I decided to go the explicit route and add a --no-remote-unit option
  <fwereade> wwitzel3, ok, that sounds sane
  <fwereade> wwitzel3, what do we do if nothing's specified and no units are in the relation? require --no-remote-unit, or just execute without one?
  <perrito666> ericsnow: ping
  <wwitzel3> fwereade: my thought was require --no-remote-unit .. easiest is to just ask what they'd prefer as the desired behavior and make sure it is documented accordingly
  <perrito666> rogpeppe: your timing is not the best :p
  <rogpeppe> perrito666: i'm sorry - i only just saw it
  <rogpeppe> perrito666: it shouldn't be hard to change though, should it?
  <perrito666> nope, its just that I have to revert it :p it would be nice to have a $$revert$$
  <rogpeppe> perrito666: ha, yes
  <rogpeppe> perrito666: git makes it very easy to do that though
  <perrito666> rogpeppe: It does since the bot treats it as a merge
  <perrito666> rogpeppe: Ill make sure that you did not miss anything before making the change but you might be right
  <rogpeppe> perrito666: thanks
  <rogpeppe> perrito666: i don't like seeing nice small APIs made bigger because of testing issues :)
  <perrito666> rogpeppe: I like it no more than having smart dummy objects :p It was a fast trade off, anyway going for your way now BUT, you will have to review it :p
  <rogpeppe> perrito666: np
  <fwereade> wwitzel3, I'm inclined to think that not having a remote unit is actually perfectly valid
  <fwereade> wwitzel3, and I'm also starting to feel sure that specifying a remote unit that appears not to exist *may* be valid
  <fwereade> wwitzel3, so it shouldn't be an automatic error
  <fwereade> wwitzel3, but it should probably be some sort of warning..?
  <fwereade> sorry bbiab
  <wwitzel3> fwereade: ok, that sounds fine to me
  <voidspace> rebooting
  <voidspace> dimitern: ping
  <ericsnow> dimitern: could you take a look at the patches I have up?
  <ericsnow> dimitern: the most important are http://reviews.vapour.ws/r/67/ and http://reviews.vapour.ws/r/135/)
  <ericsnow> dimitern: however, I could really use eyes on http://reviews.vapour.ws/r/79/ and http://reviews.vapour.ws/r/126/ as well
   * perrito666 looks at the 47" tv and wonders how comfortable its as a work screen
  <dimitern> voidspace, pong
  <dimitern> ericsnow, sure, looking
  <ericsnow> dimitern: thanks :)
  <dimitern> I was on a call, sorry for responding slowly
  <voidspace> dimitern: so we have stub implementation of AllocateAddress for all providers
  <voidspace> dimitern: that needs to change
  <voidspace> dimitern: I've mostly done that
  <dimitern> voidspace, great
  <voidspace> dimitern:  AllocateAddress(_ instance.Id, _ network.Id, _ network.Address) error
  <voidspace> dimitern: ?
  <dimitern> voidspace, that sounds good to me
  <voidspace> dimitern: cool, there's an existing test (dummy provider) that needs to change too
  <voidspace> just finishing that
  <dimitern> voidspace, possibly with the addition of a concrete error type returned for "conflict" or "address already reserved" sort of error
  <voidspace> dimitern: I figured we could sort that out later...
  <voidspace> dimitern: as we also need to handle "request failed - couldn't reach server" errors too
  <dimitern> voidspace, yeah, fair point -- just a reminder
  <voidspace> changing error type to a concrete type is trivial
  <dimitern> voidspace, "request failed", as in "i did 5 attempts so far and it failed all" ?
  <voidspace> dimitern: I guess
  <voidspace> whatever strategy we pick
  <voidspace> I did discuss that with John a bit yesterday but it morphed into a discussion about race conditions
  <dimitern> voidspace, I think ec2 deserves some retrying logic, being occasionally flaky
  <voidspace> dimitern: yep, ideally we'd abstract the retry machinery so it can just be reused
  <voidspace> rather than ad-hoc retrying of all api calls
  <dimitern> voidspace, there's *some* reusable bits already - around the AttemptStrategy type
  <voidspace> right
  <dimitern> voidspace, I even seem to recall a strategy that takes rate limiting into account for ec2, but I might be wrong
  <voidspace> dimitern: right, we did some rate limiting work
  <voidspace> man, that was one of the first things I worked on
  <dimitern> voidspace, yeah :) -- but wasn't that related to our apiserver only?
   * TheMue listens
  <dimitern> ericsnow, hey
  <voidspace> dimitern: I don't recall the details...
  <dimitern> ericsnow, so wrt http://reviews.vapour.ws/r/126/ - did you reach an agreement with davecheney about the approach ?
  <voidspace> dimitern: http://reviews.vapour.ws/r/274/
  <dimitern> voidspace, cheers -- will have a look after I'm done with ericsnow's
  <voidspace> dimitern: cool, thanks
  <TheMue> voidspace: AllocateAddress looks fine, implementation is nice short so far :D
  <voidspace> TheMue: :-)
  <voidspace> TheMue: yeah, done...
  <TheMue> +1
  <voidspace> TheMue: thanks
  <ericsnow> dimitern: wrt 126, while I addressed all his comments, I never heard back from Dave
  <dimitern> ericsnow, right, I'd send him a mail and ask for a final look, considering his initial not lgtm, just to be on the nice side
  <ericsnow> dimitern: oh, that...we did sort that out.  I re-wrote it using direct HTTP requests rather than going through the websockets API
  <dimitern> ericsnow, ah, ok -- it's fine then :)
  <dimitern> voidspace, you've got a review
  <dimitern> ericsnow, 1 review done, 3 to go
  <ericsnow> dimitern: rock on! :)
  <voidspace> dimitern: thanks
  <voidspace> dimitern: cool, I'll remove the TODO and merge
  <dimitern> voidspace, +1
  <voidspace> how odd
  <voidspace> internet on my desktop just stopped working
  <voidspace> was still working for everyone else in the house
  <voidspace> anyway, a reboot fixed it
  <perrito666>  rogpeppe https://github.com/juju/juju/pull/978
  <rogpeppe> perrito666: one thing: please make sure that the call to MgoTestPackage is in peergrouper_test, not peergrouper
  <rogpeppe> perrito666: otherwise the tests in your workerJujuConnSuite will never be tested
  <rogpeppe> perrito666: (in Go 1.3 at any rate)
  <rogpeppe> perrito666: ah, i see that's already the case
  <perrito666> rogpeppe: it is there where already tests in peergrouper_test
  <rogpeppe> perrito666: BTW in the peergrouper version I've got, workerJujuConnSuite is already defined in peergrouper_test
  <perrito666> rogpeppe: ?
  <rogpeppe> perrito666: that was a week ago, commit 9301dcf0b882a1ab389297518a145b29668da51e
  <rogpeppe> perrito666: so i'd just revert the name to the same thing - worker_test.go
  <rogpeppe> perrito666: (there are other external tests too, such as initiate_test.go)
  <rogpeppe> perrito666: anyway, reviewed
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs:  1385289 1386766
  <perrito666> rogpeppe: I did not change the name, I have worker_external_test.go and worker_test.go is still there
  <perrito666> tx for the review btw
  <rogpeppe> perrito666: but worker_test.go is in peergrouper_test already, right?
  <perrito666> rogpeppe: no more, I also reverted that :)
  <rogpeppe> perrito666: ok
  <perrito666> my lunch was so diet that I actually took me less to cook it than a full test run
  <perrito666> rogpeppe: this patch was sponsored by my new diet :p you can thank my wife
  <rogpeppe> perrito666: i highly recommend the 5-2 diet BTW
  <rogpeppe> perrito666: means you only have to try hard on two days in a week
  <perrito666> rogpeppe: there is no spec on wp on how "normal" can be a normal day
  <rogpeppe> perrito666: whatever you like, pretty much
  <perrito666> rogpeppe: according to the dietitian my problem is not that I eat caloric things, its just that I eat a lot of it, for any definition of it
  <rogpeppe> perrito666: yeah, so the 5-2 diet is good for just that
  <rogpeppe> perrito666: you can still pig out :)
  <rogpeppe> perrito666: (all but those two days)
  <perrito666> :p
   * perrito666 gives it a shot
  <perrito666> uff, we are blocked.. again?
  <rogpeppe> perrito666: you should watch the original documentary
  <rogpeppe> perrito666: google for [horizon 5-2 diet documentary]
  <sinzui> natefinch, can you arrange for a rollback to address bug 1386766? when it is removed, we can then test a fix the the upgrade regression that merged a few hours ago
  <mup> Bug #1386766: generate-tools breaks scripted and documented behaviour <ci> <metadata> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1386766>
  <TheMue> dimitern: thx
  <dimitern> TheMue, np :)
  <dimitern> ericsnow, all reviews done
  <ericsnow> dimitern: awesome! Thanks for doing that.
  <dimitern> ericsnow, np :) thanks for doing all that work
  <natefinch> sinzui: ok
  <voidspace> dammit, dimitern has left
  <voidspace> he always escapes before I can harrass him
  <wwitzel3> having this bag of pistachios by my desk was a mistake, I can't feel the tips of my fingers now
  <voidspace> wwitzel3: :-)
  <voidspace> wwitzel3: a bag of pistachios is never a mistake
  <voidspace> pretty much by definition
  <perrito666> wwitzel3: that sounds a lot like you are alergit to pistachios
  <voidspace> TheMue: have you made progress?
  <wwitzel3> perrito666: haha, my fingers are alergic to cracking open 100 of them
  <perrito666> wwitzel3: oh, you shoult learn to do with your midle front tooth, its a life changer
  <perrito666> :p
  <natefinch> wwitzel3: you should buy them shelled :)
  <natefinch> wwitzel3: I actually did the math and at least at my grocery store, they actually come out to almost the exact same price per pound of edible bits
  <perrito666> oh, that is an option too
  <perrito666> natefinch: for that you actually had to peel all of them and weight them
  <perrito666> right?
  <wwitzel3> natefinch: yeah, but then, I eat too many and I've complaining about a stomach ache .. either way I'd be complaining :P
  <natefinch> haha yep
  <perrito666> anyway I cannot eat pistachios while not having beer, so I eat peanuts which are easier to peel
  <natefinch> perrito666: yes, actually.  I bought a bag of the in-shell version, ate them, then weighed the shells
  <perrito666> natefinch: clever
  <natefinch> perrito666: I forget the percentage right now... maybe 30% of the weight was shell... something like that
  <perrito666> ericsnow: ping
  <ericsnow> perrito666: hey
  <perrito666> ericsnow: priv
  <wwitzel3> well that 14.10 upgrade did not go well .. had to boot a live CD and do a boot repair, and now things are a bit wonky
  <wwitzel3> all my dev stuff still works fine though, so that's good
  <perrito666> wwitzel3: ah I got an "interesting" behavior in network for a moment
  <perrito666> wwitzel3: and in the rest of my system too as you most likely saw in my attempt at a hangout yesterday
  <natefinch> sinzui: reverting that merge here: https://github.com/juju/juju/pull/979
  <sinzui> thank you very much natefinch .
  <perrito666> natefinch: you missed the fix tag
  <natefinch> dammit
  <natefinch> now remind me, because I always forget... do I put that in the $$ $$ or do I have to put it in the previous commit message?  Also does it have to be 'fixes-1386766' or is it supposed to be 'fixes 1386766'  (space vs. hyphen)?
  <natefinch> sinzui, perrito666 ^^
  <sinzui> natefinch, fixes-1386766
  <sinzui> you can optionally __fixes-1386766__ to make it bold and show the works you mean it
  <natefinch> sinzui: yes, but does that work inside the $$stuff$$?  I feel like I've tried that in the past and it didn't work
  <natefinch> since I think it expects just alphanumeric inside the $$ $$
  <sinzui> natefinch, "$$merge$$ fixes-1386766" in the same comment is fine
  <natefinch> ok
  <sinzui> natefinch, and if you think CI is blocking you from merging a critical fix, add __JFDI__ to make it be tested and merged on success. __JFDI__ is our safety to ensure an automated system doesn't stop os from doing the right thing
  <natefinch> ok, I thought i remembered as JFDI, but couldn't remmeber the exact syntax.  Is there documentation of this stuff somewhere?
  <sinzui> natefinch, in an email to canonical-juju.
  <natefinch> email is not documentation :)
  <sinzui> natefinch, I can add a page to reports.vapour.ws
  <natefinch> sinzui: that would be awesome, thank you :)
  <sinzui> natefinch, I am not saying it is, but I write a lot of official gdocs, an no one reads them
  <katco> sinzui: i am familiar with that problem; one solution is to have much fewer docs, but more sections so that you have a single starting point to give people
  <katco> sinzui: and when these situations arise, you just point and go: "go forth and learn"
  <natefinch> sinzui: I understand the frustration.  hopefully we can help people remember that documentation exists
  <natefinch> yep
  <katco> it doesn't help that docs is not very navigatable imo
  <natefinch> yes, sorry, I thought gdocs was a typo.   Google docs is a terrible place to keep documentation
  <katco> natefinch: well, it's not _horrible_, but i feel like there needs to be some sort of indexing mechanism
  <natefinch> docs should be at a website with a url that means something
  <rick_h_> katco: howdy, I mentioned it to wallyworld last night. There was a small issue that came out of the work to not destroy envs when there's no jenv file that we hit. https://bugs.launchpad.net/juju-core/+bug/1386703
  <mup> Bug #1386703: New instance created when bootstrapping an existing env without the jenv file <juju-core:New> <https://launchpad.net/bugs/1386703>
  <natefinch> like juju.ubuntu.com/docs
  <katco> rick_h_: thanks, i'll have a look. are you all blocked on this or what's going on?
  <rick_h_> katco: as an fyi, not sure where it'll fit on the todo list but wanted to bring it front/center as it's an issue for quickstart
  <katco> rick_h_: yeah... i'll bring it up to wallyworld_ tonight in our standup
  <rick_h_> katco: no blocked, but would be great if the release didn't close one bug and open another right next to it.
  <rick_h_> katco: ty
  <katco> rick_h_: gotcha. thanks for the ping.
  <voidspace> g'night all
  <natefinch> I wish godeps didn't say "failed" when it's actually working as designed
  <natefinch> ah dammit.... google's winsvc package isn't importable in linux, because it's all // +build !windows   ... and so godeps can't find it, so godeps -t doesn't include it in the output
  <perrito666> natefinch: fork it?
  <natefinch> ahh, you know what works?  GOOS=windows godeps -t ./... > dependencies.tsv
   * natefinch feels pretty smart
  <natefinch> it works because windows dependencies are a strict superset of the linux dependencies
  <natefinch> perrito666: quick review? http://reviews.vapour.ws/r/277/diff/#
  <perrito666> natefinch: what is the 58 added at the end of service?
  <natefinch> perrito666: I forget.  some hg thing.... possibly an easy to remember incremental diff id.  It's optional, so people who insert dependencies in the file by hand often forget it.... but since I generate the file with godeps, it remembers and puts it in there
  <perrito666> good enough for me
  <perrito666> natefinch: lgtmd
  <natefinch> perrito666: thanks
  <thumper> morning folks
  <natefinch> morning thumper
  <alexisb> morning thumper
   * thumper looks sadly at his very full inbox
  <natefinch> thumper or fwereade: got a second for a UX question?    charm level constraints include "container type blacklist"  what do we do if the user says to deploy a charm to a container that the charm says it doesn't support?  Warn the user?  prevent the deployment?  If we prevent it, is there a way to override?
  <natefinch> (or anyone else who has an opinion is free to speak up too :)
  <thumper> I think that we shouldn't stop them, but yes to warning
  <thumper> and say "the charm says this won't work"
  <perrito666> did the metadata fix land?
  <thumper> but if the user explicitly says "do it this way" we let them try
  <natefinch> thumper: sounds good
  <natefinch> thumper: do we still skip over newly created containers made with add-machine?
  <natefinch> thumper: actually... do we auto-deploy to an empty container?
  <natefinch> thinking about 'juju add-machine && juju deploy foo' will add a machine and then deploy foo to it... but not sure if that's true of 'juju add-machine lxc:4 && juju deploy foo'
  <thumper> maybe...
  <waigani> menn0_: was it relations or relationScopes that broke the world?
  <natefinch> thumper: easy enough to test.... still, I presume it would skip containers by default
  <natefinch> dammit
  <natefinch> stupid replicaset tests, who wrote that crap, anyway?
  <natefinch> sinzui: tests giving "No space left on device"
  <sinzui> hmm
  <sinzui> which one natefinch
  <natefinch> sinzui: http://juju-ci.vapour.ws:8080/job/github-merge-juju/1042/console
  <sinzui> natefinch, this might be tricky I see 29G free.
   * sinzui pokes arround
  <sinzui> ah, this is the machine instance
  <waigani> menn0: remember those selects from yesterday? They just selected _id. But then the unmarshalled charm doc is expected to have a URL - which *was* the id but is now it's own field. So I've added "url" to the select and all works again.
  <sinzui> natefinch, this perplexing. I don't see evidence of the instance size being changed. We can see about 6G available on the root disk
  <menn0> waigani: that makes sense
  <waigani> menn0: yep, cool
   * sinzui ponders changing the working dir to one of the mounted disks
  <perrito666> sinzui: I am having an interesting issue, I have my script, that creates a stream structure with "juju metadata generate-tools -d ${BUILD_FOLDER}/juju-dist/" creating, among other things, juju-dist/tools/streams/v1/index.json but then I have juju failing because it cannot find juju-dist/tools/streams/v1/index.sjson
  <perrito666> sinzui: happened to you?
  <sinzui> perrito666, pull trunk, natefinch reverted wallyworld_'s commit that changed the default behaviour
   * perrito666 pulls again
  <waigani> menn0: and with that state passes
  <waigani> \o/
  <menn0> waigani: awesome
  <natefinch> perrito666: it hasn't landed yet.... it keeps failing
  <perrito666> natefinch: ah
  <perrito666> a world of joy
  <natefinch> indeed
  <natefinch> sinzui: I'm happy to retry, but my EOD is in 5 minutes
   * perrito666 drinks a beverage at 90Â°C to protest against the summer heat
  <sinzui> perrito666, maybe something else is broken or we aren't testing the reverted commit. CI still cannot deploy anything using streams
  <sinzui> natefinch, retry your branch
  <natefinch> sinzui: it's going
  <sinzui> thank you for your time natefinch
  <waigani> menn0/thumper: what do you thing about creating an interface for environment specific collection documents with DocID and EnvUUID fields? One use would be testing for those fields in the upgrade steps. Would there be other value to make it worthwhile?
   * thumper otp
  <menn0> waigani: what would the interface look like? remember that interfaces are only for methods, not for fields
  <waigani> menn0: ah right, it would have to be an embedded struct - messy
  <perrito666> ok I am taking a break while natefinch chanes land, bbl
  <perrito666> and as I said that, the change landed
  <natefinch> huzzah
  <thumper> perrito666: plz send some summer heat my way
  <thumper> bit chilly here still
  <perrito666> thumper: sure, Ill be glad to get rid of some summer heat, specially because its still freaking spring
  <alexisb> wallyworld_, you around?
  <wallyworld_> alexisb: yes, ia am, i have a few emails to respond to, did you want to chat? i'm in a meeting now but will be free soon
  <alexisb> yeah I do, if you are in a meeting now then I will ping you in about 20 minutes
  <wallyworld_> sure
  <perrito666> ericsnow: what implementation of tar are you using?
  <ericsnow> perrito666: pretty sure it's the one you put into the utils repo
  <perrito666> ericsnow: the symlinks implementation is broken and I recall fixing it
  <perrito666> ericsnow: 972 has a rb link?
  <waigani> what should happen if you upgrade juju while a unit agent-state is pending?
  <davecheney> thumper: the uniter tests don't pass for me at trunk
  <davecheney> have you seen that ?
  <anastasiamac> perrito666: thnx 4 z review!
  <perrito666> np, don't take me as an authority anyway
  <perrito666> most where suggestions
  #juju-dev 2014-10-29
  <davecheney> all: master is not passing the tests for me
  <davecheney> uniter, api client, worker/uniter
  <davecheney> all fail
  <perrito666> anastasiamac: I actually added anothe rcomment but apparently did not go trough reviewboard
  <perrito666> anastasiamac: try to change the uses of fmt.Errorf for errors.Annotate
  <anastasiamac> perrito666:of course! thnx :-) will do
  <anastasiamac> perrito666: constructive suggestions r an amazing feedback, especially atm ;-)
  <anastasiamac> perrito666: english hard atm too...
  <perrito666> anastasiamac: I am south american I am certainly in no position to criticize anyone's english
  <anastasiamac> perrito666: :-)
  <thumper> davecheney: master as of 9am this morning passed for me
  <thumper> davecheney: what is your last commit?
  <davecheney> thumper: i just pulled master
  <davecheney> i was trying to get my branch up to date and the tests were all failing
  <davecheney> so i switch to master
  <davecheney> and they are still failing
  <thumper> davecheney: let me try
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs:  None
  <thumper> davecheney: running now
  <davecheney> thumper: eg, http://paste.ubuntu.com/8726080/
  <davecheney> http://paste.ubuntu.com/8726089/
  <thumper> works here
  <thumper> smells of timing issue
  <davecheney> well shit
  <davecheney> ok, i'll break out my tools
  <davecheney> oh hang on
  <davecheney> these are the actions
  <davecheney> tests
  <davecheney> i raised bugs about this before
  <thumper> I thought this was parallel charm upload?
  <davecheney> thumper: what rev of charm.v2 do you have ?
  <davecheney> i don't trust godeps any more
  <thumper> um...
  <thumper> it should be charm.v4
  <thumper> I think
   * thumper looks
  <waigani> menn0_: added another step for the migrations. state/watcher.go update <coll>Watcher.merge() - change.Id -> localID
  <thumper> davecheney: yeah, I'm using charm.v4
  <davecheney> thumper: can you tell me the exact rev
  <bigjools> hey axw, can you remind me what causes juju to try and release twice when bootstrap goes awry?
  <waigani> menn0_: I think that one is a bug that will bite a few more times
  <davecheney> charm.v4, for all of the promise of gopkg.in
  <davecheney> doesn't ensure we're running the same code
  <menn0_> waigani: yeah, that's one that I catch while looking for transactions that touch the collection, but it's good to mention it explicitly
  <thumper> davecheney: 8b3cc836f54c2c78ce73d198100a30bb31d28392
  <thumper> davecheney: I have godeps working fine
  <davecheney> same
  <davecheney> ok
  <davecheney> scratch that
  <davecheney> thumper: could you run
  <thumper> davecheney: if you have "export JUJU_MAKE_GODEPS=true" and use "make check"
  <davecheney> thumper: i don't trust godeps
  <thumper> it makes sure your deps are up to date and runs the tests
  <davecheney> i know how to run the command
  <waigani> menn0_: yeah, saves some brain power. the change.Id being passed in will always be a DocID but needs to be a localID. I misunderstood that during the transaction sweep.
  <davecheney> i just don't trust it to work
  <davecheney> thumper: can you please try
  <thumper> it fails for me when I have different origins and upstreams
  <davecheney> env GOMAXPROCS=42 go test github.com/juju/juju/api/uniter
  <thumper> so I have to fetch upstream manually
  <thumper> davecheney: sure
  <thumper> davecheney: test passed here
  <menn0_> waigani, thumper: i'm currently trying to figure out a regression with the relation/relationscopes branch.
  <waigani> menn0_: want to pair?
  <menn0_> waigani: not at this stage thanks
  <menn0> waigani: I'm slowly making progress
  <waigani> menn0: cool. If you need a rubber duck let me know
  <menn0> waigani, thumper: when I destroy a service some relations aren't being deleted
  <menn0> waigani, thumper: i'll get there but it's painful
  <waigani> menn0: makes sense if it's looking for the wrong id right?
  <thumper> menn0: hmm... interesting
  <menn0> waigani: yes. i'm sure it's something like that but it's not obvious. it could be a watcher problem.
  <menn0> it only seems to affect subordinate relations AFAICT
  <waigani> menn0: checked the merge func? ;)
  <menn0> waigani: no but I think I just found it. missed a change in WatchRelations
  <waigani> nice
  <menn0> stupid prefix matching crap
  <waigani> ah right
   * thumper heads off to see specialist
  <thumper> bbl
  <menn0> waigani: the unit tests didn't notice by fluke
  <menn0> waigani: with an _id like this "60d7b1b0-4fea-452f-8748-70c05cf489ec:wp0:db mysql:server"
  <menn0> waigani: WatchRelations was matching on _ids starting with "mysql:" or containing " mysql:"
  <axw> bigjools: if bootstrap fails after acquiring, it'll release the bootstrap node; juju will then destroy the environment. whether it releases again depends on whether it's possible for MAAS to report that the bootstrap node still has the agent_name we gave it
  <waigani> ah fuck
  <axw> bigjools: now that I write that out in full, it probably never happens
  <waigani> menn0: is it a regex match?
  <menn0> waigani: it just happened to always match with the "containing" case in the test but the prefix match is no longer correct
  <bigjools> axw: heh, ok :)
  <menn0> waigani: no just a simple bit of string matching
  <menn0> waigani: I'll update the test and then fix
  <bigjools> axw: it's just that we have observed this behaviour but now we can't make it happen again
  <menn0> waigani: it's a bit of a fluke that I found it
  <menn0> waigani: which is worrying
  <bigjools> axw: in here: https://bugs.launchpad.net/maas/+bug/1386327
  <mup> Bug #1386327: Juju fails to destroy-environment with wipe enabled: Node cannot be released in its current state ('Disk erasing'). <MAAS:Incomplete> <https://launchpad.net/bugs/1386327>
  <waigani> menn0: might be worth while searching all the transactions
  <axw> bigjools: if the first release failed, then it would attempt again (just the once)
  <waigani> menn0: or at least adding it to the doc for something to check
  <bigjools> axw: right
  <menn0> waigani: yeah, I'll add it to the doc
  <waigani> MESS doc that is
  <waigani> menn0: you drew the short straw on that one
  <waigani> but good you spotted the problem, we can now keep an eye out for it
  <axw> bigjools: is it possible for release to return before agent_name is removed from a node?
  <bigjools> axw: yes
  <bigjools> in the bug, it has to wipe the disk before it returns, so sits (internally) at a different state.  The extra state is swallowed in the API to avoid confusing unaware API clients
  <waigani> menn0: how did you notice relations were left behind after a service was destroyed?
  <bigjools> axw: so you end up seeing what looks like a node that didn't start releasing, but it really is
  <menn0> waigani: the service wasn't completely being removed
  <menn0> waigani: the units and machines were but the service was being left behind in a Dying state with one relation intact
  <menn0> waigani: I've just updated the manual test instructions to include testing of service removal
  <waigani> menn0: ah right, fair call
  <davecheney> thumper: solved it
  <davecheney> out of date names dependency
  <davecheney> godeps didn't detect it
  <waigani> menn0: does mediawiki take ages to install for you?
  <waigani> it's been sitting on pending for well over 5min
  <menn0> waigani: I'm not sure exactly how long but that seems on the long side. also depends on you machine and how busy it is of course.
  <menn0> waigani: or, you've broken something :)
  <waigani> yeah...
  <waigani> might be time for a restart
  <davecheney> ok, now I have some other failing tests
  <davecheney> christ
  <perrito666> :| gmail now adds a button "view issue" and "view pull request" on the email list for PRs
  <perrito666> davecheney: want me to try?
  <thumper> davecheney: have you updated godeps recently?
  <perrito666> nite ppl
  <perrito666> thumper: btw, for that heat you wanted, near 30Â° its 22:51 and has been night for 3hs :p
  <thumper> perrito666: it is about 15 here
  <waigani> and has been day for more hours
  <menn0> waigani: I'm adding State.strictLocalID as part of this
  <menn0> it's like localID() but errors if the prefix isn't there
  <waigani> menn0: okay...
  <waigani> menn0: used when we assume the id passed in is a DocID
  <menn0> waigani: it's useful in cases where ids for multiple envs could be seen but you just want to deal with the ones for the State's environment
  <menn0> waigani: for example in the watchers
  <waigani> menn0: oh I see
  <menn0> waigani: i'll push this branch soon so you can see
  <thumper> arse
   * thumper needs git help
  <thumper> menn0: around?
  <menn0> thumper: yep
  <rick_h_> thumper: what'd you break?
  <thumper> my branch
  <rick_h_> stop doing that :P
  <thumper> my branch was 5 commits behind master
  <thumper> and I did "git reset master" without "git merge master"
  <thumper> and now I have a bunch of changes I didn't make
  <thumper> was trying the other rebase options
  <thumper> forgot it only really works when you have all of the master revisions
  <thumper> bugger
  <thumper> so I have some dangling head somewhere
  <thumper> that I want back
  <thumper> as the tip of the branch
  <thumper> how do I find it?
   * menn0 googles
  <rick_h_> oops
  <rick_h_> trying to think if you can get it without reflog'ing it
  <rick_h_> reset's not something I ever do except for git reset --hard HEAD when I want to really kill some work
  <rick_h_> thumper: highly recommend https://github.com/juju/juju-gui/blob/develop/HACKING.rst#syncing-your-feature-branch-with-develop-trunk for the whole 'sync with master to my feature branch' just s/develop/master for your use.
  <thumper> I blame menn0
  <rick_h_> hah
  <rick_h_> good call, find a scape goat
  <thumper> I'm just going to revert everything I know isn't mine
  <thumper> i looks like it will be quicker
  <menn0> thumper: i'm not sure why I get the blame. I don't think I encouraged you to use reset did I?
  <thumper> menn0: if you had never mentioned reset, I'd still be happily rebasing :P
  <thumper> menn0: and you fit my need of a scapegoat
  <menn0> thumper: ok, to get back to where you were
  <menn0> thumper: git reflog | head
  <thumper> I'm there already
  <thumper> :)
  <menn0> thumper: ok
  <thumper> however...
  <thumper> for future me
  <thumper> what does that do?
  <menn0> thumper: shows you the history of what you've been doing
  <thumper> ok, I see the commit I care about
  <menn0> thumper: to recover you find the hash from when you last committed to the branch
  <menn0> thumper: and then git reset --hard <hash>
  <thumper> huh
  <menn0> thumper: I just tried it out to be sure
  <thumper> is that all?
  <menn0> thumper: yep
  <rick_h_> reflog ftw
  <thumper> ok, will remember that for next time I screw up
  <menn0> thumper: the thing is that dangling references will eventually get garbage collected by git
  <menn0> thumper: so you don't want to leave it too long
  <menn0> thumper: after the screwup to recover
  <menn0> thumper: this link is helpful: http://effectif.com/git/recovering-lost-git-commits
   * thumper bookmarks
  <menn0> thumper: best quote: "it's like time travel, only cheaper" :)
  <davecheney> thumper: no i have not updated godeps
  <davecheney> i think the last time I did was when we were on that sprint in NZ
  <menn0> waigani, thumper: finally! http://reviews.vapour.ws/r/284/
  <menn0> waigani: reviewing your charms branch now
  <waigani> menn0: thanks, I've got minUnits up too
  <waigani> menn0: let me nut out this problem then I'll get onto your branch
  <menn0> waigani: np
  <menn0> waigani: I think we need to a EnvUUID method to state to avoid repeating "st.EnvironTag().Id()" all over the place
  <waigani> menn0: agreed
  <ericsnow> menn0: when you get a chance, I'd love to get your feedback on http://reviews.vapour.ws/r/243.
  <menn0> ericsnow: will do. I had seen that review request - just hadn't gotten there yet.
  <ericsnow> menn0: no worries, I appreciate your time :)
  <davecheney> can someone help please
  <davecheney> http://paste.ubuntu.com/8727429/
  <davecheney> ^ this goes into stress.bash
  <davecheney> cd $GOPATH/src/github.com/juju/juju/apiserver/client
  <davecheney> bash stress.bash
   * menn0 will run
  <davecheney> ta
  <davecheney> there is a logical race between uploading the charm and whatever the test is doing
  <menn0> davecheney: it's still running but I see a clientSuite.TestAddCharmConcurrently failure
  <menn0> davecheney: done now
  <davecheney> http://paste.ubuntu.com/8727454/
  <davecheney> failing hiallriously
  <menn0> davecheney: that's what I see
  <davecheney> i have anothe rfailure
  <menn0> davecheney: actually, on closer inspection my failure is different
  <menn0> davecheney: http://paste.ubuntu.com/8727496/
  <davecheney> https://bugs.launchpad.net/juju-core/+bug/1386968
  <mup> Bug #1386968: apiserver/client: clientSuite.TestAddCharmConcurrently failure do to local race <juju-core:New> <https://launchpad.net/bugs/1386968>
  <waigani> ugh
  <menn0> ericsnow: ping
  <ericsnow> menn0: hey
  <menn0> ericsnow: so doesn't option 2 involve using --oplog
  <menn0> ericsnow: ?
  <ericsnow> menn0: yeah
  <menn0> ericsnow: but it looks like it's not being used when specific dbs are backed up
  <ericsnow> menn0: correct
  <menn0> ericsnow: so what's the plan?
  <ericsnow> menn0: with option 2 we don't dump specific DBs
  <menn0> ericsnow: ok. so why leave the support in there for backing up individual dbs?
  <ericsnow> menn0: I'm hopeful that we will be able to stick to option 2
  <ericsnow> menn0: in case option 2 doesn't pan out :)
  <menn0> menn0: right. got it.
  <waigani> menn0: started looking but Bella will be here in 10min, I'll be doing the dad thing
  <menn0> waigani: np
  <menn0> waigani: i have an errand to run shortly myself
  <waigani> menn0: btw cleanup docs has an id type of bson.ObjectId - which screws up our upgrade step assumtions
  <waigani> menn0: not a huge problem, but I'll have to do some rejigging tomorrow
  <thumper> owwie...
   * thumper goes to lie down
  <thumper> shoulder is really starting to ache now
  <menn0> waigani: yeah, I've seen that. let's discuss tomorrow. there's a few ways we could tackle that.
  <thumper> go an injection in it this afternoon
  <thumper> hopefully should settle down in the next two weeks
  <thumper> here's hoping
  <waigani> menn0: sounds like a plan
  <menn0> ericsnow: are you sure this branch even compiles?
  <menn0> ericsnow: mongoDumper.Dump looks incomplete with both unused and undeclared variables
  <menn0> ericsnow: and mongoDumper.strip() has no code
  <ericsnow> menn0: yeah, I created the review request thinking I'd wrapped things up
  <ericsnow> menn0: I have it fixed up and will be updating momentarily
  <menn0> ericsnow: ok
  <waigani> menn0: shouldn't we always use strictLocalID unless we have a reason not to?
  <davecheney> tada, data race http://paste.ubuntu.com/8727559/
  <davecheney> func (s *MockCharmStore) WithTestMode(testMode bool) charm.Repository { s.TestMode = testMode return s
  <davecheney> }
  <davecheney> man
  <davecheney> we have _SO MANY_ of these races
  <menn0> :(
  <davecheney> oh wow
  <davecheney> that methods is pants on head retarded
  <davecheney> it doens't return you a _new_ repository with a test flag set
  <davecheney> it updates the repo that you passed to it ...
  <davecheney> oh, and fantastic, the callers of that expect this behavior
  <davecheney> fml, https://bugs.launchpad.net/juju-core/+bug/1386968/comments/2
  <mup> Bug #1386968: apiserver/client: clientSuite.TestAddCharmConcurrently failure do to local race <juju-core:Confirmed> <https://launchpad.net/bugs/1386968>
  <ericsnow> menn0: I've updated that patch
  <ericsnow> menn0: and am signing off
  <menn0> ericsnow: ok. i'm looking at your other one now
  <ericsnow> menn0: cool
  <menn0> ericsnow: I'll try to finish reviewing both
  <ericsnow> menn0: thanks again
  <menn0> ericsnow: np.
  <wallyworld_> axw: i've found a problem with the recent work to validate credentials, and the use of Open vs Prepare - it does break validate-tools
  <wallyworld_> you want to run validate-tools before deploying an environment
  <wallyworld_> to see where the tools will come from
  <wallyworld_> but that's not possible now, since the environment is not prepared, and hence is missing derived attrs
  <wallyworld_> like control-bucket for ec2
  <wallyworld_> in any case, control-bucket should now be obsolete for deploying new environments
  <axw> wallyworld_: yuck, that's a problem. I want to be able to change the manual provider so we can specify the bootstrap host in placement directives, but that involves restricting Prepare to bootstrap
  <davecheney> menn0: if you read this and are not filled with dredd, http://reviews.vapour.ws/r/286/diff/#
  <davecheney> let me know
  <wallyworld_> axw: yeah. the environment is created in order to get parameters specific to the environment to use to search
  <menn0> davecheney: I have to stop for a bit soon so I won't be able to get to this for a few hours
  <wallyworld_> axw: given the validation commands are now broken, short term to get alpha3 released, we'll need a fix that works with what we have
  <wallyworld_> i have not raised a bug yet
  <davecheney> menn0: that's ok
  <davecheney> you don't need to review it
  <axw> wallyworld_: revert to using Prepare for now I guess, will just need to change the metadata tests to set up a mock ec2 endpoint
  <wallyworld_> axw: will that revert affect the credentials validation
  <axw> nope
  <wallyworld_> i can change to using Prepare in my current redo of the tools stream stuff
  <wallyworld_> i found this issue by testing that work
  <axw> wallyworld_: if it's not too much hassle. otherwise let me know and I'll fix it
  <wallyworld_> it's no problem
  <wallyworld_> i'm in the area anyway
  <menn0> davecheney: i've just had a quick look and although it's not pretty it does look reasonable
  <davecheney> hey
  <davecheney> it's much better
  <davecheney> the caller doesn't need to remember to defer this magic cleanup function
  <davecheney> that's something the test suite should do
  <davecheney> and more improtantly
  <davecheney> it highlights just how horrible that test is
  <davecheney> updating a shared mutable package level variable
  <menn0> davecheney: you know that CleanupSuite (embedded in BaseSuite) already provides something very similar to what you've done with afterFunc?
  <menn0> davecheney: there's an AddCleanup method
  <davecheney> menn0: cool
  <davecheney> i was going to use that
  <davecheney> but for some reason I thought it was only on the suites that had something to do with mongo
  <davecheney> i'll fix the patch to use AddCleanup
  <menn0> davecheney: ok
  <menn0> davecheney: I'll be back on later but I need to go know
  <menn0> now
  <davecheney> thanks for pointing me to the place it lives
  <wallyworld_> axw: sadly, the metadata commands fail with an access credentials check error - i wonder if the verifyCredentials calls could be moved to Open() instead of Prepare(). I wonder if that's too late in the bootstrap process
  <axw> wallyworld_: the bug (raised by mark) requested that we do this ASAP in the bootstrap process - and it is kinda necessary to ensure we get the error message early on
  <axw> wallyworld_: what is going wrong with the metadata commands?
  <wallyworld_> axw: yeah, i understand the context. the metadata commands can be run without env credentials, so that verify check fails
  <axw> ah.
  <wallyworld_> we could add a method to the BootstrapContext
  <axw> wallyworld_: can we do this some other way? without opening an environment? cos really, we're not exactly opening a fully defined environment
  <wallyworld_> right now, an env instance is needed to ask for things from
  <axw> yes... can we change that.
  <wallyworld_> not that easily, but yes
  <wallyworld_> because creating an env manipulates config
  <wallyworld_> i'll have a poke around, but there's potentially a fair bit to unpick
  <axw> yeah I took a look, doesn't look like something we can do easily
  <wallyworld_> axw: could add a method to BootstrapContext - ValidateCredentials() bool
  <wallyworld_> would work for now
  <wallyworld_> so that we can get alpha3 out
  <wallyworld_> i also gotta make sync-tools backwards compatible, bit of a pita
  <axw> wallyworld_: we can make it bootstrap only I suppose, would be nice not to but it'll do for now
  <wallyworld_> yeah, agreed it's not nice :-(
  <wallyworld_> i'll try it after fixing sync tools
  <wallyworld_> lots of touch points there
  <dimitern> morning all
  <axw> wallyworld_: FYI I've not sent a PR with my metadata changes yet, because I want to explore the usage of it a bit first. I would like to avoid churning packages
  <wallyworld_> axw: +1
  <wallyworld_> axw: i'm almost finished the tools rework, including credentials verification fix. keep getting mongo replicaset errors testing sadly
  <wallyworld_> timeouts mainly
  <axw> okey dokey
  <axw> gotta make dinner shortly, ping when it's ready and I'll take a look
  <dimitern> wallyworld_, hey
  <wallyworld_> hi
  <dimitern> wallyworld_, so re that network-bridge-related bugs
  <dimitern> wallyworld_, that guy probably misread or misunderstood what the code he pasted does
  <wallyworld_> likely
  <wallyworld_> i'm not 100% either
  <dimitern> wallyworld_, but seeing lxcbr0 in there *at all* is wrong and should be fixed
  <wallyworld_> dimitern: which bit?
  <dimitern> wallyworld_, in fact I'm thinking of doing that today, as this is likely the 15th bug report related to that piece of code in the maas provider
  <fwereade> ok, uniter tests passing again, popping out to shops, bbs
  <wallyworld_> dimitern: that would be awesome
  <wallyworld_> i don't know enough about networking
  <dimitern> wallyworld_, there is some code in there that aims to "restore" /etc/network/interfaces the way they *should* look like in a stock trusty install, but that's really not the point - we shouldn't be using the bridge at all, but first we should respect the network-bridge setting
  <dimitern> at least
  <wallyworld_> sounds reasonable
  <TheMue> morning
  <dimitern> wallyworld_, cool, thanks
  <dimitern> TheMue, morning
  <wallyworld_> dimitern: no, thank you, i have no idea really
  <dimitern> wallyworld_, well, I have to admit the simple solution of just using "network-bridge" in there as a temporary fix (so at least it's configurable) simply didn't occur to me earlier, and we can't just remove the bridge creation until we've implemented the static ip reservation api maas 1.7 gives us; so tl;dr - thanks for pointing that out :)
  <wallyworld_> np, it was an educated guess :-)
  <mattyw> morning all
  <voidspace> morning all
  <TheMue> mattyw, voidspace: heya
  <mattyw> TheMue, morning morning
  <fwereade> voidspace, would you be free to take a look at a couple of gsamfira's PRs? https://github.com/juju/juju/pull/782 https://github.com/juju/juju/pull/833
  <fwereade> voidspace, gsamfira: I should be able to get to them later, but I realised I broke something in the uniter and I'm focused on unbreaking it
  <gsamfira> fwereade: heh, no worries
  <fwereade> and if anyone *else* is free -- jam, dimitern maybe? -- http://reviews.vapour.ws/r/288/diff/# is the code, on which I'd really a appreciate a logic/style pre-review, but I haven't yet fixed all the tests (sigh)
  <jam> fwereade: looking
  <jam> fwereade: is "State().Members" a map and you want the keys to be in the member name slice ?
  <jam> (I wonder if memberName, _ :=  would be an obvious way to write that)
  <jam> at least, I first read it as "Members is a slice" and we just want to copy the slice, but then iteration doesn't work like that :)
  <jam> fwereade: I also thought: "contextRelations := map[int]*ContextRelation{}" was bad form because you end up with a nil map, you actually want make() there, don't you?
  <jam> hmm. play.golang says I'm wrong
  <jam> var x map[int]string is a nil map, but x := map[int]string{} is an empty map
  <voidspace> jam: have you picked up both PRs or should I look at one?
  <jam> voidspace: I'm looking at fwereade's not gsamfira's
  <voidspace> jam: ah, cool
  <voidspace> fwereade: sure, I'll look at the gsamfira reviews
  <fwereade> jam, yeah, I've never understood the preference for make()ing maps over just using a literal
  <jam> fwereade: well, because "var foo map" doesn't work
  <jam> fwereade: so it would seem that the way to create a map is with "make"
  <fwereade> jam, sure, but the literal syntax works fine, so I'm not sure why people prefer to avoid it
  <jam> fwereade: in *my* case, not knowing (or expecting) it to wokr
  <jam> work
  <fwereade> jam, it's essentially just the same as slices
  <fwereade> jam, var gives nil, type{} gives empty
  <jam> fwereade: except empty slice lets you append as well as a nil slice
  <jam> and call len
  <fwereade> jam, indeed, *that*'s the reason people use var foo string{} :)
  <jam> and iterate
  <jam> etc
  <fwereade> jam, len and range work on nil maps too
  <fwereade> jam, it's just insertion
  <jam> fwereade: right, but you can't add stuff
  <jam> which is why it is odd
  <jam> "the things I want to do with a slice" function the same with nil or empty
  <jam> "the things I want to do with a map" all work the same except for, ya know, adding stuff
  <fwereade> jam, it's always felt a bit warty to me
  <fwereade> jam, defensible ofc but a bit inelegant
  <jam> fwereade: question about "members" cache vs "others" cache
  <jam> why do we clean out others here and there, does that effect our transactionality ?
  <jam> (things which we touch are never refreshed
  <voidspace> dimitern: the network id that AllocateAddress will take, is that a "provider specific" id (i.e. one we can pass directly to ec2)
  <dimitern> voidspace, that's the idea, yes
  <dimitern> voidspace, for ec2 it probably will be a subnet-id
  <fwereade> jam, so, the idea is that you *can* always get settings for remote units that aren't currently part of the relation
  <dimitern> voidspace, but it's unused for now - intended to be future-proof :)
  <fwereade> jam, but that we expect that to be "rare"
  <jam> fwereade: I don't disagree on either of those points
  <fwereade> jam, so we persist them only for the lifetime of the context
  <jam> is this a lifetime of data cached in memory issue ?
  <fwereade> jam, it doesn't affect the guarantees we make
  <fwereade> jam, yeah
  <fwereade> jam, it's always been, and always will be, possible to see relation data "from the future"
  <fwereade> jam, but within a given context, once you've asked for some settings, they won't change
  <jam> fwereade: sure, but what causes Prune() to be called
  <jam> as if you do "relation-get NOT_A_MEMBER" and then trigger a Prune and then another "relation-get NOT_A_MEMBER" you will get different data
  <jam> (*can* get different data)
  <voidspace> dimitern: ah, we're designing in imaginary future use cases...
  <dimitern> voidspace, it's not imaginary, it's in the draft model :)
  <voidspace> dimitern: :-p
  <jam> fwereade: also "relation-get A_MEMBER" and then mark that as no longer a member and "relation-get WAS_A_MEMBER" will give different data
  <fwereade> jam, we Prune to the current membership list at context creation time
  <fwereade> jam, calling prune/remove/invalidate while a context is in play will not have happy consequenes, I agree
  <jam> fwereade: so I do see one possible bug, which is if you do "context.Settings("x/2"), then context.InvalidateMember("x/2"), and then context.Settings("x/2") it won't reread the data
  <jam> because it will pull it out of "other"
  <fwereade> jam, how would it get into "other" in the first place?
  <jam> fwereade: the first Settings call
  <fwereade> jam, TestInvalidateMemberUncachesMemberSettings
  <jam> fwereade: but it isn't a member
  <jam> until you call InvalidateMember
  <fwereade> ah!
  <fwereade> wait
  <fwereade> surely it will
  <jam> fwereade: I think it is just a "delete self.other[memberName]" in InvalidateMember
  <fwereade> ah yes I see
  <jam> fwereade: RelationCache.Settings unconditionally falls back to other
  <fwereade> jam, thanks, well spotted
  <jam> if settings is nil
  <fwereade> jam, yeah
  <fwereade> jam, I think I'd prefer:
  <fwereade> TestInvalidateMemberUncachesMemberSettings
  <fwereade>         if !isMember {
  <fwereade>             settings = cache.others[unitName]
  <fwereade>         }
  <fwereade>         if settings == nil {
  <fwereade>             var err error
  <fwereade>             settings, err = cache.readSettings(unitName)
  <fwereade>             if err != nil {
  <fwereade>                 return nil, err
  <fwereade>             }
  <fwereade>         }
  <jam> fwereade: lgtm
  <fwereade> jam, cheers
  <jam> it still seems a bit odd that you might have the data sitting around
  <jam> fwereade: but still test-worthy however it is solved
  <fwereade> jam, agreed
  <jam> fwereade: so going back to understand the lifetime of these objects
  <fwereade> jam, I think the fundamental problem is an assumption that I'll always call Prune before Remove or Invalidate
  <jam> fwereade: I'm more thinking about when does the cache actually get invalidated
  <voidspace> dimitern: is the default-vpc represented as a network interface in ec2?
  <fwereade> jam, expand please?
  <jam> so we have this data which is cached, which if you explicitly call InvalidateMember will refresh the cache
  <jam> but otherwise the cache has unbounded lifetime
  <jam> it is created when you start a context
  <jam> (and pruned at that point)
  <jam> I would have thought a cache would only exist for the lifetime of a contextx
  <dimitern> voidspace, no, it's an account attribute containing a vpc id in the form "vpc-xxxx" an interface would have id like "eni-xxxx"
  <fwereade> jam, consider charms which real the full state of a given relation to determine their config
  <fwereade> jam, they enter a relation with 10 remote units
  <voidspace> dimitern: ok, so we need to tell ec2 which network interface to assign the ip address to
  <fwereade> jam, even if they don't have a -joined hook implemented, they'll run 10 -changed hooks
  <voidspace> dimitern: do we have to get all of them and work out which subnet the address belongs to and which interface the subnet is on?
  <fwereade> jam, and in the Nth hook they'll grab N sets of settings
  <fwereade> jam, despite N-1 of them being ones they got just a second or so ago
  <jam> fwereade: sure. though when the hooks trigger tomorrow won't they want updated data ?
  <jam> Is this just live until the current set of hooks finish ?
  <fwereade> jam, they *might*, but we're guaranteed to run a -changed hook, at some point, for any that have changed their data
  <fwereade> jam, so we invalidate just before running the -changed hook and call it good
  <jam> fwereade: but if you do so, then isn't that still true for the case of running 10 changed hooks ?
  <jam> anyway, otp now, I'll need to context switch, but I want to discuss it with you just to get an understanding.
  <fwereade> jam, ok, I will quickly dump in here, ping me when you want
  <fwereade> 10 units in scope with cached settings
  <fwereade> all 10 remote units change their settings
  <fwereade> we trigger a changed for the first one, and invalidate its settings
  <fwereade> that changed hook asks for all settings, downloads exactly one (the invalidated one)
  <fwereade> changed for the second one, with invalidated settings
  <fwereade> again asks for all settings, only needs to hit the api for one set
  <fwereade> <repeat>
  <fwereade> total impact: 10 settings gets over the 10 hooks; unit saw a consistent -- if not strictly accurate -- view of the changes, and is now fully up to date
  <fwereade> (and next week, when just one of them changes settings, we again just download one set of changed settings)
  <fwereade> jam, see above
  <jam> fwereade: as long as the jujud daemon wasn't restarted
  <fwereade> jam, so, yes, it would be nice to cache them on disk
  <fwereade> jam, I'm not doing that right now
  <jam> fwereade: well, our API shouldn't be so expensive that that is necessary, should it?
  <fwereade> jam, depends how big the services get
  <fwereade> jam, all data is big data when you have enough of it ;p
  <fwereade> jam, but, yes, I don't see a strong need to cache it on disk
  <jam> fwereade: I'm personally more in favor of less caching until you know that it will help
  <fwereade> jam, I think people have asked for it once or twice
  <jam> as cache invalidation is hard
  <fwereade> jam, agreed :)
  <fwereade> jam, I think this case is well-understood, though
  <jam> and tends to do well on benchmarks but when you have real load hit rates may be low
  <fwereade> jam., not to mention pre-existing ;)
  <fwereade> jam, quite so
  <jam> IIRC some of launchpad spend up when they stopped caching some stuff because the extra cached data had low hit rates and was pushing out good data
  <jam> very different use acse
  <jam> case, though.
  <fwereade> jam, however, given that the gimme-all-the-relation-data tool is a pretty frequent request
  <fwereade> jam, and seems to be the first thing people write when doing helpers for a new language
  <fwereade> jam, I'm reasonably confident that it maps well to many use cases
  <natefinch> fwereade: Have you looked at the output variables spec? https://docs.google.com/a/canonical.com/document/d/1Bz8YGJL7OcVgcK-K47MEH7-vlJVPvm34wKUPWO8FQko/edit
  <natefinch> fwereade: I'm supposed to be sizing this for alexis for this cycle, and I want to make sure the spec actually describes the thing we want to build
  <natefinch> fwereade: ahh, hmm, hazmat left a comment on that one about there being a new document from brussels, but I can't seem to find it.  Fun.
  <fwereade> natefinch, dammit
  <fwereade> natefinch, nor can I
  <fwereade> natefinch, niemeyer put it together, would you ask him when he comes online?
  <natefinch> fwereade: sure
  <wallyworld_> axw: finally got it sorted after having dinner, if you have time, or else it can wait till tomorrow https://github.com/juju/juju/pull/988 just the 2nd commit really
  <natefinch> fwereade: also - charm level constraints have container and architecture blacklists..... how would those actually work, from a user's point of view?  we don't generally put things in containers automatically, so the only time we do is when a user specifically tells us to, at which point, do we just warn them that the charm probably won't work there?  That doesn't seem like the best UX, but I'm not sure how else to implem
  <natefinch> ent it
  <wallyworld_> fwereade: i'd love 5 mins of your time if you have it
  <fwereade> wallyworld_, sure, start a hangout? natefinch, with you properly in 5?
  <natefinch> fwereade: sure sure
  <wallyworld_> fwereade: yeah, our 1:1
  <fwereade> wallyworld_, regular-catchup?
  <wallyworld_> yup
  <fwereade> wallyworld_, I think I'm in there
  <wallyworld_> https://plus.google.com/hangouts/_/canonical.com/ian-william
  <fwereade> natefinch, fwiw, I would be treating it like series
  <hazmat> natefinch, re output variables?
  <hazmat> natefinch, https://docs.google.com/a/canonical.com/document/d/1JcWkE4SNxXuFClZGBcwnU3w13IpRU1yxMhddQG6mKyE/edit#heading=h.swixk07ju5hq
  <natefinch> hazmat: thanks.... google docs couldn't find that one for some reason
  <hazmat> natefinch, odd.. its setup for find and edit by everyone @ canonical
  <natefinch> hazmat: yeah, I don't know why search wasn't finding it
  <natefinch> hazmat: of course now that I've opened it, search finds it just fine :/
  <natefinch> hazmat: great, this looks much more like what I was expecting than the old spec
  <fwereade> natefinch, I think there are some things to worry about with that spec -- the concept of a "schema" is so tortured as to be a positive impediment to understanding
  <fwereade> natefinch, but I will make my comments on the doc (and try to talk to gustavo about it tomorrow)
  <fwereade> natefinch, anyway, re blacklists
  <fwereade> natefinch, IMO we shouldn't even try to deploy things in places they declare they won't work
  <natefinch> fwereade: that seems to make sense
  <fwereade> natefinch, in particular I don't think arch/container blacklists are even constraints
  <fwereade> natefinch, for constraints in charms, I think there's more wiggle room -- the constraints in charms are *recommendations*, that we will follow if we can but deploy anyway and warn if we can't meet them
  <fwereade> natefinch, and they are ofc just suggestions, and completely ignored if the service has real constraints
  <fwereade> natefinch, indicating that the administrator knows he knows better
  <fwereade> natefinch, that said
  <fwereade> natefinch, constraints in charms are not currently a thing AFAIK?
  <natefinch> fwereade: my thought was that charm constraints would turn into real constraints during deploy, but get overridden by any other constraints specified by the CLI etc.
  <natefinch> fwereade: correct
  <natefinch> fwereade: it is a thing I am being tasked with writing :)
  <natefinch> fwereade: well, my team :)
  <hazmat> fwereade, unrelated.. but was curious.. what's the status on leader election?
  <fwereade> hazmat, ramping up
  <fwereade> hazmat, I'm churning on the uniter
  <hazmat> fwereade, k, thanks
  <fwereade> hazmat, it's taking a bit of time to actually get us to having a global sense of hook ordering
  <fwereade> hazmat, and katco is about to make a start on the server side
  <fwereade> natefinch, given that container type and arch are already constraints though
  <fwereade> natefinch, probably it makes sense to draw a distinction between preferred and required constraints in a charm
  <natefinch> fwereade: *nod* there's RAM and rootdisk charm-level constraints in the spec as well
  <natefinch> fwereade: those are a little easier to grok than the arch and container blacklists, since blacklists are the opposite of what we do with constraints now
  <fwereade> natefinch, provider-specific constraints are gonna be fun, too
  <fwereade> natefinch, do you overlap with thumper?
  <natefinch> fwereade: yeah, late in the day for a couple hours
  <fwereade> natefinch, please talk to him about the upcoming distinctions between provider/account/environment, and think through the implications
  <fwereade> natefinch, I will hopefully be in a position to pop on late tonight so I will join you if I can
  <fwereade> natefinch, I would suggest it's something to write into the spec but explicitly carve out as a "phase 2" task
  <natefinch> fwereade: cool.  I was hoping to push off provider-specific constraints until v2, but not sure if everyone will agree with that. :)   there's just instance-type and cloud-blacklist
  <natefinch> (v2 of charm level constraints that is)
  <fwereade> natefinch, ah, didn't know about cloud-blacklist
  <fwereade> natefinch, to my mind, that implies you need to think it through properly for v1
  <natefinch> yeah....
  <fwereade> natefinch, the big issue os how we identify clouds
  <natefinch> I can think it through for v1, I just don't want to write it for v1 ;)
  <natefinch> fwereade: yes, since right now openstack is openstack, not HP vs. canonistack
  <fwereade> natefinch, the core question is "what is a cloud" :) ...and how do we clearly identify them
  <fwereade> natefinch, exactly
  <fwereade> natefinch, not to mention that azure != azure-in-china
  <natefinch> bleh
  <fwereade> natefinch, and that in some ways that matters (does an account bind to a cloud?)
  <fwereade> natefinch, and in some ways it doesn't (instance sizes are probably the same in the azure case at least)
  <fwereade> natefinch, (although, yeah, instance types across openstacks will surely vary)
  <fwereade> natefinch, I think it's about "provider" as a concept distinct from "provider *type*"
  <fwereade> natefinch, anyway thumper is at the sharp end re providers/accounts, and your notions of "what is a cloud" will need to align
  <fwereade> natefinch, is that enough context for you to have a useful chat with him, do you think?
  <natefinch> fwereade: yep, thanks, that's very helpful.
  <fwereade> natefinch, cool
  <jam> fwereade: so I think I've covered the logic question and aside from that one bug, lgtm
   * TheMue has to step out, yesterday mentioned appointment in garage. bbiab
  <perrito666> morning all btw
  <natefinch> morning perrito666
  <fwereade> jam, lovely, thanks
  <natefinch> The longer I spend trying to come up with  estimates for work items, the larger the estimates get
  <perrito666> natefinch: it usually is the other way around
  <natefinch> perrito666: no way... the longer I spend on it, the more paranoid I get that I'm not estimating enough time.  I don't think anyone in the history of programming has ever over-estimated the amount of time something will take to implement.
  <perrito666> a truck full of oxygen tanks just took a speedbump at 40-60 ... I am still trying to have my blood unfreeze.
  <perrito666> that is km not miles
  <natefinch> heh
  <perrito666> anyone feels like reviewing this overly trivial change? http://reviews.vapour.ws/r/289/diff/#
  <perrito666> brb, mechanic
  <voidspace> fwereade: ping
  <fwereade> voidspace, pong
  <voidspace> fwereade: are you up to date with the latest strategy for reboot and the clearing of the reboot flag?
  <voidspace> fwereade: the latest strategy is to clear the flag *immediately before* reboot
  <voidspace> fwereade: so there is still a very small window for jujud to die and the reboot not to happen
  <fwereade> voidspace, I think so, yeah -- so long as we schedule a delayed reboot before clearing the flag, and then do an immediate reboot afterwards to save time, I'm ok with it
  <voidspace> fwereade: cool
  <voidspace> fwereade: some of your comments on the PR, from a while back,  haven't been replied to since discussions
  <voidspace> but they're now out of date
  <fwereade> voidspace, it leaves the possibility that someone will cancel the reboot but I can live with that
  <voidspace> right
  <voidspace> fwereade: by "someone", you mean a user logged into a machine?
  <fwereade> voidspace, yeah
  <voidspace> heh
  <voidspace> they get the mess they asked for then
  <fwereade> voidspace, or, y'know, just any old manifestation of the universe's desire to teach us humility
  <voidspace> :-)
  <voidspace> I have quite a few friends like that
  <voidspace> or in fact, just juju itself...
  <kwmonroe> kv7tbgks
  <kwmonroe> welp.. that's the berries
  <jam> kwmonroe: that won't be the first password that this irc channel has voided  :)
  <kwmonroe> ugh.  just got good at typing it too.
  <jam> kwmonroe: if it had been longer lived, I would just say "you probably needed to rotate your passwords anyway"x
  <ashipika> Helloâ¦ could somebody help me out with the lxc provider?
  <jam> ashipika: what's up ?
  <ashipika> my question is: is there a performance benefit to reusing existing containers for new units instead of creating new ones.. especially in the case of lxc-clone-aufs being defined in the environments.yaml
  <ashipika> jam: i'm trying to fix a really annoying bug..
  <ashipika> jam: where there's a leftover container that fails to start in a previous deploy and is not cleaned up during destroy-envrionment
  <ashipika> jam: so when i bootstrap a new environment and try to add a unit, it reuses the leftover container.. and of course, fails to start again
  <ashipika> jam: a workaround i wrote is to retry by starting a new container.. and it works.. but the questions is why reuse containers in the first place..
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs:  1387172
  <sinzui> natefinch, :(  the two restore tests are failing. I thought they would be fixed with the other fixes landed yesterday. Can you arrange for someone to look into bug 1387172
  <mup> Bug #1387172: restore consistently fails <backup-restore> <ci> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1387172>
  <perrito666> sinzui: looking
  <bac> sinzui, mgz: jenkins-github-lander has been updated.  bin/lander-check-pulls now accepts a --debug flag to be a little less mysterious about what it is doing.
  <mgz> bac: I'll have a look
  <natefinch> sinzui: sure
  <perrito666> natefinch: o/
  <sinzui> perrito666, sorry, I know restore code didn't change
  <perrito666> sinzui: worry not, I am pretty sure I know what changed
  <perrito666> natefinch: sinzui I am pretty sure this is because of the changes on _id field for machines
  <marcoceppi> what's the default scope?
  <marcoceppi> on a relation
  <mattyw> has anyone seen/ is fixing this failure? http://paste.ubuntu.com/8733212/. It's very intermittent
  <mgz> mattyw: that one is new to me
  <mattyw> mgz, I'm being lucky today, It happened here: http://juju-ci.vapour.ws:8080/job/github-merge-juju/1053/console
  <mgz> I saw another one of yours on that too
  <mattyw> mgz, and I got a different failure here http://juju-ci.vapour.ws:8080/job/github-merge-juju/1052/console
  <mattyw> mgz, yup, pretty* sure it's not my change doing it
  <mgz> seems worth a bug with intermittent-failure tag
  <perrito666> mattyw: mgz devecheney was complaining of the same last night iirc
  <perrito666> mattyw: updated dependencies later?
  <perrito666> lately?
  <mgz> perrito666: he got it in a landing job, which should always have deps as specified
  <mattyw> perrito666, yep, I've not actually seen the test fail locally though
  <natefinch> perrito666: yeah, that id change seems like it screwed up a bunch of stuff
  <perrito666> natefinch: it did
  <perrito666> I am also not sure if my fix is right, I am for the moment just trying to figure if I got the issue right
  <natefinch> perrito666: this is why you don't encode metadata into an id field
  <mgz> backup/restore on ci is currently very unhappy
  <perrito666> mgz: I know I know, working on it
  <mgz> perrito666: <3
  <perrito666> man, testing old restore requires some patience
  <katco> hey everyone, i'm ocr today, but i've been directed to try and fix a bug before our cut tomorrow, so i'll be working on that. sorry for any inconvenience.
  <perrito666> its ok we can optically recognize characters for you
  <natefinch> fwereade: you have any time to talk about exposing zones to charms? There's a spec, but it's huge and not super-focused
  <perrito666> this is how waiting for the old restore to finish feels :p https://www.youtube.com/watch?v=qLlUgilKqms
  <natefinch> hehh
  <perrito666> who wants to be the hero to unlock CI with a mighy review? http://reviews.vapour.ws/r/292/
  <mgz> perrito666: done
  <perrito666> tx
  <fwereade> natefinch, probably in an hour?
  <natefinch> fwereade: sure
  <perrito666> sinzui: mgz how far are we from a new run of functional-backup-restore* ?
  <sinzui> perrito666, 1h
   * perrito666 cries
  <sinzui> perrito666, this is a rebuild of the last rev
  <perrito666> sinzui: tx
  <voidspace> and back...
  <TheMue> voidspace: made a knot into your internet wire?
  <voidspace> TheMue: they actually fixed it!
  <voidspace> TheMue: I'm back up to 15mbs
  <voidspace> after six months...
  <voidspace> TheMue: only 1mbps upstream, but it's still double yesterday...
  <TheMue> voidspace: is this the promised bandwidth?
  <voidspace> TheMue: yep
  <TheMue> voidspace: that's good. for a longer time I only had about a half of the promised bandwidth, the backend has been too slow for all who ordererd a higher bandwidth
  <TheMue> voidspace: now I'm at 80 to 90% downstream and 100% upstream most of the times
  <TheMue> voidspace: means 40 to 45 mbits down and 10 up
  <voidspace> TheMue: yeah, that's pretty normal to get much less than the "maximum" you pay for
  <TheMue> voidspace: exactly
  <voidspace> TheMue: in theory this should be a 12mbit line - it's currently 15mbit and I'm actually getting 12
  <voidspace> TheMue: 4045mbits
  <voidspace> TheMue: lucky man :-)
  <TheMue> voidspace: I am, yeah. but you know, it never can be enough *lol*
  <voidspace> right
  <voidspace> true enough
  <voidspace> gsamfira: ping
  <voidspace> networkInterface := instancesResp.Reservation.Instances[0].Interfaces[0]
  <voidspace> heh :-)
  <gsamfira> voidspace pong
  <voidspace> actually I want
  <voidspace> networkInterfaceId := instancesResp.Reservation.Instances[0].Interfaces[0].Id
  <natefinch> fwereade: in moonstone whenever you're ready: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1
  <perrito666> sinzui: mm, something is odd in the test run its failing even before getting to the test per se
  <sinzui> perrito666, :( Hp might be having issues. Your rev is still queued to be tested
  <natefinch> hazmat: do you have time in the next hour to talk about exposing zones to charms?  It's on my list of stuff to do this cycle and I want to make sure I understand what needs to be implemented
  <hazmat> natefinch, i have time now for 12m
  <natefinch> hazmat: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1
  <perrito666> can I get a second review on http://reviews.vapour.ws/r/289/? its one line
  <natefinch> perrito666: I guess there's no test for that?
  <natefinch> perrito666: maybe there should be?
   * natefinch gives more work to Horacio, as usual.
  <perrito666> natefinch: yes, good idea
  <perrito666> natefinch: its my job to do the work you give me :p not that I am going to complain
  <natefinch> :D
  <natefinch> gah, why the hell can't people document their friggin' features?
  <natefinch> latest offender:  zone placements   evidently juju deploy zone=foo is a thing, but you wouldn't know it from juju help deploy
  <perrito666> natefinch: well it wouldnt be fair, if they did I would be the only one with more work
  <perrito666> ;)
  <perrito666> dont we have tests that check that all flags are documented?
  <natefinch> probably not.... that's a little hard to do
  <perrito666> natefinch: well you know the flags you just need to call help and check that they at least appear there
  <natefinch> perrito666: well, in this case, zone is a placement directive, which means it's not actually a flag
  <perrito666> bu
  <natefinch> it's 'juju deploy mysql zone=us-east-1b'
  <natefinch> (not -zone=us-east-1b)
  <natefinch> perrito666: er, I guess it's only add-machine
  <natefinch> https://juju.ubuntu.com/docs/charms-ha.html
  <perrito666> natefinch: tested
  <mfoord> g'night all
  <natefinch> perrito666: looking
  <natefinch> perrito666: reviewed, couple minor things in the test
  <perrito666> natefinch: did you notice that you can actually select blocks of code and comment on that?
  <sebas5384> hi o/
  <sebas5384> i was wondering if I still need swift in openstack in order to bootstrap an openstack environment
  <natefinch> sebas5384: we're working on removing the storage requirement for juju, but I'm not sure it's in stable code yet
  <perrito666> natefinch: fixed
  <sebas5384> natefinch: yeah, think I sow that in a release notes somewhere
  <natefinch> sinzui: the storage requirement for juju was removed recently, but I presume that's in 1.21, which isn't a stable release, right?
  <sinzui> natefinch, that is right
  <natefinch> sebas5384: so if you want to do some testing with 1.21 alpha, you're welcome to.  We're pushing to get it to stable in the next few weeks
  <sebas5384> ahh great, I'm going to download it to test it :)
  <sebas5384> thanks!! :D
  <natefinch> perrito666: ship it!
  <perrito666> utils dont have a bot right?
   * perrito666 feels a bit silly
  <natefinch> I think that's correct
  <perrito666> man do I feel dumb :p
  <perrito666> reminds me of an european guy I found in a public restroom at a party on the weekend he was making all kind of gestures to the water tap until I passed and just turned the handle on and he stood looking at it and said "oh, one of those"
  <natefinch> haha, I do that
  <perrito666> sinzui: natefinch mgz aaand we have a pass http://juju-ci.vapour.ws:8080/job/functional-ha-backup-restore/963/
  <natefinch> perrito666: yay!
  <sinzui> yep I was just marking the bug fix released
  <perrito666> sinzui: tx
  <sinzui> there is something a miss in the unittest runs. both an instance and a real machine disappeared. I am rerunning the tests
  <perrito666> sinzui: I  have my finger ready to trigger a $$merge$$ :p
  <sinzui> perrito666, go ahead, you are unblocked
  <perrito666> davecheney: could you tell me a bit more about http://reviews.vapour.ws/r/275/#comment1666 ?
  * ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs:  None
  <davecheney> perrito666: it's better to write code that doesn't need that bit of boiler plate
  <davecheney> 'cos if that method gets refactored and that code lost
  <davecheney> all hell will break loose
  <natefinch> I agree...   foo := foo is cute for language nerds, but it's never a good idea in production code.
  <thumper> menn0: change pwd code updated
  <menn0> thumper: ok, i'll have a look shortly
  <thumper> cheers
  <perrito666> davecheney: I guessed so much, then I git blamed and suddenly I doubted myself
  <menn0> thumper: done
  <menn0> thumper: if you have a chance could you look at http://reviews.vapour.ws/r/284/ pls?
  <katco> is there a way to specify an ec2 availability zone when bootstrapping?
  <perrito666> katco: having ec2 complaining of no more machines?
  <katco> perrito666: i am =/
  <perrito666> katco: change region, goes faster
  <katco> perrito666: you sound like you've run into this before :)
  <katco> perrito666: tyvm, sir. i'm unblocked :)
  <perrito666> katco: backup and restore testing makes me bump into this often :p
  <katco> perrito666: hehe
  <wwitzel3> menn0: ping
  <menn0> wwitzel3: hi
  <wwitzel3> menn0: I am unable to replicate that bug, can you try destroying your lxc template and see if you run in to the same issue?
  <menn0> wwitzel3: will do
  <perrito666> davecheney: natefinch if I understand correctly that is done because the member from the for declaration is always the same var but the one inside is subject to the scope of that loop right?
  <wwitzel3> menn0: I tried with both trusty and precise and I'm not getting any of those errors, what i think is likely the cause is the lxc-template we generate for machine-0 probably has the old "anything" certs
  <menn0> wwitzel3: that's quite likely. I think I've had these templates for ages.
  <wwitzel3> menn0: I'm testing out ec2 right now just to be sure.
  <menn0> wwitzel3: if that is the problem (I'm bootstrapping again now) how do avoid it biting other people?
  <menn0> wwitzel3: should we have an upgrade step for alpha2 or 3 that deletes the templates?
  <wwitzel3> menn0: good question, lets see if removing the template indeed fixes it for you, and if so, I'll defer to thumper or jam on what they want to do.
  <menn0> wwitzel3: sounds good
  <wwitzel3> menn0: and then what ever they tell me to do, I'll do :)
  <wwitzel3> menn0: we could always go with the olde, "local provider isn't production"
  <menn0> wwitzel3: true
  <wwitzel3> menn0: but that isn't very friendl ;)
  <menn0> wwitzel3: no it's not and deleting the templates probably isn't that hard
  <menn0> wwitzel3: although I guess it's not an upgrade problem actually
  <menn0> wwitzel3: this happens even with a fresh bootstrap
  <menn0> wwitzel3: I wonder if there's a way that Juju could detect that the certs are bad
  <wwitzel3> menn0: maybe, I don't know enough about how that template is generated .. I think we won't that the machine-0 certs are bad until another machine attempts to connect.
  <menn0> waigani: since thumper doesn't appear to be around could you have a look at http://reviews.vapour.ws/r/284/ please?
  <waigani> menn0: half way through
  <menn0> waigani: ok cheers :)
  <wwitzel3> menn0: ec2 is good
  <menn0> wwitzel3: my bootstrap just finished here and I still see the problem
  <menn0> wwitzel3: :(
  <menn0> wwitzel3: the template was definitely re-created from scratch. I watched Juju do that by monitoring lxc-ls --fancy
  <davecheney> perrito666: https://golang.org/doc/faq#closures_and_goroutines
  <perrito666> davecheney: tx
  <davecheney> talks about a different problem, but the underlying cause is the same
  <davecheney> for _, x := range A_SLICE_OF_VALUES {
  <davecheney>         go something(x)
  <davecheney> }
  <davecheney> in this case it was
  <davecheney> for _, x := range A_SLICE_OF_VALUES {
  <davecheney> append(something, &x)
  <wwitzel3> menn0: hrmm, i've got nothing .. all clean http://paste.ubuntu.com/8738740/ (machine-0), http://paste.ubuntu.com/8738764/ (unit-mysql)
  <perrito666> davecheney: that is why it does x:=x to get a non shared x I assume
  <wwitzel3> menn0: all-machines is aggregating .. machine-1 and machine-2 are good .. not even one occurance of that error you are seeing :/
  <menn0> wwitzel3: well I'm seeing the same as what I reported on the ticket. All machine logs except machine-0 and and all unit logs are reporting the rsyslog worker continually restarting
  <menn0> wwitzel3: so you've tried it with the local provider on your machine?
  <perrito666> and yes, it is waiting for someone to not know this to delete that line thinking its useless
  <davecheney> perrito666: yup
  <davecheney> but that nastyness can be avoided by changing the type of the slice being iterated over
  <wwitzel3> menn0: yeah those logs are local provider
  <perrito666> well since I am refactoring, why not
   * thumper is back
  <thumper> physio again
  <menn0> wwitzel3: and you're using current master not 1.20?
   * davecheney pops champaign 
   * thumper is somewhat broken
  <perrito666> davecheney: or, I could delete that line and let the hell break loose and then go to sleep :p
  <davecheney> perrito666: whatever works for you
  <wwitzel3> menn0: yep, current master, 4e078f90b7d31b80feac167197679ad67a59650c
  <menn0> wwitzel3: that's what i'm using
  <menn0> wwitzel3: wtf!
  <menn0> wwitzel3: there must be some difference between our machines
  <wwitzel3> yeah :/
  <menn0> wwitzel3: go version? I'm using 1.2.1 linux/amd64
  <wwitzel3> menn0: go version go1.2.1 linux/amd64
  <wwitzel3> menn0: Linux wwitzel3-ThinkPad-W540 3.16.0-23-generic #31-Ubuntu SMP Tue Oct 21 17:56:17 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  <davecheney> perrito666: i thought this was matty's change anyway ?
  <menn0> wwitzel3: remind me how to get that?
  <perrito666> nope, I refactored the tests for peergrouper because it was causing me an import cycle
  <wwitzel3> menn0: uname -a
  <menn0> wwitzel3: Linux bazhou 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  <waigani> menn0: done http://reviews.vapour.ws/r/284/
  <menn0> wwitzel3: your kernel is a little ahead but I don't see how this would affect this problem
  <wwitzel3> menn0: yeah, me either
  <menn0> wwitzel3: are you on utopic or trusty
  <menn0> wwitzel3: i'm still on trusty?
  <menn0> s/\?//
  <wwitzel3> menn0: utopic, deploying trusty
  <wwitzel3> menn0: but my ec2 was all trusty
  <menn0> wwitzel3: but juju was built on utopic right?
  <menn0> wwitzel3: that might be the difference
  <menn0> wwitzel3: I think we need to try on some other people's machines to narrow this down
  <anastasiamac> katco: morning :-) saw ur email
  <wwitzel3> menn0: maybe .. we need to draw some straws
  <wwitzel3> exactly :)
  <katco> anastasiamac: good morning :)
  <anastasiamac> katco: makes sense to me but what do i know
  <menn0> thumper: what distro is your machine running?
  <thumper> menn0: ubuntu :-)
  <menn0> thumper: which series I mean
  <thumper> trusty
  <wwitzel3> thumper wins!
  <menn0> can you follow the repro instructions for bug 1387388 with current master please?
  <mup> Bug #1387388: rsyslog worker broken due to certificate signing problem <juju-core:New> <https://launchpad.net/bugs/1387388>
  <menn0> thumper: it happens consistently for me but never for wwitzel3 but I'm on trusty and he's building on utopic
  <menn0> thumper: we're trying to establish if that's the difference
  <thumper> kk
   * thumper grabs upstream
  <thumper> :-(
  <thumper> WARNING unknown config field "tools-url"
  <thumper> I get that six times
  <thumper> just destroying the local environment
  <wwitzel3> yeah, I got that too
  <menn0> that happens when you mix Juju versions
  <menn0> I think what's happening is that tools-url is no longer a known config option
  <menn0> everything still works but it is annoying
  <thumper> is there an upgrade step that fixes the config?
  <menn0> I don't think that will help
  <menn0> it's the client
  <thumper> no it isn't
  <thumper> um...
  <thumper> wat?
  <menn0> perhaps i'm wrong...
  <thumper> where is that warning coming from
  <thumper> ?
  <thumper> oh...
  <thumper> I bet it is the bootstrap-config stored in the JENV
  <menn0> that sounds right
  <wwitzel3> menn0: oh, the certs are stored in the local.jenv .. so if that was stale, that might also impact the rsyslog clients.
  <wwitzel3> menn0: but we remove that with destroy-environment right?
  <menn0> wwitzel3: the .jenv is certainly supposed to get removed on destroy
  <wwitzel3> menn0: yeah, I thought so
  <wwitzel3> menn0: brb,grabbing some food
  <thumper> menn0: what should I see?
  <menn0> thumper: check the logs of any of the machines except 0, or any of the units
  <thumper> seems to work here
  <menn0> thumper: do you see the rsyslog worker restarting
  <thumper> although I have different logging issues
  <thumper> but I think that is an older currently running local provider for another user
  <perrito666> oh I see why this change was not made before
  <anastasiamac> thumper: wallyworld assured me that noone in the world has "tools-url" anymore
  <anastasiamac> thumper: and if they do, they will change it :-)
  <wallyworld_> tools-url was deprecated post 1.16
  <anastasiamac> wallyworld_: yes and removed now-ish ;-)
  <menn0> waigani: thanks for the review. looks like you weren't looking at the latest version as the first thing you picked up was fixed in rev 2 (pushed last night)
  <thumper> wallyworld_: so where did it come from?
  <rick_h_> thumper: just looking at the title, admin can see other user's passwords?
  <thumper> rick_h_: not see, but change
  <rick_h_> thumper: ah ok /me goes back to reviews
   * wallyworld_ reads backscroll
  <thumper> rick_h_: juju only stores a salted hash of the password
  <thumper> rick_h_: so if you forget or lose it, we have to have a way to generate another
  <rick_h_> thumper: ok cool, sorry. Later in my day so misread that as admin could see it
  <perrito666> davecheney: sorry mate, making the change to a pointer slice goes all the end to replica set public API and replicaset.Config document it is not worth it, Ill make it cleaner so it is obvious to someone that plays go by the ear
  <thumper> rick_h_: np
  <wallyworld_> thumper: i'm guessing you have an environments.yaml that is old, one which had a tools-url config attribute specified perhaps
   * thumper just hit two different intermittent test failures
  <thumper> wallyworld_: there is no tools-url in my environments.yaml
  <thumper> I looked
  <thumper> and the environment I created was a few days ago
  <thumper> wallyworld_: it is possible that the tools-url was populated but empty
  <thumper> so it existed as a value, but empty
  <wallyworld_> in that case i'm not sure, will look into it
  <wallyworld_> it should have been totally deleted
  <wallyworld_> from the code
  <wallyworld_> so should not be present, empty or not
  <anastasiamac> wallyworld_,thumper: tools-url is not in current codebase
   * anastasiamac thinks
  <wallyworld_> thumper: maybe you have an older jenv file which is being used?
  <thumper> wallyworld_: nope
  <thumper> anastasiamac: yes, but when I created the environment a few days ago, was it there?
  <wallyworld_> it could have been
  <anastasiamac> thumper: can u do a search in ur codebase?...
  <thumper> anastasiamac: I have master :)
  <thumper> I expect it to be the same as your master
  <thumper> dvcs ftw
  <anastasiamac> thumper: my master is the same as urs
  <thumper> not there
  <menn0> wwitzel3, thumper: I'll do some more experimentation to see if I can figure out the rsyslog issue
  <thumper> wallyworld_: tools-url was in the 1.20 branch
  <wallyworld_> yes, it was processed as a deprecated attribute
  <menn0> thumper, wallyworld_, anastasiamac: I certainly see this warning when I've bootstrapped with 1.20 and then upgrade to 1.21
  <menn0> when I later destroy the environment with the 1.21
  <wallyworld_> what we should do in 1.21 is add an upgrade step to delete it
  <menn0> client
  <anastasiamac> menn0: could u paste the warning u get?
  <wallyworld_> i think because it is defined in the config schema, an empty value is inserted
  <wallyworld_> we already have an upgrade step to delete old values
  <thumper> wallyworld_: 1.20 branch, environs/config/config.go:852
  <waigani> menn0: just the first comment if from rev 1. The rest are rev 2.
  <wallyworld_> we just need to add tools-url to that existing step
  <thumper> wallyworld_: it always added a blank "tools-url" to the config
  <menn0> waigani: yep. got it
  <wallyworld_> right, that explains it
  <wallyworld_> we'll add the upgrade step to delete it
  <thumper> wallyworld_: also, bootstrap-config in the jenv on the user's machine will still have it
  <thumper> wallyworld_: which means they will still get the warning
  <thumper> every time they use the environment
  <thumper> unless you update the bootstrap config in the jenv file
  <wallyworld_> maybe we should only emit that warning if the value is not ""
   * thumper nods
  <thumper> that would reduce the likelihood of false errors
  <wallyworld_> cause changing jenv = yuk
  <thumper> ack
   * wallyworld_ looks at anastasiamac :-)
  <thumper> alexisb: ping
  <thumper> ha, it is only 4:23pm for alexisb
  <thumper> so close
  <thumper> alexisb: last part of first iteration of user management has landed for 1.21
  <thumper> \o/
  <thumper> it is DONE DONE
  <wallyworld_> about time :-P
  <thumper> yeah
  <thumper> true that
   * anastasiamac looks back at wallyworld
  <thumper> now... on to this messy things
  <anastasiamac> wallyworld_: m onto it ;-)
  <rick_h_> thumper: and this means the remote auth stuff is there to hook into?
  <thumper> anastasiamac: wallyworld_ is looking at you with the not so subtle suggestion that you do all the work
  <thumper> rick_h_: no
  <rick_h_> thumper: but but but :P
  <thumper> rick_h_: this means that juju has users and environments have users
  <wallyworld_> anastasiamac: awesome - you need to add to the existing upgrade step, and also not emit a warning if an unknown alue is empty
  <thumper> rick_h_: and they can log in
  <thumper> rick_h_: cmars is working on the remote identity stuff
  <thumper> rick_h_: and making good progress
  <wallyworld_> thumper: it's called delegation :-)
  <rick_h_> thumper: so how long before until we need the GUI to be speaking this stuff?
  <thumper> rick_h_: so this remote identity stuff is not blocked on us at all
  <cmars> rick_h_, we can stub out the GUI until you're ready :)
  <thumper> rick_h_: not sure what you mean there
  <rick_h_> thumper: cmars I mean when will someone bootstrap with these users and want to type that username/password into a GUI?
  <thumper> rick_h_: the GUI doesn't need to do any of this
  <thumper> rick_h_: so, this isn't the boostrap user being different
  <cmars> rick_h_, ah, that's not remote identity
  <rick_h_> thumper: so if I juju deploy juju-gui into an environment as user XXXX, I don't have the secret, but need too login with user/password?
  <thumper> rick_h_: that will come in 1.22
  <rick_h_> thumper: ah ok
  <rick_h_> thumper: yea, that's whaat I'm tring to track when we need to start updating our login box
  <rick_h_> thumper: ok, but we do need to update the username box
  <ericsnow> menn0: when you have a sec, take a look at the latest revision revision of those two review requests (243, 268)
  <thumper> rick_h_: you can use "juju api-info user password"
  <rick_h_> because logging in with the secret as 'admin' might not be true
  <thumper> rick_h_: and that will give you the username and password to log into the gui
  <rick_h_> thumper: ok, is this documented so we can poke at it?
  <thumper> rick_h_: I added it to the 1.21 release notes
  <thumper> rick_h_: poke away
  <thumper> rick_h_: it is in 1.21-alpha2
  <menn0> ericsnow: I will try. juggling a bunch of stuff right now
  <rick_h_> thumper: ok, so this will be in 1.21 mid-month it sounds like
  <ericsnow> menn0: no worries :)
  <thumper> rick_h_: ack
  <rick_h_> thumper: ok
  <thumper> rick_h_: but for now, the initial user is still "admin"
  <thumper> rick_h_: and the password is still stored in the same place
  <rick_h_> thumper: right, but they can create more
  <thumper> rick_h_: ack
  <rick_h_> thumper: and those new ones the user will not be able to login to the GUI without updating it
  <thumper> rick_h_: when you add a user, it gives you a jenv file
  <rick_h_> thumper: so sounds like I need to get some work items down
  <thumper> rick_h_: ah... wat?
  <thumper> rick_h_: why would they not be able to log in?
  <rick_h_> thumper: I bootstrap env as admin.
  <rick_h_> thumper: I create rick user with password
  <thumper> rick_h_: call me
  <rick_h_> thumper: k, sec
  <ericsnow> menn0: can I give you a hand?
  <alexisb> thumper, AWESOME!!!!!!
   * thumper runs to let the dog out
  <rick_h_> thumper: https://plus.google.com/hangouts/_/canonical.com/daily-standup?authuser=1
   * fwereade also cheers at thumper
  <menn0> ericsnow: thanks but I don't think so. I need to get a branch landed and narrow down a rsyslog worker problem that only seems to happen on my machine
  <ericsnow> menn0: k, let me know if that changes (I should be on a little longer at least)
  <thumper> wallyworld_: I need to put our 1:1 off for 30 minutes
  <thumper> wallyworld_: is that ok with you
  <thumper> ?
  <thumper> just found something I was supposed to do an hour ago :-(
   * thumper waits for 15s for a response
  <wallyworld_> thumper: sure, np
  <menn0> waigani: i've responded to your review
  <waigani> menn0: done
  <menn0> waigani: thank you
  <waigani> alexisb: ready when you are
  #juju-dev 2014-10-30
  <thumper> wallyworld_: thanks
  <wallyworld_> sure :-)
  <perrito666> ericsnow: not to be a party pooper but login using github is no longer working
  <perrito666> ericsnow: I get a 404 from github
  <ericsnow> perrito666: (â¯Â°â¡Â°)â¯ï¸µ â»ââ»
  <perrito666> lol
  <ericsnow> perrito666: I'm so gladd thumper showed me that URL :)
  <ericsnow> perrito666: you get a 404 in your browser or in rbt?
  <perrito666> browser I changed browser and when I try to re-login I go to oauth, log into github and then get redirected to a 404 page
  <ericsnow> perrito666: try it now
  <perrito666> still now the oauth button goes directly to the 404 since I am already logged in
  <ericsnow> perrito666: a github 404
  <ericsnow> perrito666: ?
  <perrito666> yup, that is what I meant by: <perrito666> ericsnow: I get a 404 from github
   * fwereade has updated http://reviews.vapour.ws/r/288/ and is busy pushing right now
   * fwereade would *really* like to be able to merge it tomorrow morning -- and, fwiw, jam has already taken a look at the logic, but it necessitated a depressing quantity of test changes that could use a look
   * fwereade goes to bed
  <mwhudson> thumper: is one of the ideas between multi-environment to support things like having most things in MAAS but being able to burst to EC2?
  <thumper> no
  <thumper> not multi-provider
  <thumper> still one provider
  <thumper> but separate environments
  <mwhudson> ah
  <wallyworld_> axw: standup?
  <axw> wallyworld_: sorry, brt
  <waigani> menn0: http://reviews.vapour.ws/r/295/
  <waigani> menn0: I have to take Bella to ice skating, back online in 40min or so
  <waigani> menn0: see what you think of the solution for interface ids
  <thumper> menn0: https://github.com/juju/loggo/pull/5
  <menn0> thumper: just finishing up another review. You're next :)
  <thumper> menn0: ack
  <menn0> thumper: I've had a look and have one question.
  <menn0> thumper: (see PR)
  <thumper> kk
  <menn0> thumper: done
  <menn0> wallyworld_, axw, thumper: I have a local environment that's failing to upgrade from 1.20 to current master due to:
  <menn0> upgrade step "migrate tools into environment storage" failed: failed to fetch 1.20.11.1-precise-amd64 tools: stat /home/menno/.juju/local/storage/tools/released/juju-1.20.11.1-precise-amd64.tgz: no such file or directory
  <wallyworld_> hmmm
  <menn0> looking in my .juju it looks like the directory should be "releases" not "released"
  <menn0> does that ring any bells?
  <wallyworld_> yeah, it needs to retry
  <wallyworld_> i'll fix
  <menn0> the upgrade logic is retrying all the steps anyway
  <menn0> but it's not getting there yet
  <menn0> wallyworld_: is "released" correct?
  <wallyworld_> menn0: it is for 1.21
  <menn0> wallyworld_: right
  <wallyworld_> i did test that upgrade a couple of days ago
  <wallyworld_> something must have changed
  <wallyworld_> sigh
  <wallyworld_> so i need to fix
  <menn0> wallyworld_: I think I've seen this once before but then it stopped happening
  <wallyworld_> could be, i'll take a look. thanks for letting me know
  <menn0> wallyworld_: np
  <menn0> wallyworld_: in case it's relevant, fwereade would like us to split out the upgrade steps that act state.State directly from the ones that use api.State, running all the state.State steps first (on state servers only obviously)
  <wallyworld_> sounds reasonable
  <menn0> wallyworld_: it's actually more important then I actually first realised, especially now that we have upgrade steps defined for various alpha releases. You really need all the DB migrations for all intermediate releases to have run before other upgrade steps.
  <axw> wallyworld_: the upgrade step will need to use LegacyRelases or whatever you named it
  <wallyworld_> yes, changing tha now
  <axw> cool
  <jam> axw: are you still around?
  <jam> I'm running into problems with "AddCharmConcurrently" sometimes failing (about 1 in 3 for me, and it failed on the bot)
  <axw> jam: I am
  <axw> oh
  <jam> and you were the last person to touch the "c.Assert(blobs, gc.HasLen, 10)" line
  <axw> yeah, I was trying to fix it :)
  <axw> jam: anything useful in the error output?
  <jam> http://juju-ci.vapour.ws:8080/job/github-merge-juju/1073/console
  <jam> it lists the blobs it did find
  <axw> thanks
  <jam> which in my case is 9 total blobs, and only 1 of them has "true"
  <jam> axw: I *do* see 10 calls to Client.AddCharm
  <jam> oddly, I then see 10 "successfully dialed mongo on 127." but only 8 "successfully dialed mongo on ::1"
  <jam> that seems a bit WTF to me
  <jam> why would we be dialing mongo separately, because of the storage abstraction?
  <jam> resource catalog entry created with id "47d6e7812099383e8266f590d37b7f5369ef43931aecc72a4b86674e7052f346a391648209b8e69ea5b649d2c0e86da0"
  <axw> jam: possibly because we're Copying the mgo.Session
  <jam> concerns me, because it is exactly the same identity
  <axw> jam: IIRC that particular id is based on a hash of the content
  <axw> and so it's expected
  <jam> I do only see 9 "managed resource entry created" calls
  <jam> log entries
  <axw> yeah, odd.
  <jam> and 9 "resource catalog entry created" lines
  <jam> axw: ok, weird... I see all of the requests result in a success response
  <jam> (you can match up the RequestId 3-12 and see the Response for each one of them)
  <jam> axw: I have a theory... that one of those "concurrent" uploads actually doesn't start until the first success finishes
  <jam> and thus it doesn't have anything to upload because the content is already in the db
  <axw> sounds very plausible...
  <axw> yep
  <axw> that'd do it
  <axw> we should just check <= 10
  <jam> axw: but why would it ever be >10
  <axw> jam: heh, true
  <jam> axw: I think we need to check with dimitern on what his thoughts are
  <jam> I understand that we want to make sure if there is *some* concurrency, only one entry wins
  <jam> axw: but I think line 1019 of apiserver/client/client.go is responsible
  <jam> "if stateCharm.IsUploaded(), return nil)
  <axw> jam: yep
  <jam> axw: so how was it failing before that you were trying to fix it ?
  <axw> jam: I misremembered. I just changed it to work with the blobstore-backed charm storage; some bugs were found in the blobstore as a result of running that test, but it didn't end in me changing the test for that purpose
  <jam> axw: yeah, I'm looking at your diff, trying to see how the test was intended to work
  <axw> jam: IMO we should just remove the count check. I guess it'd be nice to force the concurrency, but seems like a lot more work than it's worth
  <axw> would be nice for it to not be random though
  <jam> axw: so I'm trying to figure out if the old test was just as wrong, or if it is just that we introduced deduping and now only sometimes it fails
  <axw> jam: I'm pretty sure it was broken before. the only change, looking at the diff is how the charm archive is stored - in the blobstore as opposed to in provider storage
  <axw> and the existence check was there before
  <jam> axw: it doesn't help that "git blame apiserver/client/client.go" lists the last modification for "IsUploaded()" lists a revision that "git log" say doesn't include the file
  <axw> huh
  <jam> axw: I'm guessing that "git blame" is tracking hunks being moved between files while "git log" isn't
  <jam> axw: how about a reasonable compromise of asserting that we do have at least 2 entries (so at least *some* concurrency was observed)
  <jam> fair enough?
  <axw> jam: it's still possible to fail though... I'm not super keen on that
  <axw> what if we synchronise 2 of the calls in the storage Put?
  <axw> then we at least exercise the concurrency of the state changes
  <jam> axw: so it would have to not be concurrent at all on 10 requests, and ATM we've seen at least 9
  <jam> axw: so you mean have Put wait to return until it gets a second request ?
  <axw> yeah I guess it's pretty unlikely to fail
  <jam> I do like that more
  <jam> (I'm all for "deterministic" concurrency)
  <jam> I'm not 100% sure how to do that, though
  <jam> I guess recordingStore could do it for us?
  <axw> jam: yep, if we have a waitgroup or something in there we could use that
  <axw> we could just have them all wait actually - the bits after storage.Put don't interact with blob storage
  <axw> oh actually I lie
  <axw> they can Remove
  <jam> axw: I think they have to Remove because they end up as "false"
  <axw> right
  <axw> and if we sync them all then we lose potential testing of concurrent Put/Remove
  <jam> axw: well we can do it as a barrier
  <jam> so the code waits at the beginning of Put until it accrues 10 requests and then all of them unblock
  <axw> yes that'd be better. I was thinking in terms of having it at the end
  <jam> axw: so... hmm. could it be done with a sync.WaitGroup ?
  <jam> I don't know if it is evil or not, but rs.Put() could do a "wg.Done(); wg.Wait()" sort of thing?
  <jam> so we have 2 WGs, one for "the goroutine is completely done" and another for "each goroutine has gotten to the start of Put()"
  <axw> jam: that seems fine to me
  <jam> axw: so if I put a time.Sleep(10ms) into the for loop, it fails reliably
  <jam> and with putBarrier and the sleep it passes reliably
  <jam> axw: http://reviews.vapour.ws/r/297/
  <axw> jam: thanks, just finishing a review will take a look in a sec
  <axw> actually this may take a bit, I'll switch now
  <axw> jam: LGTM, thanks
  <wallyworld_> axw: this makes upgrades look in the old releases dir http://reviews.vapour.ws/r/296/
  <axw> wallyworld_: on it
  <wallyworld_> ta
  <jam> dimitern: morning
  <mattyw> morning folks
   * fwereade has $$merged$$ 987, and is going back to bed for a bit, not feeling so good
  <fwereade> bollocks, forgot to push the last tweaks, at least they're trivials
  <fwereade> I'll do them later
  <TheMue> morning
  <dimitern> morning all
  <perrito666> morning
  <axw> wallyworld_: team meeting?
  <jam> voidspace: TheMue: team meeting
  <TheMue> jam: I'm in there.
  <jam> TheMue: not standup, juju-core-team
  <jam> TheMue: https://plus.google.com/hangouts/_/canonical.com/juju-core-team
  <TheMue> jam: oh, ouch, coming.
  <jam> voidspace: you as well, if you didn't realize
  <jam> fwereade: are you coming ?
  <perrito666> jam: he mentioned going to bed
  <voidspace> oops
  <voidspace> sorry
  <voidspace> I forgot both...
  <voidspace> we have three m1.small instances running on the ec2 account
  <voidspace> I think they've been there for a few days
  <voidspace> ah no, launched October 30th
  <mattyw> mgz, looks like we just landed some code that doesn't build
  <mattyw> http://juju-ci.vapour.ws:8080/job/github-merge-juju/1081/console
  <perrito666> mattyw: looks like fwereade change https://github.com/juju/juju/pull/987
  <mattyw> submitted a fix for it: http://reviews.vapour.ws/r/299/
  <mattyw> axw, you're ocr today right? could you take a look at this ^^?
  <perrito666> mattyw: do the test pass with those alone?
  <mattyw> perrito666, they do
  <axw> mattyw: sorry, EOD as soon as this meeting is up. been online for about 10h and back for another meeting later :/
  <mattyw> axw, ok no problem, it is quite trivial - fixing build failures that have landed
  <axw> mattyw: looks like perrito666 already did it anyway
  <perrito666> mattyw: I just lgtmd but you might need davecheney love
  <perrito666> although its a really trivial change
  <mattyw> axw, perrito666 ok cheers guys
  <perrito666> mattyw: but as will said: <fwereade> bollocks, forgot to push the last tweaks, at least they're trivials
  <perrito666> which I presume conains your change
  <wallyworld_> axw: yeah, was at soccer tonight, see you again in 90 mins
  <mattyw> could I get another review from a "graduated" reviewer so we can get this landed?  http://reviews.vapour.ws/r/299/
  <TheMue> mattyw: done
  <mattyw> TheMue, thanks very much
  <TheMue> yw
  <jam> mgz: mattyw: do we know how we were able to land the code on the bot if it doesn't even compile ?
  <mattyw> jam, no idea, I pinged mgz to see if the logs can tell us any more
  <mattyw> jam, this is the offending merge: http://juju-ci.vapour.ws:8080/job/github-merge-juju/1080/console but it shows the context test as passing. Then it fails in the next merge http://juju-ci.vapour.ws:8080/job/github-merge-juju/1081/console
  <jam> mattyw: what is really strange is that it is failing inside "cmd/juju" how would that even have visibility into worker/uniter/context/factory_test.go
  <mattyw> jam, there's something odd happening there, because the line that contains uniter/context correctly states the build failed - just the build errors are nowhere near it
  <rogpeppe> review appreciated on this change that makes it possible to use Go 1.4 with our testing stuff: https://github.com/juju/testing/pull/36
  <mattyw> jam, that must just be a bug with the way output is captured
  <jam> mattyw: it may be a stderr vs stdout buffering issue
  <jam> (stdout is oftend buffered and stderr is immediate)
  <mattyw> jam, sounds most likely
  <jam> though you have 300s from the time of that error messages and when it gets to uniter build failed, but I guess it isn't that many lines
  <jam> mattyw: I think we need someone investigating this, but I'm at a loss for any more information myself
  <jam> this feels a *lot* like we can't trust our bot to reject bad patches, which is a bad place to be
   * jam away
  <mattyw> jam, I don't really see how that could land at all. Maybe running tests with -a to force rebuilding all the packages might help?
  <mattyw> jam, that's all I can think of
  <jam> mattyw: yeah, I just did a checkout of 2224d55 and it clearly doesn't "go test" here.
  <mattyw> jam, it clearly shows it passing in the log of the merge
  <mattyw> jam, even though we know it doesn't
  <jam> mattyw: ok  	github.com/juju/juju/worker/uniter/context	17.402s yeah
  <jam> mattyw: this feels a *lot* like we were running the tests on raw "master" rather than the merge of "master" and the new code, but we've had rejections for bad patches before
  <mattyw> jam, we've had this issue before, where broken code lands
  <mattyw> jam, and we've tried landing intentionally broken stuff before to check it gets rejected
  <jam> mattyw: so I feel like we are well into territory that we need mgz/someone who set up this bot/ to be doing the investigation
  <jam> but we clearly were able to land broken code on trunk, and than is (IMO) a Critical priority bug because it causes us to land Critical breakage accidentally
  <perrito666> mattyw: jam I have seen the delay between error and the test stdout when there is a compilation errror, same happened to me when I encountered an import cycle
  <jam> perrito666: sure, I'm not particularly worried about that. I *am* worried that William's patch caused a build failure but it didn't fail the merge, but it did fail the *next* merge.
  <jam> and I can checkout the revision that the bot landed and "go test" fails in that directory for me
  <perrito666> do you have the link for the merge job that passed?
  <axw> wallyworld_: sorry
  <axw> had clicked the button and it was lagging :p
  <wallyworld_> np :-)
  <axw> to answer, yes, I think so
  <wallyworld_> we'll catch up tomorrow
  <wallyworld_> it was a good meeting
  <axw> cya tomorrow
  <axw> yep
  <perrito666> anastasiamac_: hey is http://reviews.vapour.ws/r/300/diff/# released? I see that the launchpad bug is marked as fix released
  <anastasiamac_> perrito666: there was a minor inconvenience with the fix...
  <perrito666> oh ok, so that still needs a review?
  <anastasiamac_> perrito666: actually 2: 1. warning was displayed when not needed 2. upgrade did not clean deleted attribute
  <TheMue> phew, back again after some troubles with my network
  <anastasiamac_> perrito666: would be gr8 :-)
  <anastasiamac_> perrito666: thnx for the review :-)
  <perrito666> np
  <anastasiamac_> perrito666: hope no oxygen tanks at high speed for u today :-)
  <perrito666> anastasiamac_: I would certainly hope a) they deliver once a week only and b) the already know there is a speed bump there for next time
  <anastasiamac_> perrito666: u have oxygen delivered once a week?
  <perrito666> anastasiamac_: I believe they where delivering to a construction nearby, its oxygen used for soldering not for hospitals
  <anastasiamac_> perrito666: ahh! makes sense ...
  <wallyworld_> ericsnow: since you're ocr, could you take a look at http://reviews.vapour.ws/r/302/ ? it's a critical bug fix for alpha3
  <wallyworld_> :-)
  <ericsnow> wallyworld_: will do
  <wallyworld_> thanks
  <wwitzel3> fwereade: ping
  <wwitzel3> natefinch: are you able to get on to the tosca call?
  <wwitzel3> natefinch: I keep getting disconnected when I dail in
  <natefinch> wwitzel3: oops, lost track of time, lemme try
  <natefinch> wwitzel3: works for me
  <wwitzel3> natefinch: just worked for me too *shrug*
  <perrito666> something tells me that our standup is pushed?
  <ericsnow> perrito666: presumably it got tosca'ed an hour later
  <perrito666> lol, well if we could toscate :p the calendar appt that would be awesome
  <natefinch> yeah sorry guys
  <perrito666> dont worry everything is awesome
  <perrito666> :p
  <perrito666> ill get you to see that movie
  <natefinch> I just bought it, actually :)
  <natefinch> I had opportunity to watch it on the way to and from brussels, but I didn't want to watch it without my wife
  <perrito666> natefinch: lol, I watched all the movies in the plane it wwas a long flight
  <perrito666> natefinch: you could include your kids too, you know, as an excuse to watch a kids movie
  <ericsnow> natefinch-afk, perrito666, wwitzel3: standup?
  <perrito666> ericsnow: my calendar says like in 2 hs
  <ericsnow> perrito666: my bad
  <katco> fwereade: ping
  <rogpeppe> mgz: ping
  <mgz> rogpeppe: hey
  <rogpeppe> mgz: hiysa
  <rogpeppe> mgz: we're just debating bzr stuff
  <rogpeppe> mgz: if i've done "bzr checkout --lightweight something -r N
  <rogpeppe> "
  <mgz> yup
  <rogpeppe> mgz: is there some way of finding out the revision digest that i've just checked out?
  <mgz> bzr revno --tree
  <rogpeppe> mgz: unfortunately that prints:
  <rogpeppe> bzr: ERROR: Server sent an unexpected error: ('error', 'GhostRevisionsHaveNoRevno', 'Could not determine revno for {roger.peppe@canonical.com-20130430112516-r2iav07y5kqd9cmk} because its ancestry shows a ghost at {roger.peppe@canonical.com-20130913075328-6ok3ntchesxkt04o}')
  <mgz> you've done a checkout of a remote branch? not a local one?
  <rogpeppe> mgz: yes
  <mgz> Don't Do That
  <rogpeppe> mgz: because we're only interested in the tree (and now, the revision-info of that tree)
  <rogpeppe> mgz: we don't care about the history at all
  <mgz> (it's a bug, but you're asking bzr to do bascially everything over the api)
  <rogpeppe> mgz: yeah, ok
  <mgz> you can fixup the remote branch to not tickle the bug probably, but I'd have to go lookup how
  <rogpeppe> mgz: it needs to work with any branch
  <rogpeppe> mgz: as this is part of an automated process, ingesting charms/bundles to the charm store
  <mgz> rogpeppe: bug 1161018
  <mup> Bug #1161018: bzr revno fails on ghost ancestry <Bazaar:Confirmed> <https://launchpad.net/bugs/1161018>
  <mgz> has a workaround (which is slow - needs to fetch some things over http, but fine for just revno I think as that shouldn't touch packs)
  <mgz> (just less fine for update if that also hits the issue)
  <rogpeppe> mgz: i don't see a workaround there
  <mgz> use nosmart+ for the remote branch url
  <mgz> as in, force bzr not to use the api, but just touch the remote objects directly
  <rogpeppe> mgz: what would a url using nosmart+ look like?
  <alexisb> natefinch-afk, perrito666 : can someone on moonstone take a look at the request in comment #9 and get it resolved asap:
  <alexisb> https://bugs.launchpad.net/juju-core/+bug/1384001
  <mup> Bug #1384001: Juju doesn't retry hard enough when destroying MAAS environments <cloud-installer> <destroy-environment> <maas-provider> <Go MAAS API Library:Fix Committed by rvb> <juju-core:Triaged> <MAAS:Fix Committed by rvb> <https://launchpad.net/bugs/1384001>
  <mgz> rogpeppe: literally just any other url with nosmart+ on the front
  <mgz> so, nosmart+lp:... nosmart+http://...
  <perrito666> alexisb: looking
  <rogpeppe> mgz: interesting. that's about 20 times faster too.
  <rogpeppe> mgz: (and it works)
  <alexisb> mgz, the version change issue you are working, is it related to a specific bug?
  <alexisb> mgz, I just want to make sure core is not also trying to work the same bug :)
  <mgz> alexisb: it looks like it's just an accidential change
  <rogpeppe> mgz: sorry, that's bollocks
  <rogpeppe> mgz: but it does work
  <alexisb> mgz, is there a bug number?
  <mgz> commit 6ae54f8... changes version back from alpha3 to alpha2
  <mgz> alexisb: filing now
  <perrito666> alexisb: natefinch-afk running now tests before PR the change
  <alexisb> perrito666, mgz thanks guys!
  <mgz> bug 1387764 filed, putting up a branhc now
  <mup> Bug #1387764: Version reverted from alpha3 to alpha2 <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1387764>
  <perrito666> ok anyone ? https://github.com/juju/juju/pull/1004
  <alexisb> ericsnow, Please take a look ^^
  <mgz> perrito666: looks fine to me, what's the reviewboard bits up to atm?
  <mgz> https://github.com/juju/juju/pull/1005 for the version fix
  <ericsnow> alexisb: will do
  <alexisb> mgz, may have already done it :)
  <perrito666> mgz: reviewboard?
  <alexisb> thanks all for pulling together to get these critical bugs nailed
  <perrito666> mgz: http://reviews.vapour.ws/r/306/
  <mgz> perrito666: do I need to manually create still? I thought it was magically handled now
  <mgz> (lgtmed btw)
  <perrito666> mgz: should be magically created
  <perrito666> though you have to add the comment by hand
  <perrito666> meh 1387764 is blocking me
  <mgz> perrito666: mwahaha, easy way to resolve that :)
  <perrito666> mgz: go to rb
  <perrito666> look for your pr
  <perrito666> make sure its not draft
  <perrito666> if so publish it
  <perrito666> because its not there
  <mgz> I do not see anything...
  <perrito666> mgz: rbt post it
  <perrito666> plz
  <perrito666> or even better
  <perrito666> hold
  <mgz> I can rbt post
  <perrito666> mgz: there, you have the lgtm in github... but now I just realized that you need a graduated
  <perrito666> mgz: like this http://www.youtube.com/watch?v=hsdvhJTqLak :p bad joke, I know
  <perrito666> rogpeppe: could you lgtm mgz branch? you have super powers
  <rogpeppe> perrito666: which branch?
  <perrito666> https://github.com/juju/juju/pull/1005/files
  <rogpeppe> mgz, perrito666: i don't understand the issue but i trust martin :)
  <rogpeppe> reviewed
  <perrito666> rogpeppe: someone backed the version number
  <perrito666> by accident I assume
  <perrito666> mgz: merge your thing
  <mgz> doing
  <mgz> perrito666: wtf, that trailer is basically the whole film
  <perrito666> mgz: 70s
  <perrito666> anyway, how do I know when CI is unlocked?
  <mgz> I shall do it as soon as pos
  <mgz> perrito666: at least a while for it to go through merge though
  <perrito666> I never get if that is automated or someone needs to do something by hand
  <mgz> the marking is by-hand
  <perrito666> is it the lp search query?
  <mgz> I'll do that as soon as the build succeeds, rather than waiting for all test jobs though
  <mgz> yeah, but peoples are the oneses who mark bugs fix released
  <jw4> review request for the on call reviewer today : http://reviews.vapour.ws/r/307/
  <jw4> I did a pre-review just commenting on things that might make the intent clearer.
  <perrito666> mgz: are we there? are we there?
  <mgz> not yet, needs to go through build revision and publish
   * perrito666 runs in circles
   * perrito666 feels assigned
  <perrito666> natefinch-afk: ericsnow wwitzel3 ?
  <perrito666> standup?
  <ericsnow> wwitzel3: standup?
  <natefinch> perrito666, wwitzel3, ericsnow: sorry guys... the halloween party at my kid's preschool ran over
  <ericsnow> natefinch: no worries, perrito666 and I talked about restore
  <perrito666> natefinch: ah you have that :p
  <perrito666> I was not aware it was to the level of having school parties
  <perrito666> I thought it was only the disguises + getting candy
  <natefinch> perrito666: this was basically a shortish thing at the end of the day where all the kids dress up and parade around the parkinglot :)
  <mgz> perrito666: publish-revision running now
  <mgz> (was slow as eric had sent a change through just before mine, and I didn't really want to interrupt)
   * perrito666 eyes eric
  <mgz> perrito666: so, eta 20 mins
  <perrito666> heh, how nice it would be to do $$mergeafter##$$
  <natefinch> I love the way everyone rags on simple streams for putting the "simple" in the name.
  <mgz> natefinch: I view it as like, refering to someone as simple
  <mgz> a semi-polite way of saying it's a bit dumb
  <natefinch> lol
  <natefinch> nice
  <perrito666> ericsnow: isnt there a metadata FromJson?
  <ericsnow> perrito666: nope
  <perrito666> booo
  <perrito666> :p
  <ericsnow> perrito666: I figured if we needed it for restore we could add it then :)
  <perrito666> ericsnow: I dont want to lie to you, at this point I am ranting at my dog for your decission
  <ericsnow> perrito666: â¬ââ¬ï»¿ ã( ã-ãã)
  <perrito666> mgz: ?
  <mgz> looks like it's just finishing
  <mgz> publish-revision/1104 is what I'm watching
  <ericsnow> natefinch: didn't this get superceded: http://reviews.vapour.ws/r/237/
  <ericsnow> natefinch: I.E. you can discard it :)
  <perrito666> mgz: finished \o/
  <mgz> itsblue!
  <mgz> perrito666: go ahead and land
  <natefinch> ericsnow: yeah, thanks for reminding me
  <ericsnow> natefinch: it's my job, sir
   * ericsnow shines badge
  <perrito666> ericsnow: its rbt post -v ### right?
  <perrito666> to update
  <ericsnow> perrito666: for an update it -r ### (or just use -u)
  <perrito666> ericsnow: rbt post -u?
  <ericsnow> perrito666: yeah, it will figure out the right one and ask you
  <perrito666> fwereade: hey, did you ever got a chance to look at charm-sync spec?
  <mfoord> g'night all
  <menn0> davecheney, mwhudson: email standup today?
  <perrito666> afghahsgfhagsfh I hate local restriction to have juju-local package
  <jw4> ericsnow: thanks for looking at that review
  <ericsnow> jw4: yeah, still working on it :)
  <waigani> mwhudson,davecheney,menn0: did I miss standup or are we skipping it today?
  <jw4> ericsnow: ta x2
  <menn0> waigani: we haven't had it. I suggested email standup but davecheney and mwhudson aren't around yet.
  <waigani> menn0: davecheney just popped up
  <davecheney> menn0: sure
  <waigani> scratch that, davecheney just dropped out
  <waigani> okay email
  <wwitzel3> menn0: you dig up more on that local provider issue you were having?
  <menn0> wwitzel3: no I haven't but that's one of the things I want to do this morning
  <menn0> wwitzel3: had a number of other things to take care of yesterday
  <menn0> wwitzel3: and since it appears to just be my machine it seemed less important
  <natefinch> perrito666: you might as well start writing doc strings now, no need for my review to go through first ;)
  <mwhudson> menn0: i'm on leave today
  <menn0> mwhudson: oh yeah, sorry. have a good day off :)
  <waigani> menn0: just manually testing upgrade for my branch, then saw in your standup notes about Ian's branch.
  <menn0> waigani: yeah, there's a tools related upgrade step that keeps failing
  <perrito666> natefinch: ok ok
  <waigani> menn0: okay, so we'll wait for the fix
  <menn0> waigani: i'm checking now to see if it's been fixed
  <perrito666> natefinch: you exagerate, its not that bad
  <natefinch> perrito666: it's not that good if I stop writing comments that are just "docs"
  <natefinch> (because there's too many)
  <menn0> wwitzel3: I think I see something that could be related to this rsyslog problem
  <menn0> wwitzel3: can you do this: cd $GOPATH/src/code.google.com/p/go.crypto ; hg heads
  <menn0> wwitzel3: what do you get?
  <perrito666> natefinch: that might depend on your patience :p (documenting as we speak)
  <menn0> wwitzel3: for me, I'm not on the version listed in dependencies.tsv. which could be the problem. godeps issue?
  <menn0> wwitzel3: i'll manually switch to the version in deps.tsv and see if that resolves the issue
  <menn0> wwitzel3: never mind. it turns out I don't understand mercurial well enough. the right command to see the checked out rev is "hg id" and that shows the version matching deps.tsv
  <menn0> wwitzel3: i'll keep digging
  <menn0> waigani: I just did a successful 1.20 to current master upgrade and I can see Ian's fix landed yesterday so you should be good to try your manual test again
  <waigani> menn0: awesome, thanks for checking that
  <fwereade> perrito666, wwitzel3, katco: sorry, I've been in bed most of the day
  <fwereade> perrito666, I didn't :(
  <fwereade> perrito666, mail me with a link (again I presume...)
  <perrito666> fwereade: feeling better?
  <perrito666> fwereade: you are lucky that my backlog is long :p or that would make absolutely no sense
  <fwereade> perrito666, I'm not planning to do lots of work right *now*
  <fwereade> perrito666, but, yeah, hopefully will be with it again tomorrow
  <wwitzel3> fwereade: no worries, hope you feel better :)
  <fwereade> wwitzel3, cheers
  <perrito666> fwereade: for whenever you return, we might have broken your uniter context remaining commits
  <fwereade> wwitzel3, what can I do for you?
  <fwereade> perrito666, I had a super-quick look back just now, yeah
  <perrito666> sorry
  <fwereade> perrito666, looks like I broke things pretty hard myself
  <fwereade> perrito666, whatever needed to be done, needed to be done
  <fwereade> perrito666, I will try to catch up
  <katco> fwereade: no worries at all. hope you're feeling better
  <menn0> waigani: here's that EnvironUUID branch. Can you have a look please? http://reviews.vapour.ws/r/308/
  <wwitzel3> fwereade: well after rebasing master I broke, I was before doing relationer.Context() and then calling UnitNames() on that context.
  <fwereade> katco, yeah, will try to sleep better tonight :)
  <wwitzel3> fwereade: that is for the remoteUnit inference method
  <fwereade> wwitzel3, ah, sorry
  <waigani> menn0: will do
  <katco> fwereade: i have some material for you to review which might help put you to sleep :)
  <fwereade> wwitzel3, I think you can just use .ContextInfo().MemberNames
  <waigani> fwereade: thanks for the recommendations :)
  <wwitzel3> fwereade: I see that I need to now get the RelationContext to get UnitNames, so was wondering if I should ... oh ok
  <fwereade> wwitzel3, that's what will be called in the factory to determine membership
  <wwitzel3> fwereade: I will give that a shot right now, thanks
  <fwereade> wwitzel3, or since you're in-package, .state.Members(), but that's a map
  <wallyworld_> menn0: you have a minute for a chat?
  <waigani> menn0: done
  <menn0> waigani: thanks
  <menn0> wallyworld_: sure
  <wallyworld_> https://plus.google.com/hangouts/_/canonical.com/tanzanite-stand
  <alexisb> katco, ping
  <katco> alexisb: howdy
  <alexisb> howdy!
  <katco> :)
  <katco> what can i do for you?
  <alexisb> I am working on milestone breakdowns, was wondering where you were with the LE work you and fwereade broke down in brussels
  <katco> alexisb: working on that right now actually. i'm wrapping up the 1st draft of the spec
  <katco> alexisb: then wallyworld_ will give it a once-over to make sure i'm not wasting fwereade's time, and we'll propose it for feedback
  <alexisb> katco, ah ok, so official work has not begun yet
  <katco> alexisb: well, i consider the architecture part of the work ;)
  <alexisb> lol
  <alexisb> very true
  <alexisb> but that answers my q I will make sure to account for it in the roadmap
  <jw4> ericsnow: great comments; I'll respond to some of the points and update the PR.  Thanks
  <alexisb> thanks!
  <ericsnow> jw4: cool
  <katco> alexisb: sure thing! good talking with you again
  <alexisb> dido
  <wallyworld_> menn0: bug 1387884 raised
  <mup> Bug #1387884: juju upgrade from 1.20 to 1.21alpha3 is broken <regression> <upgrade-juju> <juju-core:In Progress by menno.smits> <https://launchpad.net/bugs/1387884>
  <menn0> wallyworld_: thanks
  <menn0> waigani: i've just been chatting to wallyworld_. part of the recent upgrade problems is due to the splitting out of upgrade steps by alpha releases
  <waigani> menn0: oh really>
  <waigani> ?
  <fwereade> katco, had a brief chat with wallyworld_ about it a day or two ago, I presume he passed on what needed to be?
  <menn0> waigani: it means when an upgrade from 1.20 to 1.21 alpha3 happens, the steps are really being run in the wrong order
  <katco> fwereade: indeed he did, thank you
  <menn0> waigani: going to merge them back into one list for 1.21 now
  <fwereade> katco, cool -- broadly sane in your estimation?
   * wallyworld_ afk for a bit to have breakfast etc
   * katco has a good... "not manager but manages people"
  <waigani> menn0: I don't follow. why the wrong order?
  <katco> fwereade: yes absolutely
  <menn0> waigani: if you upgrade from 1.20 to 1.21alpha3 the steps for alpha1 are run, then alpha2, then alpha3
  <fwereade> katco, excellent, so long as it really is
  <menn0> waigani:  the steps for alpha3 have the machine and instancedata upgrade
  <fwereade> katco, it's your job to tell me when I'm being crazy
  <fwereade> katco, everyone else's too ofc
  <katco> fwereade: do i constrain my comments to professional work?
  <menn0> waigani: which means the steps for alpha1 and alpha2 have trouble because they sometimes need to be able to look up machines
  <menn0> waigani: because the running software is actually alpha3
  <katco> fwereade: ;)
  <menn0> waigani: the steps shouldn't be split out by alpha version
  <fwereade> katco, I'm grinning broadly but not yet able to formulate a coherent response
  <waigani> menn0: got it
  <menn0> waigani: I'm fixing now.
  <menn0> waigani: alpha3 is being held up until this is sorted (should be easy though)
  <katco> fwereade: haha ok good the joke made it across OK. that pause is always nerve wracking :p
  <waigani> menn0: I've hit a really interesting bug, in the process of hunting it down. A test failed once I merged the machines branch with the sequences branch. I'll touch base with you on it later
  <menn0> waigani: ok cool
  <perrito666> ok EOD, have a nice one people
  <menn0> wallyworld_: if we have upgrade steps defined for 1.21 (final) they currently won't run when an upgrade to a 1.21 alpha or beta release happens
  <menn0> wallyworld_: I can tweak the upgrade steps logic to ignore the release tag (i.e. alpha/beta)
  <menn0> wallyworld_: or we can define the upgrade steps against alpha1
  <menn0> wallyworld_: I prefer the former
  <menn0> wallyworld_: but it does reduce our flexibility a little
  <menn0> wallyworld_: what do you think?
  <wallyworld_> menn0: my vision was always for the release tag to be ignored, thus we would be testing upgrade steps along the way as if it were final release, so the former
  <menn0> wallyworld_: ok will do
  <wallyworld_> we hadn't really got to the point of needed to do this till now
  <wallyworld_> so it upgrades were sort of done initilly as a mvp
  <wallyworld_> with improvements as needed
  <menn0> it should be easy to fix
  <menn0> and it's just the current crop of upgrade steps
  <menn0> that are exposing problems
  <menn0> this is also fallout from the release number change
  <wallyworld_> yeah
  <waigani> wtf
  <menn0> katco: I love the tabular status output btw. I use it all the time.
  <katco> menn0: so glad to hear it! i love it myself :)
  <waigani> katco: I like the oneliner, thanks :)
  <katco> waigani: np :)
  <katco> really i just coded it all up. not my ideas... thank you to ecosystems/tanzanite for guidance
  <waigani> menn0: so a sequence doc is being entered into the db with an old id. I can't track down where from (yet)
  <menn0> waigani: sounds fun :)
  <waigani> #$@$
  <ericsnow> wallyworld_: I gave you a ship-it on #302 but you'll need someone else to follow up. :)
  <wallyworld_> ericsnow: thank you, really appreciate it; the streams stuff can be hairy
  <ericsnow> wallyworld_: is it hairy because you pulled your hair out while writing it <wink>
  <wallyworld_> ericsnow: yes, yes i did. and now you've done a review, you are an expert and fair game to write the next simplestreams patch :-P
  <ericsnow> wallyworld_: sorry, you're breaking up, I didn't catch that, going into a tunnel now
  <wallyworld_> lol
  <alexisb> menn0, wallyworld_ davecheney was one of you designated lead in thumper's absence?
  <alexisb> oops
  <menn0> alexisb: i don't believe so
  <wallyworld_> not me
  <alexisb> menn0, ack
  <alexisb> sorry wallyworld_ I meant waigani
  <wallyworld_> ah yeah :-)
  <alexisb> davecheney, we have had a request come in that requires some arm skills, I am going to send you and mwhudson a note
  <alexisb> davecheney, please work with me on priorities compared with other tasks
  <menn0> wallyworld_: I have a branch to sort out the 1.21 upgrades
  <wallyworld_> menn0: awesome, i'll review. did you test live?
  <menn0> wallyworld_: all the steps are currently assigned against 1.21-alpha1
  <menn0> wallyworld_: yep, tested an upgrade of a complex env from 1.20
  <wallyworld_> not 1.21?
  <menn0> wallyworld_: that will come in a later change today
  <wallyworld_> +1
  <menn0> wallyworld_: but this change solves the immediate problem
  <wallyworld_> first priority is to unblock the release
  <wallyworld_> yep
  <menn0> wallyworld_: pushing now
  <wallyworld_> tyvm
  <menn0> wallyworld_: nothing like impending family commitment to focus your efforts :)
  <wallyworld_> :-)
  <menn0> wallyworld_: http://reviews.vapour.ws/r/309/
  <wallyworld_> looking
  <wallyworld_> menn0: +1, ship that sucker
  <menn0> wallyworld_: thanks. merging now.
  <wallyworld_> no, thank you
  <anastasiamac> ericsnow: thnx for the review :-) i agree readibility is important
  <ericsnow> anastasiamac: glad to help :)
  <fwereade> waigani, talk to jw4 re sequence documents
  <anastasiamac> ericsnow: i also agree to not changing code unrelated to my patch...
   * fwereade really should sleep now
  <waigani> fwereade: sorted it, thanks
  <ericsnow> anastasiamac: no worries, sometimes it's okay to slip a few little things like that in
  <fwereade> anastasiamac, ericsnow: it's a hard balance, but I will not complain about driveby readability fixes that don't change behaviour
  <ericsnow> fwereade: +1
  <fwereade> anastasiamac, ericsnow: and if you spot such things, and separate them out into separate PRs, I will be positively delighted
  <anastasiamac> fwereade: thnx :-) will aim to delight u positively :-)
  <fwereade> anastasiamac, ericsnow: because apart from anything else, separating them out makes it possible to do them that little bit better
  <jw4> fwereade: (waigani) fwiw, we're using sequence() far less widespread than I thought, and I think we can fold it into the same transaction that we use it in (need to verify though)
  <fwereade> jw4, waigani: my main concern there is that sequence() output ends up user-visible
  <jw4> fwereade: I see.  If it's within the same transaction that seems acceptable
  <fwereade> jw4, waigani: and if we're returning values that depend on sequence() result -- ie "what unit we just created" -- we don't want to unnecessarily burn sequence numbers
  <fwereade> jw4, waigani: actually, dammit, it is more complex
  <fwereade> waigani, you said it was sorted? expand super-quickly please?
  <fwereade> jw4, the issue there is
  <fwereade> jw4, that if we return "someservice/123" from a sequence() call outside a txn
  <fwereade> jw4, we can be sure we really did create someservice/123
  <fwereade> jw4, if we're doing a sequence call inside a txn
  <fwereade> jw4, we either lose visibility on what we actually created
  <fwereade> jw4, or we end up adding an effective txn-revno assert on the sequence doc
  <fwereade> jw4, thus screwing parallelisability
  <fwereade> jw4, am I making sense?
  <jw4> fwereade: yes...
  <jw4> fwereade: but...
  <wallyworld> perrito666: did you get the msg i just posted? my connection may have dropped?
  <jw4> fwereade: I'm thinking we might be able to guess at the next sequence number, set the id's for both the sequence and the usage of it in a multi-try txn loop, and then assert in the transaction that the number is what we predicted
  <fwereade> jw4, so we can probably guess pretty well, that's not the issue
  <jw4> if the assert doesn't pass it fails the txn loop just like any other contention
  <jw4> no, I mean guess *before* we use it
  <fwereade> jw4, the issue is that the *existence* of the assert enforces txn serialization
  <fwereade> jw4, thus preventing parallel uses of the same sequence doc
  <jw4> fwereade: hmm - since all our multi doc transactions are two phase commit's that's what would happen anyway right?
  <fwereade> jw4, well, that's the advantage of assigning sequence numbers outside the txn
  <fwereade> jw4, there may well be
  <jw4> fwereade: if we want to ensure ordered sequence numbers we give up parallelism (or is it concurrency? i forget)
  <fwereade> jw4, -- in the add-unit case, there certainly are --
  <fwereade> jw4, other considerations that prevent concurrent db ops
  <jw4> fwereade: brb - keep going
  <perrito666> wallyworld: I did not
  <fwereade> jw4, but in the general case we need to choose between knowing what sequence number we picked, and allowing parallel ops against the same sequence
  <wallyworld> <wallyworld_> perrito666: hi, bug 1320543
  <wallyworld> <wallyworld_> did you mean to assign that to yourself?
  <wallyworld> <wallyworld_> anastasia is working on it already, but it hadn't been assigned to her (sorry)
  <mup> Bug #1320543: debug-log uses internal names for filtering <debug-log> <usability> <juju-core:Triaged by anastasia-macmood> <https://launchpad.net/bugs/1320543>
  <perrito666> wallyworld: I did but did not yet work on it so be my guest re-assigning it
  <wallyworld> already done :-)
  <fwereade> jw4, wait, actually, I'm net even sure it's a choice
  <perrito666> I just started reading code and was suddenly redirected to fix one of my PRs
  <anastasiamac> wallyworld: thnx :-)
  <fwereade> jw4, atucally that's an overstatement
  <fwereade> jw4, so
  <fwereade> jw4, picking the sequence inside the txn *requires* that we sacrifice parallelism, because we have to be able to assert the state of the sequence doc
  <fwereade> jw4, in some situations that's acceptable
  <fwereade> jw4, ie actions, I think, but push back if you're not 100% convinced
  <fwereade> jw4, it's *coincidentally* acceptable even in the add-unit case
  <fwereade> jw4, because the write to the service doc serialises the txns anyway
  <fwereade> jw4, and unless we get *really* clever that's unavoidable, because refcount
  <fwereade> jw4, even if it's not on the service doc
  <fwereade> jw4, I think I need a refresher on where we're using sequence() :)
  <jw4> fwereade: good news
  <jw4> fwereade: not very many places
  <fwereade> jw4, brb myself, keep talking yourself :)
  <jw4> fwereade: grep -r 'sequence(' yields 3 usages : relations, actions, machines
  <jw4> fwereade: now, maybe other collections are using the same mechanism not with the state/sequence.go helper func
  <jw4> need to dig on that one
   * jw4 is juggling bread in the oven - brb
  <jw4> (figuratively)
  #juju-dev 2014-10-31
  <wallyworld> axw: hiya, i'd love a second lgtm on http://reviews.vapour.ws/r/302 if possible
  <fwereade> jw4, what was the last thing you saw me say?
  <fwereade> jw4, last thing I saw you say was (figuratively)
  <wallyworld> perrito666: ping
  <wallyworld> bug 1384001 was marked as Fix Committed in juju core, except it's not. It's only fix committed in core when the dependencies file is updated
  <mup> Bug #1384001: Juju doesn't retry hard enough when destroying MAAS environments <cloud-installer> <destroy-environment> <maas-provider> <Go MAAS API Library:Fix Committed by rvb> <juju-core:In Progress by wallyworld> <MAAS:Fix Committed by rvb> <https://launchpad.net/bugs/1384001>
  <jw4> fwereade: I wandered off when your autocomplete quit working
  <jw4> fwereade: :)
  <jw4> fwereade: I guess I don't see any way around serializing sequence() if we want to ensure ordered sequence numbers
  <jw4> fwiw, I found that the unit sequence number is using a similar but less stringent mechanism using 'unitseq' right on the service doc
  <wallyworld> perrito666: ignore me
  <wallyworld> my godeps failed
  <wallyworld> sorry
   * jw4 eod-ish
  <axw> wallyworld: will take a look in a sec
  <wallyworld> np, ty
  <cmars> of all the api client tests, which one is a good example of how to mock test an api client? I don't want to follow the wrong example.
  <perrito666> wallyworld: I just came back from dinner, I am pretty sure I committed that and juju bot thinks likewise
  <wallyworld> perrito666: yeah, forget i even mentioned it, my master was out of date and godeps didn;t work, so toally my mstake
  <perrito666> that is why you changed the state of the bug I assume :)
  <perrito666> ok then, going to sleep, see you all tomorrow
  <wallyworld> perrito666: see ya. i did change the bug back to waht it shold be
  <wallyworld> cmars: sadly, in juju/api/client, I am not aware of a test suite which doesn't run up the full stack. it's todo to not do that, but i don't thinks it's been done yet anywhere :-(
  <waigani> menn0: wired up the step: http://reviews.vapour.ws/r/310/
  <menn0> waigani: will take a look shortly
  <menn0> waigani: np
  <menn0> wallyworld: here's part of the upgrade steps rework: http://reviews.vapour.ws/r/311/
  <menn0> part 2 even
  <wallyworld> menn0: ok, ta. just about to go to lunch, will look when i get back
  <menn0> wallyworld: np
  <menn0> waigani: change looks good
  <waigani> menn0: thanks
  <wallyworld> menn0: one query in the review
  <menn0> wallyworld: looking
  <menn0> wallyworld: just responded
  <wallyworld> looking
  <wallyworld> menn0: maybe make it so that if Tag != "", then run steps for fromVersion? But if we are already fully on fromVersion, seems unnecessary to run steps again?
  <menn0> wallyworld: that's how it works already I believe
  <wallyworld> hmmm, ok. if thats how it works now, then we shouldn't change the behaviour in this branch perhaps. but seems wrong
  <menn0> wallyworld: this branch changes the behaviour in terms of the target version
  <menn0> wallyworld: those 2 extra tests just confirm already existing behaviour
  <menn0> wallyworld: it makes sense because 1.21-alpha/beta < 1.21
  <wallyworld> yeah, that appears true. i was just surpised to see it running steps unnecessarily
  <wallyworld> yes when Tag != "", run the steps
  <menn0> wallyworld: so if you're upgrading past 1.21 the steps for 1.21 should get run
  <wallyworld> yes, but not if you're upgrading from 1.21
  <menn0> wallyworld: I wouldn't say it's running steps unnecessarily. more steps may have been added between alpha/beta and final.
  <wallyworld> yes, and in those cases, Tag != ""
  <wallyworld> and we don't support upgrading from dev versions
  <wallyworld> but anyway, if it's existing behaviour, no need to hld up this branch
  <menn0> wallyworld: ok
  <wallyworld> just surprised me is all
  <wallyworld> ie i don't agree with it :-)
  <menn0> wallyworld: I think the behaviour is ok but whatever :)
  <menn0> wallyworld: I'll make that test change you suggested and re-push
  <wallyworld> well, so long as the steps are idempotent, it diesn't matter
  <wallyworld> but there's a cost to running unnecessary steps
  <wallyworld> just added a ship it, thanks for extra 1.22 change
  <menn0> wallyworld: the problem is I guess, that we don't know which steps were run when coming from a alpha/beta release
  <wallyworld> yes, but we don't officially support upgrading from alpha/beta
  <menn0> wallyworld: I'm not adding 1.22 to the test data btw. I just changed the releases to be 1.20-alpha/beta to 1.21 to take advantage of what's already there.
  <wallyworld> yep, +1
  <menn0> wallyworld: thanks. will merge now.
  <wallyworld> thanks for fixing
  <menn0> wallyworld: FYI fwereade has asked me to look at the relation scope issue for subordinate relations
  <menn0> wallyworld: where they should be scoped to the machine, not the principal service
  <menn0> wallyworld: apparently it's affecting Landscaope
  <menn0> Landscape
  <menn0> wallyworld: ring any bells?
  <wallyworld> menn0: vaguely
  <wallyworld> menn0: as yes
  <wallyworld> they want to add a single global chamr on a machine
  <menn0> wallyworld: he asked me check with you regarding the priority of this
  <menn0> wallyworld: that's the one
  <wallyworld> menn0: if you ask landscape, very high
  <wallyworld> we did rule it out of cope this cycle
  <wallyworld> but
  <wallyworld> if you had bandwidth to fix, that would be amazeballs. but it will be big
  <menn0> wallyworld: he's asked me to look because i've just been in that code to do the env UUID upgrade
  <wallyworld> and not just landscape want it, wil be very useful
  <wallyworld> ok, can you then see if you can spec out what the changes will be, so it can be estimated and scoped?
  <menn0> I don't strictly have the cycles. thumper wants me focused on the multi-environment work
  <wallyworld> ok
  <wallyworld> in that case, leave it, as we already told them it would not be in scope this cycle
  <menn0> i'll do the initial thinking and scoping and see how it looks
  <wallyworld> ok, ta, anything would be good as ti will be more than we have now
  <jog> good afternoon wallyworld, FYI bug 1364410 has become consistently reproducible again in Juju CI
  <mup> Bug #1364410: API server fails to start with "address already in use" in MachineSuite tests <ci> <intermittent-failure> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1364410>
  <wallyworld> hi
  <wallyworld> i wasn't aware of that bug, thanks for letting us know
  <wallyworld> jog: sadly, looking at the bug report, if it is the issue i think it is, it will not be easy to fix
  <jog> it was a blocking CI regression bug in early September but sinzui demoted it after it became intermittent and we were able to get a blessed build revision
  <wallyworld> any fix will likely be a lot of work from what i now remember after reading the bug, as it changes the way tests are set up
  <wallyworld> so i'm not sure off hand how timely a fix can be
  <wallyworld> i'll take a look, just got to finish some wip
  <jog> wallyworld, ok currently the last 10 ppc64el unit test runs have failed
  <wallyworld> i thought ppc64el was non voting?
  <jog> it was non-voting for awhile but once the compiler test was worked around/disabled we set it back to voting
  <wallyworld> maybe not
  <wallyworld>  ah ok
  <wallyworld> i guess a call needs to be made if we want to ship alpha3 this week
  <jog> maybe this test needs to be disabled as well, so we can at least gain coverage from the rest? Depending on how risky this issue is to users?
  <jog> if a fix is not coming soon, disabling the entire suite is not ideal
  <jog> i.e. disable == make it non-voting
  <wallyworld> if it's only on ppc64el, maybe we can skip to get alpha3 out, and then unskip once that happens
  <jog> wallyworld, unit tests are passing for other arches but there are a number of other non-unit test failures preventing a blessed revision
  <wallyworld> jog: just local upgrades though i think?
  <wallyworld> and a fix is currently going through CI now
  <jog> I also see a  functional-ha-backup-restore and  kvm-upgrade-trusty-amd64 failure... I'll go see if I can find out why
  <wallyworld> ok, ta
  <wallyworld> axw: a small one http://reviews.vapour.ws/r/313
  <wallyworld> ty :-)
  <axw> wallyworld: done
  <wallyworld> ty
  <wallyworld> jog: i think i know why local upgrade tests are failing
  <wallyworld> juju --show-log set-env -e local-upgrade-trusty-amd64 tools-metadata-url=
  <wallyworld> tools-metadata-url is deprecated, replaced by agent-metadata-url
  <wallyworld> config will be migrated on start up, but old attr name is not valid in set-env
  <wallyworld> so test script needs to be tweaked
  <jog> wallyworld, hmm... I thought we could not depreciate something like that and would need to support both  tools-metadata-url and agent-metadata-url until the major version changes?
  <wallyworld> we do support both
  <wallyworld> in config
  <wallyworld> the approach is to print a warning when deploying an environment, and copy across the value for people
  <wallyworld> thus if you have an environments.yaml with the old attr, it will be migrated
  <wallyworld> when you deploy your environment
  <wallyworld> we've not supported set-env in that way before
  <jog> I see... you said that above... so it's just set-env you think we need to update in the test scripts
  <wallyworld> not sure if that's something (set-env) we need to do or not
  <wallyworld> just set-env yes
  <wallyworld> for end customers, tools-metadata-url is only ever read only
  <wallyworld> they could update it, but there's no reason to
  <wallyworld> if stakeholder strongly feel we need to add deprecated handling to set-env, we could do that
  <wallyworld> it just hasn't come up before as an issue
  <jog> I could see it being an issue if enterprise user scripts break. CI certainly has scripting that's setting tools-metadata-url
  <jog> those same scripts are used for <1.21, so we'll need to add some checking
  <axw> wallyworld: I'm going to take a look at #1387421, then I'll look at the scp bugs
  <mup> Bug #1387421: cmd/juju: add help on placement directives (zones, maas host name) <cmdline> <docs> <juju-core:Triaged by axwalk> <https://launchpad.net/bugs/1387421>
  <wallyworld> ok. i'd be surprised in any enterprise user set tools-metadata-url after bootstrapping. but other depecated attrs, maybe they do
  <wallyworld> axw: ok, sounds good. i'm about to look into the critical blocker to see if there's any palatable fix
  <axw> okey dokey
  <wallyworld> jog: if i'm not around later past my EOD, could you ask curtis about his opinion on the set-env handling of deprecated attributes. it's not something we've supported in the past but if we need to we can add it
  <jog> wallyworld, I think he's out Friday
  <wallyworld> ok, we can talk next week
  <wallyworld> maybe if i get time i can look to add it
  <jog> If I had to bet though, risk of breaking external customers in the same way this breaks CI will make it an issue :-/
  <wallyworld> except that external customers will not set tools-metadata-url
  <wallyworld> but yeah, for other maybe; i'd have to check to see what ones have been deprecated
  <jog> I know we're doing things a bit different, so maybe I'm wrong
  <wallyworld> for customers tools-metadata-url is static
  <wallyworld> set once in yaml
  <wallyworld> main use is for private clouds, and those are i think almost always openstack, and suck customers would use keystronr catalog instead i would think
  <wallyworld> jog: i'm looking at http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/ as linked in the bg - i can't see test runs with evidence of that same bug
  <wallyworld> i've looked at a few console logs, there's failures, but they are openstack reated
  <wallyworld> or upgrade related
  <wallyworld> not related to the api server port issue
  <jog> wallyworld, I could have mis-diagnosed it as the same... I initially opened bug https://bugs.launchpad.net/juju-core/+bug/1387936 and it has a log with the failure we're seeing
  <mup> Bug #1387936: Unit tests on ppc64el trusty failing with streams resources not found <ci> <ppc64el> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1387936>
  <wallyworld> jog: there errors look like the control hooks aren't running, which is indicative of a compiler issue
  <wallyworld> jog: that bug you just pasted isn't a bug
  <wallyworld> juju will look for simplstream data in several places, and with debug on, it logs the places it tries but is unsuccessful
  <wallyworld> this helps developers diagnose issues if people complain their system won't bootstrap
  <wallyworld> the compiler issue i mention above in the past has been a gccgo vs golang go issue
  <wallyworld> they handle stack frames differently
  <wallyworld> so some low level stuff fails on gccgo
  <wallyworld> jog: on the basis of the above, it would be great if we could unblock landings from bug 1364410
  <mup> Bug #1364410: API server fails to start with "address already in use" in MachineSuite tests <ci> <intermittent-failure> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1364410>
  <jog> ok... I thought all the compiler issues had been worked around. mgz would working with various people on that... I can drop the severity to high for now.
  <wallyworld> jog: i have no answer for why the hooks aren't being run - in the past it's been a compiler issue
  <jog> ok, it's set back to high
  <wallyworld> ty :-)
  <wallyworld> jog: i'll start the work to handle deprecated settings in set-env, so when it lands you can keep the CI scripts unchanged
  <jog> wallyworld, that would be awesome and less risky, in case of possible customer scripting... It's 11pm here I'm about to turn in, anything else before I go?
  <wallyworld> jog: nah, thanks for your help
  <jog> thanks to you as well
  <jog> FYI, I moved the functional-ha-backup-restore test from HP to AWS and it passed, so possibly HP cloud flakiness
  <jog> wallyworld, would you please comment in bug https://bugs.launchpad.net/juju-core/+bug/1387936 with your suspicions about the compiler?
  <mup> Bug #1387936: Unit tests on ppc64el trusty failing <ci> <ppc64el> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1387936>
  <hazmat> finally re azure.. instance level public ips.. http://msdn.microsoft.com/library/azure/dn690118.aspx
  <hazmat> axw, fwiw re storage, gce just added its first local disk availability .. not by itype.. by api allocation at creation (81usd per month per dev) https://cloud.google.com/compute/docs/local-ssd
  <axw> hazmat: 81 USD???
  <axw> sounds like a lot
  <axw> so they never had ephemeral storage before?
  <hazmat> axw, no
  <hazmat> axw, they always did net-storage afaics, they don't have sizing options (its per gb on local ssd) and color of black (model t ref) is 375gb
  <axw> so I see. reasonably hefty
  <mattyw> morning all
  <TheMue> morning
  <mattyw> TheMue, morning
  <voidspace> TheMue: dimitern: ok if I'm late to standup by ten minutes?
  <dimitern> voidspace, np
  <voidspace> thanks
  <voidspace> dimitern: TheMue: omw
  <hazmat> davecheney, ping
  <voidspace> gsamfira: hey
  <davecheney> hazmat: ack
  <voidspace> gsamfira: dimitern asked if reboot deals with debug-hooks in the same way it does juju-run?
  <mattyw> is fwereade around today or off still?
  <fwereade> mattyw, I'm here
  <fwereade> mattyw, what can I do for you?
  <perrito666> morning all
  <wallyworld___> axw: thanks for scp fixes
  <wallyworld___> fwereade: since you are ocr, can i pretty please get a review on a critical bug fix for alpha3 http://reviews.vapour.ws/r/317/ ?
  <axw> wallyworld___: nps
  <fwereade> wallyworld___, just toasting a sandwich, on it in 5
  <wallyworld___> sure, thanks
  <fwereade> wallyworld___, LGTM, maybe one test tweak
  <wallyworld___> fwereade: ty
  <perrito666> man in this coutry its easier to get uranium that a port replicator for my lenovo model
  <natefinch> perrito666: sorry, we'll have to move our 1:1, I have to go catch an escaped chicken
  <perrito666> natefinch: np
  <perrito666> best reason ever, btw
  <natefinch> I live to amuse.
  <perrito666> natefinch: nah, no pics nor vids of you chasing a chicken, thats not fun
  <natefinch> perrito666: closest I can come (my daughter, not me): https://fbcdn-sphotos-g-a.akamaihd.net/hphotos-ak-xpa1/v/t1.0-9/10303745_10152646984367028_7439232690755188047_n.jpg?oh=4e262e18b2e2723819b2bb8e2b250b53&oe=54E3659E&__gda__=1423914888_2c43b0a7691dfd8982b5deb1773a1595
  <perrito666> is that the runnaway chicken?
  <natefinch> nah, just one she picked up a while back.
  <natefinch> I have surprisingly few pictures of the chickens :)
  <perrito666> natefinch: she smiles too much I dont think she knows what will happen to that chicken in the future
  <natefinch> perrito666: we are actually due for a "harvest" in a week or so
  <wwitzel3> yum
  <perrito666> a good things about chicken is that they loose the cute factor when they are ready to eat
  <natefinch> Yeah, the only hard part is the initial act of going from chicken the bird to chicken the meat.  Really, once the feathers are off, they're not so much different than what you see in the grocery store.
  <natefinch> perrito666: moonstone?
  <perrito666> well once you tags.Strip("<head>") they kind of no longer look like chickens anyway
  <natefinch> haha yeah
  <alexisb> natefinch, does your team have time to take a look at https://bugs.launchpad.net/juju-core/+bug/1388073
  <mup> Bug #1388073: set-env no longer accepts empty values <ci> <compatibility> <regression> <set-env> <juju-core:Triaged> <https://launchpad.net/bugs/1388073>
  <alexisb> we need to get that resolved asap
  <natefinch> alexisb: ok
  <alexisb> thanks
  <tasdomas> natefinch, alexisb - I'm fixing #1388073
  <mup> Bug #1388073: set-env no longer accepts empty values <ci> <compatibility> <regression> <set-env> <juju-core:Triaged> <https://launchpad.net/bugs/1388073>
  <alexisb> tasdomas, thanks!
  <mgz> tasdomas: okay, I want to yell at you
  <mgz> why cd17b6f3
  <tasdomas> mgz, because we had that same piece of code pasted over 4 different places?
  <mgz> you wrote it wrong again!
  <mgz> straight up regresses bug 1379208
  <mup> Bug #1379208: TestAddMetrics fails on gccgo <ci> <gccgo> <regression> <test-failure> <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1379208>
  <perrito666> Hey I keep having power outages, running to greener pastures now
  <mgz> tasdomas: see bug 1387936
  <mup> Bug #1387936: Unit tests on ppc64el trusty failing <ci> <ppc64el> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1387936>
  <mgz> has link to one of the test runs
  <alexisb> mgz, in comment #3 of that bug, (2) Is Ian's change for a critical bug fix or is it something we can revert?
  <mgz> alexisb: I'm still trying to tease out if it's an accidental merge issue or an intended change, there are like 70 files touched, which makes a straight revert annoying
  <mgz> I'll add more details in a second
  <alexisb> mgz, ack, let core know how we can help
  <voidspace> yay, back online
  <voidspace> not sure if it's permanent or temporary
  <alexisb> and thanks mgz for triaging!
  <voidspace> they're still futzing with the line
  <alexisb> voidspace, are you one of those that gets nervous when they are disconnected
  <voidspace> alexisb: might have been the most productive hour of the day ;-)
  <voidspace> alexisb: although I compulsively click things when I'm thinking
  <alexisb> lol
  <voidspace> alexisb: and it's distracting when the clicks do nothing...
  <katco> fwereade: ping
  <fwereade> katco, pong
  <katco> fwereade: what i think is a reasonable draft of that spec is up
  <fwereade> katco, lovely, thanks
  <katco> fwereade: wallyworld didn't see anything too egregious, so i think we're ready for your +1
  <katco> fwereade: (or not as the case may be) :)
  <fwereade> katco, brilliant, I *might* get to it tonight but I'm afraid I'm off to a halloween thing at laura's school imminently
  <katco> fwereade: how dare you care about your family!
  <fwereade> katco, ;p
  <katco> fwereade: totally understand :) have fun and HAPPY HALLOWEEN! :D
  <fwereade> katco, laura is dressed as a dinosaur :)
  <katco> fwereade: oh dear. pictures!!
  <katco> fwereade: mine is a ladybug :)
  <fwereade> aww :)
  <katco> alexisb: is yours going as a tractor or something similar? :)
  <alexisb> katco, he is going as a lion as he loves the "roar"
  <katco> alexisb: aw :)
  <natefinch> katco: have you seen the ladtbug girl books?
  <katco> natefinch: i have not! link please!!
  <natefinch> http://www.ladybuggirl.com/
  <katco> natefinch: ok you are DIRECTLY responsible for me not being productive for the next bit of time.
  <natefinch> katco: great art and good morals and strong girl lead
  <katco> natefinch: love it
  <natefinch> katco: yep :)
  <natefinch> katco: Lily, our eldest: https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-xpf1/v/t1.0-9/p526x296/10624919_10152847646912028_8733903205136776490_n.jpg?oh=1ba16b1062ac2694854a3a2c57886f6a&oe=54E5B3A7&__gda__=1423915697_a6cde75ffa599528b4531dab195d58d5
  <katco> natefinch: oh my she is DARLING!
  <katco> is that... a donut on a rope? haha
  <natefinch> haha.. halloween tradition, like bobbing for apples, but more sanitary
  <katco> lol
  <natefinch> our younger daughter, ZoÃ«: https://scontent-a-iad.xx.fbcdn.net/hphotos-xap1/v/t1.0-9/10538654_10152847638537028_3233943882042977754_n.jpg?oh=8f5b74c34d729c2175c4a99e3895f8d7&oe=54EB41C1
  <katco> natefinch: that smile is infectious :D
  <natefinch> the giggles even more so :)
  <katco> you have some cute ladies :D
  <natefinch> katco: thanks :)
  <alexisb> natefinch, love the picture of Zoe!
  <alexisb> Lily is also very cute, but Zoe's smile is awesome
  <tasdomas> natefinch, mgz, http://reviews.vapour.ws/r/320/ - should fix part of 1387936 and 1388073
  <mgz> tasdomas: thanks!
   * perrito666 is back
  <alexisb> mgz, so is the last blocking the commit from Ian?
  <mgz> alexisb: I'm still not sure where it started breaking
  <mgz> ideally someone in the US can follow up and track down the point - it's confused by one of the failures being a variable segfault when running the tests
  <alexisb> natefinch, I know you have tasks today, perrito666, katco, ericsnow, tasdomas, cmars ^^ can any of you help out
  <alexisb> we need to get 1.21 released so this is top priority
  <alexisb> fwereade, ^^
  <ericsnow> alexisb: I'll take a look but no one should let that stop them from looking too :)
  <mgz> ericsnow: basically, test with -compiler gccgo in provider/openstack is fubar
  <mgz> I'm not sure exactly which change regressed, it's very messy
  <ericsnow> mgz: that means I would need to set things up to test against an openstack provider?
  <mgz> ericsnow: nope just the unit tests
  <mgz> go test -compiler gccgo
  <ericsnow> mgz: k
  <tasdomas> could somebody take another look at http://reviews.vapour.ws/r/320/ ?
  <jrwren_> natefinch: <3 ladybug girl. I read all of those to my duahgter.
  <ericsnow> mgz: so should I expect "go test -compiler gccgo" to have failed? (it didn't)
  <mgz> it fails in provider/openstack for me
  <perrito666> well ubuntu upgrade plus power issues let my machine in a not so nice shape
  <ericsnow> mgz: I just did an "apt-get install gccgo" before running it (didn't have gccgo installed yet)
  <natefinch> jrwren_: Yeah, we got one as a gift, loved it, and have been getting all the rest of them.
  <perrito666> mgz: go test -compiler gccgo for provider/openstack or all of it?
  <ericsnow> mgz: never mind (was in the state directory)
  <mgz> perrito666: provider/openstack is the interesting bit
  <mgz> ericsnow: you have the failures/segfault now?
  <ericsnow> mgz: running
   * perrito666 runs all of provider just in case
  <natefinch> ls
  <natefinch> heh
  <voidspace> agent      cloudinit             dependencies.tsv   etc       mongo       rpc      upgrades
  <natefinch> holy shit it worked
  <voidspace> :-)
  <ericsnow> mgz: aborted ("munmap of stack space failed: errno 22") in apiserver
  <perrito666> mgz: ok  	github.com/juju/juju/provider/openstack	51.623s
  <perrito666> master
  <perrito666> go test -compiler gccgo github.com/juju/juju/provider/...
  <ericsnow> mgz: (still running)
  <natefinch> on master I can't even get the test to compile in that directory
  <natefinch> /tmp/go-build382239753/github.com/juju/juju/provider/openstack/_test/_testmain.go:52:15: error: reference to undefined identifier âtesting.MainStartâ
  <natefinch>   m := testing.MainStart(matchString, tests, benchmarks, examples)
  <natefinch>                ^
  <natefinch> FAIL	github.com/juju/juju/provider/openstack [build failed]
  <natefinch> hm... I wonder if I screwed myself by upgrading to go 1.4 yesterday
  <mgz> hm...
  <perrito666> mgz: I seem to be immune to these issues
  <natefinch> ahh, I think I know what it is... the new go tool running with the old gccgo ... funny
  <mgz> perrito666: gccgo version?
  <perrito666> gccgo (Ubuntu 4.9.1-16ubuntu6) 4.9.1
  <perrito666> ubuntu 14.10
  <ericsnow> mgz: segfault in github.com/juju/juju/environs/simplestreams and in github.com/juju/juju/provider/openstack
  <ericsnow> mgz: also, github.com/juju/juju/worker/uniter/jujuc failed
  <ericsnow> mgz: so more than just provider/openstack failed for me
  <mgz> the jujuc one tasdomas has branch up fixing
  <mgz> I think simplestreams/openstack is the same regression, but I have been failing to get it to pass by reverting the changes I think may be to blame
  <natefinch> I need one of these: http://goo.gl/OPhW6a
  <mgz> eheeh
  <ericsnow> mgz: the segfaults looked similar
  <ericsnow> mgz: "panic during panic"
  <mgz> ericsnow: you sometimes get other panics as well :)
  <natefinch> yay for gccgo
  <ericsnow> mgz: for the 2 segfaults: "/home/esnow/projects/go-workspace/src/github.com/juju/juju/environs/simplestreams/simplestreams_test.go:23" and /home/esnow/projects/go-workspace/src/github.com/juju/juju/provider/openstack/provider_test.go:29
  <tasdomas> mgz, ericsnow, natefinch - landing my fix
  <ericsnow> mgz: for both a separate goroutine was sitting at the same spot in provider/dummy/environs.go, in case that matters
  <natefinch> tasdomas: awesome, thanks for getting that in, I know it's late on friday for you
  <ericsnow> natefinch, tasdomas: +1
  <tasdomas> natefinch, it was my f-up to begin with
  <perrito666> mgz: can I give you any more info?
  <mgz> perrito666: paste me a clean run with -v?
  <perrito666> natefinch: no you dont
  <voidspace> wwitzel3: you need a review on 318
  <voidspace> wwitzel3: juju-run
  <natefinch> perrito666: haha
  <perrito666> I mean, really why would they print that, its not even a dynamic link
  <ericsnow> mgz: which issue are we looking at?
  <mgz> ericsnow: the segfaults
  <perrito666> mgz: http://pastebin.ubuntu.com/8763738/
  <perrito666> mgz: hardly informative
  <ericsnow> mgz: I mean the launchpad issue
  <mgz> ericsnow: so, what I *tried* to do was back to a known good change to narrow down what the cause was
  <ericsnow> mgz: yeah, which would that be?
  <mgz> but I have been failing to get a clean run, even though the test on jenkins was blue as of 29th rev f2292c20
  <mgz> I suspect it's either Ian's giant re-landing pr #988 or the later pr #996 but that seems to not be right?
  <ericsnow> mgz: same here (metrics-related compile failure)
  <ericsnow> mgz: at f2292c20
  <mgz> ericsnow: run godeps there?
  <ericsnow> mgz: that's better (running now)
  <ericsnow> mgz: same panic at f2292c20
  <mgz> yeah... hence my general confusion
  <wwitzel3> voidspace: please :)
  <perrito666> clearly its an issue with our toolchain versions
  <ericsnow> mgz: I get all sorts of failures (not panics) in the provider/openstack tests at ae4d5862a42c443e15f3fe49c2322010fecc83b3
  <mgz> ericsnow: run multiple times as well, the results are not consistent on the same version
   * ericsnow tries at a1a74d5ff07a30942c48b75ccd2fb1bbf8dad59a (where sinzui merged in from his alpha3 branch)
  <ericsnow> mgz: ah, okay
  <voidspace> wwitzel3: hmmm... my EOD now
  <voidspace> dammit
  <voidspace> wwitzel3: it looks pretty straightforward though
  <ericsnow> mgz: same results  (74 passed, 2 skipped, 5 FAILED, 1 PANICKED) at a1a74d5ff07a30942c48b75ccd2fb1bbf8dad59a
  <voidspace> wwitzel3: providing new flags and using those in NewRunContext to find the right context
  <voidspace> wwitzel3: there's at least one place returning a bare err that should maybe be errors.Trace(err)
  <voidspace> wwitzel3: I'll go jogging and if I get back & showered before our guests arrive I'll try and get a review in
  <mgz> ericsnow: right so, I think that's just the panic varying, sometimes it hard aborts the test run, sometimes it's recovered and you see the failures
  <ericsnow> mgz: ah
  <ericsnow> mgz: I'll try it a few times in a row then
  <wwitzel3> voidspace: appreciate it, I'll double check all my error returns, and yep, that is a good summary of what is going on
  <tasdomas> mgz, natefinch, voidspace, ericsnow, my PR is in core now
  <ericsnow> mgz: yep, I get the full panic sometimes and the failed tests sometimes
  <natefinch> tasdomas: huzzah!  nice work
  <natefinch> tasdomas: now get out of here. No one should be working that late on a Friday :)
  <ericsnow> mgz: no panic at c9630181b910dd96ac32868142fd9512442b7629
  <mgz> ericsnow: thank you!
  <ericsnow> mgz: trying to narrow it down
  <mgz> yeqah, that's good for me too
  <mgz> wait... noly ran one test, what did I mess up
  <ericsnow> mgz: never mind, I still get intermittent panics at c9630181b910dd96ac32868142fd9512442b7629
  <mgz> yeah, that's what I just saw
  <mgz> so... my guess is intermittent panics with gccgo on amd64 was already a thing
  <mgz> but that still doesn't help me with which change made the ppc test job fail reliably
  <ericsnow> mgz: yep, I get the same panic back to July
  <alexisb> ericsnow, mgz you not hitting known gccgo issues are you?
  <voidspace> I've gotta go
  <voidspace> guests arriving
  <voidspace> happy weekend everyone
  <ericsnow> alexisb, mgz: perhaps...regardless, it looks like the panics are an orthogonal issue
  <natefinch> happy weekend all
  <perrito666> ericsnow: care to explain? http://reviews.vapour.ws/r/298/#comment1932
  <ericsnow> perrito666: I'll respond there.
  <perrito666> tx
  <ericsnow> perrito666: done
  <perrito666> tx a lot
   * perrito666 keeps fixing
  <ericsnow> perrito666: sorry I haven't been able to review more or otherwise take a look at that review request
  <perrito666> ericsnow: np, I have my hands full, many of your comments have been useful either to fix things or to re-think why I wrote some things
  <perrito666> many of the things I didn't like where just reactions to the re-write in backups and therer
  <perrito666> fore not as nice as I would have liked at the moment
  <perrito666> ericsnow: ah I see what you mean, well I would have expected that pattern not to exist, but for the sake of consistency I will make it follow it
  <ericsnow> perrito666: thanks!
  <perrito666> man, it is specially hard to open files when your kb distribution has / only over the 7
  #juju-dev 2014-11-02
  <menn0> mwhudson, waigani: email standup again?
  <mwhudson> if you like
  <waigani> menn0: yep, writing mine now
  <mwhudson> you guys don't like talking as much as tim, hey? :)
  <waigani> hehe
  <menn0> mwhudson: that and, then thumper know what we've been up to while he's away
  <mwhudson> fair enough
  <waigani> menn0: strictLocalID branch not working out?
  <menn0> waigani: just not seeing the point
  <menn0> waigani: looking at it now
  <menn0> waigani: many of the uses of localID still need to be relaxed because they're in generic code that could see old or new style ids
  <menn0> waigani: so what we have now (localID and strictLocalID) might actually be fine
  <waigani> menn0: yeah, i thought that is what we concluded on Friday - but use strictLocalID unless you know you have to relax the check
  <menn0> waigani: yeah we were thinking that might be best
  <bodie_> I keep getting breakage in firewaller_test L139 TestGetMachinePorts ... mismatch at .Results[0].Ports[0].PortRange.FromPort: unequal; obtained 4321; expected 1234
  <bodie_> anyone else seeing this?
  <bodie_> maybe I just need to go get -u -v, let me see if that fixes it
  <menn0> bodie_: strange. works for me.
  <menn0> wallyworld: ping
  <wallyworld> hey
  <menn0> are you still looking at that blocker?
  <wallyworld> i fixed it last night, was hoping for a review from tim when  woke up, then only just realised he's away this week
  <wallyworld>  http://reviews.vapour.ws/r/321
  <wallyworld> if you want to look
  <menn0> looking now
  <wallyworld> ty
  <bodie_> there we go.  cleared up with a little more careful updating of tools
  <menn0> wallyworld: why is it important to not write metadata if it hasn't changed?
  <wallyworld> menn0: because it otherwise updates the "updated" attribute and caused issues with the scripts used to publish the metadata
  <menn0> wallyworld: got it. the reason for the change wasn't clear to me.
  <wallyworld> np, sorry
  <wallyworld> it came up in a conversation over the weekend
  <wallyworld> the bug didn't mention it
  <alexisb> wallyworld, ping
  <wallyworld> hey
  <alexisb> hey there wallyworld how goes it?
  <wallyworld> alright consider it's monday
  <alexisb> yay for monday!
  <wallyworld> or sunday evening for sime
  <wallyworld> some
  <alexisb> I sent mail regarding an lxc issue, looks like that may be a hot item (or going to get hot)
  <alexisb> I will find out more on monday
  <alexisb> but just giving you an heads up
  <wallyworld> sure, ok. if it's the one i'm thinking of, i asked dimiter about it lasy week, and he's doing a temporary fi
  <wallyworld> fix
  <wallyworld> there's a proper fix as part of the ongoing network stuff coming up
  <wallyworld> but right now it's all a bit broken
  <alexisb> yeah there seems to be 3 related bugs (outlined in the email) but I didnt go look at all of them in detail
  <wallyworld> it depends on individual machine network setups though as to whether it's broken or not
  <menn0> wallyworld: review done
  <wallyworld> i'll read the email in more detail and see what's up with it
  <wallyworld> thanks menn0
  <alexisb> thanks wallyworld
  <wallyworld> np. now go and enjoy the rest of your sunday
  <wallyworld> and hopefully we'll have a 1.21 alpha3 elease real soon now
  <bodie_> hm
  <bodie_> I'm trying to add some actions-related testing to charm but it looks like charm master breaks juju master
  <bodie_> so I'm not sure what to base it on -- the v4 branch appears to have the breaking change as well
  <bodie_> charm commit 190b3a5d removes charm/testing/repo/quantal/metered-custom, which is tested against in juju CharmTestHelperSuite.TestTestingCharm
  <bodie_> the charm revision listed in godeps is fine
  <menn0> waigani: http://reviews.vapour.ws/r/322/
  <davecheney> dear lazy-irc, is the build blocked ?
  <menn0> davecheney: yep
  <davecheney> well shit
  <menn0> davecheney: i think Ian is about to merge the fix though
  <waigani> menn0: done
  <menn0> waigani: cheers
  <menn0> waigani: I like your suggestion. Changing now.
  <waigani_> menn0: ah great
  <menn0> waigani_: PTAL http://reviews.vapour.ws/r/322/
  <davecheney> https://github.com/juju/charm/issues/69
  <wallyworld> davecheney: one part of that bug was a map ordering issue, and has been fixed. the other failures are due to compiler issues as per the bug comments
  <davecheney> wallyworld: my mistake
  <davecheney> i missed those in the updates
  <wallyworld> davecheney: np, it's a long conversation
  <davecheney> wallyworld: be aware
  <davecheney> gccgo on ppc is completely fucked up
  <davecheney> we don't appear to be able to push the updated revision out properly
  <wallyworld> yeah?
  <wallyworld> :-(
  <davecheney> this issue is 8 months old now
  <davecheney> it's nearly a toddler
  <wallyworld> we need to escalate this then
  <davecheney> ok, not sure who to talk to about this
  <wallyworld> alexis
  <davecheney> i haven't been involved in this since vegas
  <wallyworld> give her the info and she can prod the right people
  <wallyworld> i think you are saying the issues have been fixed but we don't have the latest version in the archives?
  <davecheney> if only it were that simple
  <davecheney> wallyworld: forwarding you mail
  <wallyworld> ty
  <wallyworld> will send it on
  <bodie_> shall I open a PR against charm v4 so that it is no longer a breaking change but includes the misnamed testing repo, which tests in juju core were written against?
  <bodie_> https://github.com/juju/charm/pull/68
  <bodie_> or shall we migrate to charm v5, since I also have some changes I would like to make which deprecate some tested bits in charm v4
  <waigani> menn0: done, looks good
  <davecheney> wallyworld: ok, done
  <davecheney> importantly, all version of trusty, even 14.04.01 are known to be BROKEN
  <wallyworld> davecheney: thanks, i'l follow up
  <davecheney> you must enabled trusty-updates **BEFORE** installing juju
  <davecheney> otherwise -> FAIL
  <wallyworld> ouch
  <rick_h_> bodie_: what's up?
  <rick_h_> bodie_: I'd suggest working with roger and uros  on charm updates. The v4 is in the list for core deps. Not sure what 'breaking change' is?
  #juju-dev 2015-10-26
  <cherylj> wallyworld_: ping?
  <wallyworld_> hey
  <wallyworld_> wot you doing working on a sunday
  <cherylj> heh
  <cherylj> being concerned about lxc containers on wily
  <wallyworld_> which bits?
  <rick_h__> cherylj: how does that go?
  <mup> Bug #1509747 changed: Intermittent lxc failures on wily <bug-squad> <lxc> <wily> <juju-core:Invalid> <systemd (Ubuntu):New> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 opened: Intermittent lxc failures on wily <bug-squad> <lxc> <wily> <juju-core:Invalid> <systemd (Ubuntu):New> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 changed: Intermittent lxc failures on wily <bug-squad> <lxc> <wily> <juju-core:Invalid> <systemd (Ubuntu):New> <https://launchpad.net/bugs/1509747>
  <wallyworld> axw: that the client might see tools the bootstrap agent can't see is the reason for the patch; the approach allows the client to be opinionated about what it asks for based on user input, but provides a safety net that the agent can reject that if the tools are not available server side. if the bootstrap agent were to use a flag to determne if newer tools were required, it would essentially be the same code as written now anyway
  <wallyworld> because the bootstrap agent would search for tools and set agent-version accordingly
  <axw> wallyworld: the agent says "bootstrap, then upgrade to to the latest patch version". "latest patch version" may means different things to the CLI and the bootstrap agent. why fail to upgrade to .1 just because .2 isn't there?
  <mup> Bug #1509353 changed: local provider service cannot be enabled after disabling <local-provider> <systemd> <juju-core:Won't Fix> <https://launchpad.net/bugs/1509353>
  <wallyworld> to match user expectation
  <wallyworld> if user can see latest version is .2, and server cn't, i think we'd rather log that
  <wallyworld> and allow user to upgrade as required after
  <wallyworld> i dislike the whole auto upgrade thing anyway
  <wallyworld> for 2.0, we should flip that
  <wallyworld> i guess it's a matter of opinion which way works best
  <axw> wallyworld: what's meant to happen is that it upgrades to the latest available patch version. we're changing the meaning of "latest available" from what the client sees to what the bootstrap agent sees
  <wallyworld> i can see both sides
  <axw> wallyworld: if we're doing away with it
  <axw> wallyworld: then fine, I can live with this
  <wallyworld> it won't be changed till 2.0, so if there's a real problem with the current approach, i'd need to change that now
  <axw> wallyworld: it's not a problem, I just don't agree with the approach. let's fix the actual problem first though - shipit
  <wallyworld> i think your view is good
  <wallyworld> i did oscillate between the 2 approaches
  <axw> wallyworld: my way means a lot more work for not a lot more gain
  <wallyworld> that was part of my thnkong
  <wallyworld> ok, i'll ship it, it does solve the immediate problem
  <davecheney> ... Panic: local error: bad record MAC (PC=0x414706)
  <davecheney> every
  <davecheney> god
  <davecheney> damn
  <davecheney> day
  <axw> wallyworld: just responded to your email on simplestreams. let me know ify ou think that's a reasonable limitation to live with, and I'll make it so
  <wallyworld> ok
  <wallyworld> looking
  <axw> wallyworld: I have coded the image ID parsing, but it would be good not to have to
  <axw> wallyworld: actually I could do something hackish with the stream, and exclude the SKUs with "DAILY" in the name if stream=release, use those ones only if stream=daily
  <wallyworld> axw: the structured metadata stuff does support streams
  <wallyworld> so we don't lose anything there
  <axw> wallyworld: azure doesn't have a concept of image streams
  <axw> wallyworld: (but I can fake it by parsing the SKU name)
  <wallyworld> i think that sound s ok
  <wallyworld> axw: so i think for option a), the suggestion is to modify azure's findImage() to not query juju's structured metadata collection, but query the provider apis instead?
  <axw> wallyworld: right
  <wallyworld> axw: my concern is that perhaps the cpc folks want to control the images use
  <axw> wallyworld: query it with a fixed publisher of "Canonical", and offering of "UbuntuServer" (something similar for Windows)
  <wallyworld> dvia simplestreams
  <wallyworld> or are we saying that they would do that via what they publish to azure
  <axw> wallyworld: pretty sure this information is based on images that are provided by CPC, but we'll see what Ben says
  <wallyworld> ok
  <axw> fwereade: I see you're looking at http://reviews.vapour.ws/r/2958/, can you please also take another look at http://reviews.vapour.ws/r/2935/ later?
  <rogpeppe> a small fix to make the recent changes to fslock.NewLock backwardly compatible: https://github.com/juju/utils/pull/166
  <fwereade> rogpeppe, dammit, I sort of see the backward compatibility concerns, but... (1) I really do not want to encourage people to use wall clocks and (2) anyone using a package without some sort of pinning is asking for trouble anyway
  <rogpeppe> fwereade: in a call
  <fwereade> rogpeppe, np
  <perrito666> wallyworld: anastasiamac axw brt
  <wallyworld> sure
  <rogpeppe> fwereade: we really should maintain backward compat when possible.
  <rogpeppe> fwereade: also, we *are* using version pinning but this bit us.
  <fwereade> rogpeppe, ok, I imagine you updated for something else and this was a surprise?
  <rogpeppe> fwereade: because pinning to the latest version of juju (which hasn't yet been updated to use latest utils changes) but we need a more recent change in another utils package
  <rogpeppe> fwereade: yes
  <rogpeppe> fwereade: i don't want to predicate moving our stuff forward on landing the change in juju-core
  <fwereade> rogpeppe, I must say I'd still prefer to go for a doc note saying "if you must, you can use the NewClockWithGlobalTime func which uses the old interface"?
  <rogpeppe> fwereade: also, having the default function use wall clock time seems reasonable to me - it's idiomatic and obvious
  <fwereade> rogpeppe, right, but it's also an implicit dependency on global time which has bitten us more times than I care to count
  <fwereade> rogpeppe, NewLockWithClock implies that explicit dependencies are the special case
  <fwereade> rogpeppe, and that's subtly corrosive imo
  <rogpeppe> fwereade: it's reasonable to make new functions that do this
  <rogpeppe> fwereade: but let's *please* preserve backward compatibility rather than introducing gratuitous breakage of old code
  <rogpeppe> fwereade: when reasonablew
  <rogpeppe> fwereade: particularly when in non-gopkg packages
  <rogpeppe> fwereade: juju-core is not the only consumer of the packages under github.com/juju
  <fwereade> rogpeppe,  right, but that's not gratuitous breakage; it's addressing a real problem, that implicit dependencies hamstring our ability to cleanly use and test code
  <rogpeppe> fwereade: you're going to have to change the code anyway
  <rogpeppe> fwereade: it's not a problem currently for third party packages currently
  <rogpeppe> fwereade: but you're making it one
  <rogpeppe> fwereade: doing a search for fslock.NewLock is easy to do
  <fwereade> rogpeppe, I definitely think it should be *easy* to change -- having a func with the old interface, and a pointer to same, seems reasonable
  <rogpeppe> fwereade: currently we *cannot* move forward with this backwardly incompatible change
  <rogpeppe> fwereade: please can we agree to fix it
  <fwereade> rogpeppe, so, is the problem that there is no func with the old signature; or that a global search-and--replace is an excessive burden?
  <rogpeppe> fwereade: the problem is that there is no func with the old signature
  <rogpeppe> fwereade: so there is no version of juju-core we can pin to that both provides us with the new functionality we need in utils and compiles
  <fwereade> rogpeppe, I would happily support the introduction of a func with that signature, but I can't make peace with the obvious default constructor being the one with implicit dependencies
  <rogpeppe> fwereade: please, for the sake of backward compatibilty. you can mark it as "deprecated" if you really want.
  <mgz_> because we have godeps, I'm fine with api breakage on stuff without versioning for sanity
  <mgz_> we're not breaking the ability to build old juju versions
  <mgz_> and the code wasn't actually around long enough for anyone else to use it
  <rogpeppe> mgz_: as it happens, this *is* breaking our ability to compile stuff
  <mgz_> rogpeppe: you have packages not using godeps? I thought all the stuff you had used it?
  <rogpeppe> mgz_: no, we use godeps
  <mgz_> so, it's fine, you pin at the older version till you update to use the new api
  <rogpeppe> mgz_: but we've landed a new feature in a utils package, but we can't use it because the latest version of utils is incompatible with all versions of juju-core
  <mgz_> sure, that's expected
  <rogpeppe> mgz_: so you're saying we can't move forward with our stuff until all the clock dependency stuff is implemented in juju-core?
  <mgz_> function of how go works and the fact we haven't put enough thought into the apis we share
  <rogpeppe> mgz_: there's an easy solution: preserve a compatible API
  <mgz_> rogpeppe: nope, but you can change to use a different constructor
  <rogpeppe> mgz_: which is trivial in this case
  <fwereade> rogpeppe, I agree we should have kept a convenience func around, but I don't think it's sensible to keep the API pinned to the first version we thought of
  <mgz_> that omits the clock
  <rogpeppe> mgz_: no, i can't
  <rogpeppe> mgz_: fwereade thinks there should be no such constructor
  <fwereade> rogpeppe, is introducing a new constructor not trivial in this case? it seems that's what the CL does
  <mgz_> he can be pursuaded
  <mgz_> provided it's not the default one
  <fwereade> rogpeppe, no, I said that a convenience constructor is fine; nbut that the NewLock name shoudl be reserved for the right way to use the package
  <rogpeppe> fwereade: the CL keeps the old constructor as is
  <fwereade> rogpeppe, yes, that is my problem with iit
  <rogpeppe> fwereade: it preserves backwards compatibility
  <fwereade> rogpeppe, with a bad API, yes
  <rogpeppe> fwereade: arguably yes, but backward compatibility trumps bad API in this case, i think
  <rogpeppe> fwereade: it's just a name
  <fwereade> rogpeppe, names are important
  <rogpeppe> fwereade: yes, i agree
  <mgz_> rogpeppe: make the change NewLock (has clock) and NewLockGlobalTime or wtv and it's fine
  <rogpeppe> mgz_: that's not possible for us
  <rogpeppe> mgz_: because we're not calling NewLock
  <rogpeppe> mgz_: it's juju-core that calls NewLock
  <mgz_> it is, you just change the name when you bump the dep
  <rogpeppe> mgz_: we can't
  <mgz_> meph, that is a little fun
  <rogpeppe> mgz_: because we're dependent on juju-core
  <rogpeppe> mgz_: juju-core needs to be changed to use the new function
  <rogpeppe> mgz_: but i suspect that's not trivial
  <mgz_> and you can't pin to the right juju-core version because you need something there?
  <rogpeppe> mgz_: there is no right juju-core version
  <mgz_> rogpeppe: which juju-core are you basing from?
  <mgz_> master?
  <rogpeppe> mgz_: latest master
  <mgz_> okay, so we can still fix this
  <mgz_> utils gains a constructor
  <mgz_> master of juju/juju picks that up and renames callers
  <mgz_> you then pin at that landed master version
  <rogpeppe> mgz_: i guess we could land a change in juju master to just globally substiture NewTimer to add clock.WallClock as an argument.
  <rogpeppe> mgz_: but that's avoiding the whole reason for doing this
  <mgz_> this is true.
  <fwereade> rogpeppe, mgz, at least it's moving the config one step out towards the edges, which is worthwhile in itself I think? small step but positive?
  <rogpeppe> fwereade: tbh i'm not entirely convinced by passing the clock as an argument to everything anyway, because isn't it really something that's fundamentally global?
  <rogpeppe> fwereade: i see an argument for being able to fake it
  <mgz_> I feel the monad rising up
  <fwereade> rogpeppe, it's really not -- it may be convenient in one situation, but the pain of depending on wall-clock time in tests is fricking devastating in aggregate
  <rogpeppe> fwereade: but i'm not sure about the capability to have several kinds of clock extant in the same program
  <fwereade> mgz_, lol
  <fwereade> rogpeppe, indeed: ideally, you'll have a single clock configured at the top level and delivered wherever it needs to go
  <rogpeppe> fwereade: isn't that just because it's not possible to fake time.Time in general?
  <rogpeppe> fwereade: if everyone used juju/clock instead (which provided a fakable time), wouldn't that be good enough?
  <fwereade> rogpeppe, well, (1), I do have use cases where it's good to know what happens when clocks diverge; and (2) if the clock can't be faked without poking into globals, it's not fakeable enough imo
  <fwereade> rogpeppe, I've ended up deciding that the urge to patch is basically always wrong -- it's just working around having made implicit a dependency that ought to be explicit
  <fwereade> rogpeppe, and that leads towards heavy constructors, which aren't necessarily so pretty
  <mgz_> well, the issue is more that time can mean several things and code doesn't have a good mechanism for saying what it actually cares about
  <rogpeppe> fwereade: for most things, i tend to agree, but clock does seem pretty global to me. although testing clock divergence is interesting.
  <fwereade> rogpeppe, but are still better than the alternative -- in which you have just as many dependencies but they're hidden, and waiting to trip up the unwary maintainer
  <rogpeppe> fwereade: i wish that the mechanism used by the Go playground was generally accessible
  <fwereade> rogpeppe, that'd mitigate, yes, but I'm on a pretty strong all-global-mutable-state-is-bad kick at the moment
  <fwereade> rogpeppe, maybe it'd even mitigate enough not to matter -- if I could create a complete sandboxed runtime I wouldn't be so worried about the globals inside it
  <rogpeppe> fwereade: in fact, you probably wouldn't want that, as it would preclude actual real timeouts in tests
  <fwereade> rogpeppe, ha, good point
  <fwereade> well
  <rogpeppe> fwereade: but the idea of a sandboxed runtime started for each test is an interesting one
  <fwereade> rogpeppe, yeah, so long as you had the watchdog timer outside it I think it cold work quite nicely
  <rogpeppe> fwereade: well, the watchdog timer *is* outside it
  <fwereade> rogpeppe, but you'd want access to that for the bits that really do involve timeouts
  <fwereade> rogpeppe, so, yeah, it's fiddly
  <rogpeppe> fwereade: can we land my change (to preserve backward compatibility) so that we can at least move forward without needing to update juju-core, until juju-core is at least ready to use the new API, at which point the names can be changed?
  <rogpeppe> fwereade: because we are not developing juju-core and really don't want our moving forward to be predicated on changes there
  <rogpeppe> fwereade: think of it like this: the API has been "wrong" for >2 years now. another week or so surely isn't too bad.
  <fwereade> dooferlad, do you have the bandwidth to follow up on that; and, rogpeppe, to absorb that change when it does come?
  <rogpeppe> fwereade: we don't have a problem with absorbing the change 'cos we're not actually using the fslock API
  <rogpeppe> fwereade: (only indirectly)
  <rogpeppe> fwereade: going for lunch now
  <fwereade> rogpeppe, ok, cool -- and dooferlad, does that work for you? I presume you had a actually-use-in-juju-core change coming up anyway?
  <dooferlad> fwereade: yes, once my changes land I will be happy to fix the integration with Juju core
  <fwereade> dooferlad, rogpeppe: awesome, ty both
  <dooferlad> fwereade, rogpeppe: I am in part stuck until http://reviews.vapour.ws/r/2908/ lands, which is related, so if wise heads could take a look that would be delightful. Once that lands in utils I can change the function sigs in juju-core to use an injected time and we will all be happy.
  <rogpeppe> dooferlad: tbh i don't really know why we're using file-creation-based locking anyway when we could use flock
  <rogpeppe> fwereade: are we planning on getting rid of the global logger instances too?
  <dooferlad> rogpeppe: we started with file based locking, but if a lock wasn't released before the process holding it went away we got in a pickle. This happend a lot when rebooting machines during tests. This is attempting to have a way of automatically breaking stale locks. Flock is linux only.
  <fwereade> rogpeppe, I would like to, yeah, but it's not so near the top of my list
  <fwereade> rogpeppe, they're yucky but not generally actively painful IME
  <fwereade> rogpeppe, or at least not so often
  <rogpeppe> dooferlad: i'm sure there's some equivalent on all the platforms we use
  <fwereade> rogpeppe, whereas implicit dependencies on wall-clock time are an ongoing thorn in our side
  <mup> Bug #1506666 changed: juju bootstrap fails in virtualbox <local-provider> <network> <virtualbox> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1506666>
  <rogpeppe> dooferlad: unix has had flock or fcntl(F_SETLK) for a long time
  <rogpeppe> dooferlad: and windows locks files by default anyway when they're open
  <rogpeppe> dooferlad: it would certainly be more efficient to use the OS file lock primitives
  <dooferlad> rogpeppe: I don't know about the Windows side of things. I was trying to modify what we are already using to be more useful. I dislike using file locks, but that is what we currently have.
  <rogpeppe> dooferlad: yeah, sorry, it was just a random aside. i don't like it but that is indeed what we've got
  <dooferlad> rogpeppe: no problem. Sore topic.
  <mup> Bug #1506666 opened: juju bootstrap fails in virtualbox <local-provider> <network> <virtualbox> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1506666>
  <mup> Bug #1506666 changed: juju bootstrap fails in virtualbox <local-provider> <network> <virtualbox> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1506666>
  <mup> Bug #1506666 opened: juju bootstrap fails in virtualbox <local-provider> <network> <virtualbox> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1506666>
  <mup> Bug #1506666 changed: juju bootstrap fails in virtualbox <local-provider> <network> <virtualbox> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1506666>
  <katco`> wwitzel3: ericsnow: natefinch: standup?
  <natefinch> katco`: we're there
   * katco` mutters at computer
  <katco`> fwereade: ping
  <fwereade> katco, pong
  <cherylj> frobware: Can you make sure that someone picks up bug 1483879 as part of sapphire's bug squad work this iteration?
  <mup> Bug #1483879: MAAS provider: terminate-machine --force or destroy-environment don't DHCP release container IPs <bug-squad> <destroy-machine> <landscape> <maas-provider> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1483879>
  <katco> fwereade: can you hop in moonstone rq? https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1
  <frobware> cherylj, ack. I would like to sync up today if you have time.
  <cherylj> frobware: I'm free for the next hour
  <frobware> cherylj, OK can you jump on the following https://plus.google.com/hangouts/_/canonical.com/juju-sapphire
  <cherylj> frobware: yes, let me find my headset.
  <cherylj> frobware: I'm there
  <rogpeppe> anyone know why we're using github.com/gabriel-samfira/sys rather than the repo it's cloned from (golang.org/x/sys) ?
  <ericsnow> rogpeppe: Go 1.2 compatibility
  <rogpeppe> ericsnow: oh, marvellous
  <ericsnow> rogpeppe: yep :(
  <rogpeppe> ericsnow: couldn't we at least make our own fork rather than copying some random guy's ?
  <ericsnow> rogpeppe: gsamfira isn't exactly a "random guy" (he's one of the cloudbase folks)
  <ericsnow> rogpeppe: but I doubt anyone would object to a fork under the juju org
  <rogpeppe> ericsnow: ok, i see
  <mup> Bug #1497301 changed: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Invalid> <https://launchpad.net/bugs/1497301>
  <mup> Bug #1510129 opened: error: cannot re-bootstrap environment: cannot bootstrap new instance: Juju cannot bootstrap because no tools are available for your environment. <backup-restore> <ci> <intermittent-failure> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1510129>
  <mup> Bug #1510132 opened: redefinition of ânoMethods$MarshalYAML$hashâ <ci> <testing> <juju-core:New> <juju-core model-migration:Triaged> <https://launchpad.net/bugs/1510132>
  <mup> Bug #1510133 opened: juju's ConnSuite should not start a UnitAssigner worker <tech-debt> <juju-core:New> <https://launchpad.net/bugs/1510133>
  <rogpeppe> mgz_: looks like release testing hasn't run on master for a week now. it's making me nervous, particularly with the big chicago-cubs push recently. any chance we could prioritise it?
  <rogpeppe> mgz_: also, should feature branches be deleted after they've been merged?
  <mgz_> rogpeppe: yeah, we've been futzing with priorities and trying to get 1.24/1.25 done last week meant master has been rather neglected
  <mgz_> feature branches can be deleted after merge, I've not bothered so far as github has good interface for showing what's in and what's not yet anyway
  <perrito666> bbl outside lunch
  <fwereade> tasdomas, have you run the meter-status tests with -race? I think the code field, at least, is problematic
  <dooferlad> dimitern: hangout?
  <dimitern> dooferlad, omw
  <ericsnow> wwitzel3: ready
  <wwitzel3> ericsnow: ok, just finishing lunch
  <ericsnow> wwitzel3: np
  <natefinch> fwereade: if I'm just copying  uniter's SetAgentStatus code... why am I not just calling the uniter's version?
  <fwereade> natefinch, the functionality is the same but the auth certainly shouldn't be
  <fwereade> natefinch, different facades responsible for different auth
  <natefinch> fwereade: hmm true....
  <fwereade> natefinch, but calling through to the same underlying implementation because they're doing the same thing
  <natefinch> fwereade: ok.  I can live with that. There's only a small bit of logic in the unitAgentFinder that is shared.... got an idea of where I could move that so both implementations could share it?
  <fwereade> natefinch, I guess a subpackage of apiserver/common, ideally, but that'd really want to drag the status code in as well... so, I guess, in common/setstatus.go, unless you have time/energy to extract a package
  <natefinch> fwereade: I'll put it in setstatus.go.... time and energy for this bug are long gone, unfortunately.
  <fwereade> natefinch, sgtm
  <natefinch> fwereade: is there a non-terrible way to figure out if an error returned by the API is a notfound error?  Or if not, has someone already written the terrible code that I can reuse?
  <fwereade> look for IsCodeNotFound
  <fwereade> if you were a wonderful person you'd translate that back to a NotFoundError on the way out of the api client code but nobody else ever does that ;p
  <fwereade> that was not expressed appropriately
  <fwereade> I am entirely aware you are a wonderful person already ;p
  <natefinch> fwereade: haha, perfectly appropriately... .and I had been thinking of doing tha
  <natefinch> lol
   * fwereade eod, might pass by later
  <mgz_> how do I find the specs and shizzle for this cycle?
  <mgz_> ah, alexisb linked them all in her summary doc, that'll do
  <natefinch> man our API comments are terrible
  <natefinch> SetEntityStatus take a Status - which is a string, an Info - which is a string, and Data - which is a map[string]interface.... and no description of what each of those things are actually ofr
  <natefinch> for
  <natefinch> katco: got the unit status setting code working. Needs tests, though, which will take a bit.
  <katco> natefinch: yay :) get those tests written! :)
  <thumper> mramm: hey, we are going to need to change our call times :)
  <thumper> mramm: since nz is now in summer time
  <frobware> cherylj, ping - 1496972, should that not be for 1.24 as well?
  <cherylj> frobware: I targeted it to 1.25.1 as we're hoping to not do 1.24.8.  But yeah, I should check with the guys that brought it up in the cross team call/
  <cherylj> make sure they're not committed to 1.24 or anything
  <thumper> cherylj: is http://reviews.vapour.ws/r/2998/diff/# a forward port or did this not land yet?
  <cherylj> thumper: the 1.26 revision addressing fwereade's comments had not landed yet.  This is it.
  <thumper> cherylj: what's the difference?
  <cherylj> thumper: the biggest difference is that the retry strategy is generated by the provisioner, rather than the lxc code
  <thumper> k
  <cherylj> thumper: the original review is here:  http://reviews.vapour.ws/r/2761/
  <cherylj> if you want it for reference
  <mattyw> davechen1y, are you around?
  <perrito666> wallyworld_: axw anastasiamac I might be a bit late, suddenly I have no sound from the browser
  <wallyworld_> \o/
  <perrito666> software makes me really miserable
  <anastasiamac> :(
  <anastasiamac> perrito666: i *think* i know what u mean :P
  <perrito666> ready, found an alternate solution
  <axw> thumper: do machine provisioners run concurrently for each hosted env?
  <axw> thumper: and firewaller
  #juju-dev 2015-10-27
  <davechen1y> lucky(~/src/github.com/juju/juju) % gb test
  <davechen1y> FATAL command "test" failed: failed to resolve import path "github.com/juju/juju/api": cannot find package "golang.org/x/net/http2/hpack" in any of:
  <davechen1y> ^ who added a dependency on the new http2 stuff ?
  <thumper> axw: yes
  <thumper> I believe so
   * thumper shrugs
  <thumper> davechen1y: the shrug was for the http2 stuff
  <axw> davechen1y: "go list" shows no dependency on http2
  <thumper> perhaps just a test dep?
  <davechen1y> could be a test dep
  <davechen1y> still digging
  <davechen1y> ok
  <davechen1y> ignore this
  <davechen1y> short version: it's a go 1.6 thing
  <davechen1y> ignore it
  <thumper> :)
  <mup> Bug #1510333 opened: worker/uniter: converge on a single approach for updating goal state <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1510333>
  <mup> Bug #1510333 changed: worker/uniter: converge on a single approach for updating goal state <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1510333>
  <mup> Bug #1510333 opened: worker/uniter: converge on a single approach for updating goal state <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1510333>
  <davechen1y> http://reviews.vapour.ws/r/3000/
  <davechen1y> anyone ?
  <thumper> davechen1y: this is from where?
  <thumper> was it juju/juju/ssh?
  <thumper> I was going to suggest we just have juju/ssh rather than juju/utils/ssh
  <thumper> but then I saw the other deps on the utils package
  <thumper> and thought "meh"
  <davechen1y> thumper: the only thing worse than a utils packge is a utils repo
  <davechen1y> thumper: and the only thing worse than a utils repo is having several
  <davechen1y> one evil at a time
  <thumper> :)
  <thumper> k
  <thumper> axw: if you are happy with https://github.com/juju/retry/pull/1 I'll merge it :)
  <axw> thumper: sorry, I keep getting distracted
  <thumper> axw: I was hoping fwereade would have looked, but I'm prepared to land it now
  <axw> thumper: lemme take another quick look
  <thumper> cheers
  <sinzui> thumper: I have added github.com/juju/retry. We wont know it works until there is a branch to merge. Since there isn't a branch there now, I don't know which packages deps it imports.
  <thumper> sinzui: https://github.com/juju/retry/pull/1
  <sinzui> thank youn thumper . I will quickly add the deps I see
  <thumper> cheers
  <axw> thumper: reviewed
  <thumper> axw: ta
  <thumper> sinzui: I'll try merging now
  <thumper> sinzui: is it ready?
  <sinzui> thumper: yep, I am watching to procs now
  <thumper> sinzui: who handles which repos go to reviewboard?
  <sinzui> thumper: I don't know the config. I am sure ericsnow knows.
  <thumper> \o/ landed
  <sinzui> thumper: I see real go test was really run too
  <thumper> as opposed to what?
  <sinzui> thumper: the merge job just sending a success without actually merging the branch and running the tests
  <thumper> :)
  <thumper> I must admit, I do find it pleasing to see more tests than code :)
  <sinzui> :))
  <davechen1y>                         logger.Debugf("keeping unrecognised existing ssh key %q: %v", key, err)
  <davechen1y> ERRORS LOGGED AT DEBUGGG!
  <anastasiamac> "some error are more equal than others"? :P
  <davechen1y> "some errors are more like guidelines"
  <anastasiamac> :P
  <davechen1y> thumper: FFS
  <thumper> davechen1y: hey
  <davechen1y> tried to apply your suggestions to use the errors package, all the fucking tests expect an exact string
  <davechen1y> tempted to tell you to stick your recommendation
  <thumper> davechen1y: that's fine for now
  <davechen1y> cannot even upgrade them to errors.Annotate
  <thumper> we can "fix" it later
  <davechen1y> heh, fix
  <natefinch> Man, can't tell you how much I hate all our gc.ErrorMatches tests.  What a nightmare.
  <davechen1y> i fixed it, with a hammer
  <davechen1y> axw: re ExecuteCommandOnMachine
  <davechen1y> what if it went to juju/utils/exec
  <axw> davechen1y: SGTM
  <axw> davechen1y: oh hum, but htat means utils/exec depends on ssh?
  <axw> davechen1y: if it's a PITA, don't worry. I'd just like for hte ssh package to be a bit leaner
  <davechen1y> i think i can fix that
  <davechen1y> type Client interface {
  <davechen1y> type Client interface {
  <davechen1y>  // Client is an interface for SSH clients to implement
  <davechen1y> type Client interface {
  <davechen1y> ^ honestly, it should just say "client is an interface"
  <davechen1y> for all the good that comment does :)
  <davechen1y> axw, hmm
  <davechen1y> this is quite tricky
  <davechen1y> execonmachien requires something that mathces Command(...) *ssh.Cmd
  <davechen1y> dunno if it can be teased apart easily
  <axw> davechen1y: yeah, just drop it. maybe it can be refactored later, not worth blocking the move over
  <davechen1y> mkay
  <davechen1y> i'm going to loose my shit this week http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils/47/console
  <davechen1y> ^ anyone want to guess why this broke ?
  <axw> davechen1y: because it's using go 1.2 I'm guessing
  <axw> davechen1y: (and we're trying to compile it with that when we should just be downloading it and then checking out the older rev)
  <davechen1y> axw: i think there might be some sort of conditional compilation going on there
  <davechen1y> we pin the revision we want in juju/juju
  <davechen1y> but trying to compile juju/utils, it floats freely
  <axw> davechen1y: ah hm, cos we don't have a dependencies.tsv there ? :/
  <davechen1y> i dunno if adding one automatically help
  <axw> bit of an oversight
   * axw shrugs
  <davechen1y> here's what I propose
  <davechen1y> 1. i go an have some lunch
  <davechen1y> 2. it magically fixes itselp
  <davechen1y> sgtm ?
  <axw> :)
  <axw> sounds good. I'll do the same
  <natefinch> 1. switch to gb
  <natefinch> 2. ???
  <natefinch> 3. profit
  <dimitern> frobware, hey, I'll be a couple of minutes late for our 1:!
  <dimitern> 1:1
  <frobware> dimitern, ok
  <dimitern> jam, fwereade, standup?
  <axw> fwereade: my latest rev on http://reviews.vapour.ws/r/2935/ is quite substantial, I'd appreciate a look at the last commit when you can
  <axw> fwereade: there's an unrelated fix mised in there, but it's also up for review as http://reviews.vapour.ws/r/3002/
  <axw> s/mised/mixed/
  <fwereade> axw, thanks
  <fwereade> axw, both reviewed
  <fwereade> https://github.com/juju/juju/wiki/howto:-implement-effective-config-structs
  <fwereade> ...if anyone feels like some light reading or arguing
  <wallyworld> fwereade: did you have time for a chat?
  <fwereade> wallyworld, just about to have lunch -- what time is it for you?
  <wallyworld> 23:00, i don't mind waiting
  <wallyworld> just ping when you're free
  <mup> Bug #1510533 opened: destroy-environment panics <juju-core:New> <https://launchpad.net/bugs/1510533>
  <mup> Bug #1510533 changed: destroy-environment panics <juju-core:New> <https://launchpad.net/bugs/1510533>
  <mup> Bug #1510533 opened: destroy-environment panics <juju-core:New> <https://launchpad.net/bugs/1510533>
  <katco> ericsnow: wwitzel3: natefinch: please do the standup without me. cats shattered lots of glass on our kitchen floor and my dog ate some
  <perrito666> katco: ate some cat or glass?
  <wwitzel3> katco: ok
  <frobware> dimitern, ping
  <abentley> sinzui: chat?
  <sinzui> abentley: sure
  <mup> Bug #1403689 changed: Server should handle tools of unknown or unsupported series <upgrade-juju> <upload-tools> <juju-core:Fix Released by wallyworld> <juju-core 1.24:Triaged> <juju-core 1.25:Fix Released by wallyworld> <https://launchpad.net/bugs/1403689>
  <mup> Bug #1510619 opened: Wily: add machine fails using kvm and lxcbr0 <add-machine> <cloud-installer> <kvm> <network> <juju-core:New> <https://launchpad.net/bugs/1510619>
  <voidspace> mgz_: ping
  <mgz_> voidspace: wotcha
  <voidspace> mgz_: hey
  <voidspace> mgz_: so we have plans to get rid of the need for ignore-addresses on master
  <mgz_> ace
  <voidspace> mgz_: but we're going to land the fix and test it with those needing it *before* we actually remove ignore-addresses
  <voidspace> (I work on the proposed fix next)
  <voidspace> mgz_: so we'll land the change to ignore-addresses on containers and master first
  <voidspace> mgz_: because currently containers are broken when you use ignore-machine-addresses
  <voidspace> mgz_: and next a question...
  <mgz_> voidspace: sounds fair
  <voidspace> mgz_: when I attempt to do an upgrade from 120/1.22 to 1.24 but using --upload-tools
  <voidspace> mgz_: it fails with "unknown series wily"
  <voidspace> mgz_: dimitern thought you might have encountered this, or similar, when testing upgrades with QA and know a workaround
  <mgz_> voidspace: yeah, we've had a lot of that in testing,
  <voidspace> mgz_: did you solve it?
  <mgz_> not sure if I know a specific workaround when using --upload-tools for upgrade though
  <voidspace> :-/
  <voidspace> I'll ask aaron
  <voidspace> abentley: ping
  <mgz_> bug  1403689 is part of it
  <voidspace> bug 1403689
  <mup> Bug #1403689: Server should handle tools of unknown or unsupported series <upgrade-juju> <upload-tools> <juju-core:Fix Released by wallyworld> <juju-core 1.24:Triaged> <juju-core 1.25:Fix Released by wallyworld> <https://launchpad.net/bugs/1403689>
  <voidspace> mgz_: ah, I wondered if there was a bug for it
  <voidspace> mgz_: thanks
  <voidspace> unassigned for 1.24 currently
  <mgz_> what CI does is actually use streams, which we then control
  <voidspace> right
  <voidspace> I thought as much, and we can't suggest that to IS I don't think
  <voidspace> we want them to test the new binaries
  <mgz_> but yeah, bug suggests --upload-tools to 1.25 would work?
  <voidspace> 1.25 isn't released yet
  <mgz_> upgrades with --upload-tools are pretty dodgy anyway
  <voidspace> needed when you have new binaries for a user to test
  <sinzui> voidspace: your experience mirror's abentley's frustration over the past 48 hours
  <voidspace> sinzui: heh
  <voidspace> ouch
  <sinzui> voidspace: the stupid client makes assumptions about its host to decide what to upload to the series and since 1.20.x doesn't check distro-info-data correctly, it will die because of vivid or wily
  <voidspace> right :-/
  <voidspace> sinzui: it makes testing upgrades / fixes for upgrade bugs impossible
  <voidspace> hopefully the fix is correct, so we never have the issue again
  <sinzui> voidspace: abentley  might want to bu you beer
  <voidspace> sinzui: hah
  <voidspace> sinzui: no, I meant the fix that wallyworld has allegedly committed to 1.25 and master
  <voidspace> not that I've done it
  <voidspace> he hasn't committed it to 1.24 though, which is where we need it
  <sinzui> voidspace: CI tests 1.20 upgrades to the candidate we test. abentley is updating the test to use upload-tools as needed. so your change is exactly what we are looking to test
  <voidspace> right
  <voidspace> so we need 1.20 and onwards fixed really
  <voidspace> might be worth abentley pinging wallyworld about it
  <sinzui> voidspace: yeah, we can do a 1.20 release if we need to. We just need to make several jobs non-voting before hand to check to deal with unsupported features
  <voidspace> sinzui: although a 1.20 to 1.22 upgrade works I believe, 1.24 is the problematic version
  <voidspace> sinzui: so just fixing 1.22 on would suffice
  <voidspace> which is one less release at least
  <sinzui> voidspace: we can test/release 1.22 at anytime. we have special PPA setup for its distrubution too
  <voidspace> cool
  <voidspace> I've added a comment to the bug
  <mup> Bug #1510651 opened: Agents are "lost" after terminating a state server in an HA env <juju-core:New> <https://launchpad.net/bugs/1510651>
  <voidspace> frobware: our proposed fix for getting rid of ignore-machine-addresses can't work
  <voidspace> frobware: it's effectively what we're already doing
  <voidspace> frobware: we might need to prefer a provider fallback address over an exact match machine address
  <voidspace> frobware: I'll discuss it with dimiter tomorrow
  <voidspace> frobware: meanwhile, the fix for containers has landed on 1.25 and is landing on master
  <voidspace> frobware: and I've tested on both 1.25 and master
  <abentley> voidspace: pong
  <voidspace> abentley: unping
  <voidspace> abentley: it's about the wily issue and upgrades
  <voidspace> abentley: see the above conversation if you want... :-)
  <voidspace> abentley: I'll try and persuade wallyworld that we need the fix for 1.22 & 1.24
  <abentley> voidspace: I'm not sure whether we need 1.22.  1.24 would be nice, given that we may need to do a 1.24.8 at some point.
  <voidspace> abentley: well, if you want to test an upgrade *from* 1.22 (as I do as it happens and I thought you needed to) then we need it in 1.22
  <abentley> voidspace: Mwuh?  You mean he fixed the server instead of the client?
  <voidspace> abentley: the error message is from the api server
  <voidspace> not a client issue as far as I know
  <voidspace> abentley: see bug 1403689
  <mup> Bug #1403689: Server should handle tools of unknown or unsupported series <upgrade-juju> <upload-tools> <juju-core:Fix Released by wallyworld> <juju-core 1.24:Triaged> <juju-core 1.25:Fix Released by wallyworld> <https://launchpad.net/bugs/1403689>
  <voidspace> abentley: it's the api server that rejects the upload
  <abentley> voidspace: The client should not try to upload tools that it knows will be rejected by the server.
  <natefinch> katco: is your dog ok?
  <voidspace> abentley: not sure how it was fixed
  <mup> Bug #1510619 changed: Wily: add machine fails using kvm and lxcbr0 <add-machine> <cloud-installer> <kvm> <network> <juju-core:Invalid> <lxc (Ubuntu):New> <https://launchpad.net/bugs/1510619>
  <mup> Bug #1510619 opened: Wily: add machine fails using kvm and lxcbr0 <add-machine> <cloud-installer> <kvm> <network> <juju-core:Invalid> <lxc (Ubuntu):New> <https://launchpad.net/bugs/1510619>
  <mup> Bug #1510619 changed: Wily: add machine fails using kvm and lxcbr0 <add-machine> <cloud-installer> <kvm> <network> <juju-core:Invalid> <lxc (Ubuntu):New> <https://launchpad.net/bugs/1510619>
  <mup> Bug #1510675 opened: vSphere provider leaves behind virtual machines upon bootstrap failure <bootstrap> <vsphere> <juju-core:New> <https://launchpad.net/bugs/1510675>
  <mup> Bug #1510675 changed: vSphere provider leaves behind virtual machines upon bootstrap failure <bootstrap> <vsphere> <juju-core:New> <https://launchpad.net/bugs/1510675>
  <mup> Bug #1510675 opened: vSphere provider leaves behind virtual machines upon bootstrap failure <bootstrap> <vsphere> <juju-core:New> <https://launchpad.net/bugs/1510675>
  <mup> Bug #1510688 opened: sync-tools tries to upload agents that are not permitted by the state server <sync-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510688>
  <mup> Bug #1510689 opened: juju upgrade --upload-tools tries to upload tools agents that are not permitted by the state server <upgrade-juju> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510689>
  <mup> Bug #1510688 changed: sync-tools tries to upload agents that are not permitted by the state server <sync-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510688>
  <mup> Bug #1510689 changed: juju upgrade --upload-tools tries to upload tools agents that are not permitted by the state server <upgrade-juju> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510689>
  <mup> Bug #1510688 opened: sync-tools tries to upload agents that are not permitted by the state server <sync-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510688>
  <mup> Bug #1510689 opened: juju upgrade --upload-tools tries to upload tools agents that are not permitted by the state server <upgrade-juju> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1510689>
  <wallyworld> cmars: running a bit late, can i ping you
  <cmars> wallyworld, sure, np
  <alexisb> cmars, that was my fault, that wallyworld was late
  <cmars> alexisb, heh no worries
  <wallyworld> cmars: it's always alexisb fault
  <wallyworld> there now
  <wallyworld> axw: anastasiamac: perrito666: i have to make an emergency run to the school to drop off some stuff, so will miss standup; i'll talk to you guys when i get back to catch up
  <anastasiamac> wallyworld: k
  <perrito666> wallyworld: k
  #juju-dev 2015-10-28
  <thumper> menn0: https://github.com/juju/retry/pull/2
  <menn0> thumper: looking
  <thumper> cheers
  <thumper> wallyworld: do you know if the ci bot is even looking at master?
  <thumper> seems that the bug was fixed by cmars some time ago
  <wallyworld> thumper: they can only target one branch at a time, maybe 1.25 has been getting priority
  <thumper> probably...
   * thumper sighs
  <thumper> FARK!!!
  <thumper> peergrouper test
  <thumper> damnit
  <thumper> hate it!
  <davecheney> thumper: it _never_ passes for me
  <thumper> it mostly fails for me too
  <thumper> saying that, it just passed
  <thumper> if I run it by itself
  <thumper> second time it failed
  <menn0> thumper: review done. I ended up deleting and changing a lot of my initial comments so ignore the notification emails and go with what's on the PR
  <menn0> thumper: summary is I think MaxWait is a strange parameter to have. MaxDuration seems more intuitive to me.
  <menn0> thumper: I'd want to be able to say, "keep trying for up to 10 mins", not "keep trying while the sum of the waits between attempts is less than 10 mins"
  <menn0> thumper: if Func doesn't take long they work out to be about the same but I bet that sometimes Func can take a while making it hard to figure out exactly how long a series of attempts might take
  <thumper> menn0: yeah... I did wonder about that bit too
  <thumper> menn0: whether or not to take into consideration the time taken to call Func
  <thumper> menn0: although... it makes testing harder :)
  <thumper> menn0: because we need something to advance the clock...
  <menn0> thumper: if time is recorded when retry.Call is first called then it's easy to know how long everything has taken so far
  <thumper> menn0: however with the mock clock, it is um... different
  <menn0> the test Clock can do that right?
  <thumper> hmm...
  <thumper> we can make it do that
  <thumper> just by keeping track of now
  <thumper> and advancing now every time we are asked to sleep :)
  <thumper> menn0: here is a mind fcukingly boring review: https://github.com/juju/juju/pull/3605
  <thumper> ugh...
  <thumper> master CI failed due to deploying a local charm failing
  <thumper> WTF?
  <menn0> thumper: was otp, looking now
  <menn0> thumper: review done... super exciting that one
  <thumper> :)
  <mup> Bug #1510787 opened: juju upgrade-charm errors when it shouldn't <juju-core:Triaged> <https://launchpad.net/bugs/1510787>
  <mup> Bug #1509747 opened: Intermittent lxc failures on wily, juju-template-restart.service race condition <bug-squad> <cloud-installer> <lxc> <wily> <juju-core:Confirmed> <systemd (Ubuntu):Confirmed for pitti> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 changed: Intermittent lxc failures on wily, juju-template-restart.service race condition <bug-squad> <cloud-installer> <lxc> <wily> <juju-core:Confirmed> <systemd (Ubuntu):Confirmed for pitti> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 opened: Intermittent lxc failures on wily, juju-template-restart.service race condition <bug-squad> <cloud-installer> <lxc> <wily> <juju-core:Confirmed> <systemd (Ubuntu):Confirmed for pitti> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 changed: Intermittent lxc failures on wily, juju-template-restart.service race condition <bug-squad> <cloud-installer> <lxc> <wily> <juju-core:Confirmed> <systemd (Ubuntu):Confirmed for pitti> <https://launchpad.net/bugs/1509747>
  <mup> Bug #1509747 opened: Intermittent lxc failures on wily, juju-template-restart.service race condition <bug-squad> <cloud-installer> <lxc> <wily> <juju-core:Confirmed> <systemd (Ubuntu):Confirmed for pitti> <https://launchpad.net/bugs/1509747>
  <voidspace> dimitern: ping
  <voidspace> dimitern: do you know how I can reproduce a situation where "ignore-machine-addresses" is *required*?
  <voidspace> dimitern: (i.e. reproduce the bug it fixes)
  <dooferlad> frobware: ping
  <frobware> dooferlad, omw
  <dimitern> voidspace, I can think of an example
  <dimitern> voidspace, how about: bootstrap, juju ssh 0, add a virtual NIC with an address like 10.0.0.2, watch the logs to see it gets picked up and set among the API host ports
  <dimitern> voidspace, it won't work just by installing the lxc package outside of juju, as we're detecting the presence of /etc/default/lxc-net and parsing it to find the LXC bridge device name, then filter any addresses on it
  <voidspace> dimitern: ok, I'll try that
  <dimitern> voidspace, however, you can try: sudo apt-get install lxc -y after bootstrapping, then rm /etc/default/lxc-net, and reboot the node
  <voidspace> dimitern: our proposed fix from yesterday won't work - it's what we're already doing!
  <dimitern> (or just reboot jujud)
  <voidspace> dimitern: I think the fix might have to be that we prefer a *fallback address* from the provider list in favour of an exact scope match from machine
  <voidspace> dimitern: better would be to filter the addresses we send from the machine, so we don't send unusable addresses
  <dimitern> voidspace, we do that for lxcbr0 addresses, if we can detect them
  <voidspace> right
  <voidspace> better, we should ignore machine addresses by default and understand the circumstances where we actually need them
  <voidspace> (and use them only where we need them)
   * dimitern is thinking whether a fallback match from the provider addresses can always be preferred to an exact machine address match 
  <voidspace> dimitern: we can topic it in standup again
  <dimitern> voidspace, ok
   * voidspace is going to get coffee
  <dimitern> voidspace, hey, do you remember that change to "devices new" API that needed type=container in the arguments in order to show up in the UI?
  <voidspace> dimitern: I remember, but that isn't quite right
  <voidspace> dimitern: they have removed devices with a parent from the ui and last I saw they hadn't got round to putting them back
  <voidspace> dimitern: they're going to put them on the node detail pages
  <voidspace> dimitern: they decided not to use "type=container" afaik
  <voidspace> dimitern: so we don't need to do anything
  <voidspace> dimitern: but *currently* devices with a parent don't show in the ui at all
  <voidspace> which sucks
  <dimitern> voidspace, right, ok - as I couldn't find a reference to the type=container argument in maas src (1.8 or trunk)
  <voidspace> dimitern: if you find the maas bug (I can't recall it) they changed the title to remove the reference to type=container
  <voidspace> dimitern: they changed the title to "remove devices with a parent from the devices page"
  <voidspace> dimitern: and they have a new, not-yet-fixed bug, to add them to the node details page
  <voidspace> wallyworld: ping
  <dimitern> voidspace, I see, well - as long as the device is registered and maas can clean it up with its parent, I don't care so much it's not visible in the UI
  <voidspace> dimitern: I think it's a shame
  <voidspace> dimitern: I liked the visibility
  <voidspace> but ah well
  <dimitern> voidspace, it is a shame, but not quite ours ;)
  <dimitern> TheMue, jam, standup?
  <jam> brt
  <voidspace> dooferlad: I assume you checked those USB ethernet adaptors work with Ubuntu... ?
  <dooferlad> voidspace: reports say yes.
  <voidspace> dooferlad: cool, thanks
  <voidspace> dooferlad: and *not* the 4x SATA SSD - one disk is sufficient?
  <dooferlad> voidspace: we have been told to just go msata and the charm will be updated to use one disk
  <voidspace> dooferlad: ok
  <voidspace> dooferlad: those msata disks look funky
  <dooferlad> voidspace: overnighting a disk later if needed is a tradeoff we seem to be OK with.
  <voidspace> cool
  <dooferlad> voidspace: just plugged in one of those USB3 adapters. Shows up as enx00e060001127 on my desktop in the output of ifconfig
  <voidspace> dooferlad: great
  <mup> Bug #1510875 opened: unable to interrupt 'juju boostrap' on MAAS before the node is running <bootstrap> <juju-core:Triaged> <https://launchpad.net/bugs/1510875>
  <mup> Bug #1510875 changed: unable to interrupt 'juju boostrap' on MAAS before the node is running <bootstrap> <juju-core:Triaged> <https://launchpad.net/bugs/1510875>
  <mup> Bug #1510875 opened: unable to interrupt 'juju boostrap' on MAAS before the node is running <bootstrap> <juju-core:Triaged> <https://launchpad.net/bugs/1510875>
  <voidspace> dooferlad: and those nucs will be fine with my existing PDU
  <frobware> voidspace, AMT based NUCs, correct?
  <voidspace> frobware: non-AMT I believe
  <dooferlad> frobware: they are AMT
  <voidspace> hah
  <voidspace> frobware: believe dooferlad and not me...
  <frobware> voidspace, ok, just checking as if they're AMT based they can be shared...
  <frobware> voidspace, with other rigs should priorities change, et al.
  <dimitern> frobware, hw ordered; doc updated with actual prices (with exp. delivery to BG it turned out a bit more than 100 GBP cheaper!)
  <frobware> dimitern, great
  <dimitern> I mean, not due to the delivery, just price changes I guess
  <frobware> dimitern, dooferlad, voidspace: I suspect there's some surge pricing for those components going on in some amazon spreadsheet... :)
  <dooferlad> frobware: Amazon ran out of NUCs
  <dimitern> :D
  <dooferlad> frobware: so now they are Â£327 instead of Â£220
  <frobware> dooferlad, wow
  <dooferlad> frobware: now sold from Kikatek for that delightful mark up
  <dimitern> dooferlad, what I ordered is a bare bones kit - no ram, ssd
  <frobware> dimitern, which is pretty much the default, no?
  <dooferlad> dimitern: yes, that is what I got
  <dimitern> frobware, it is as offered from intel, but some vendors spice it up
  <dooferlad> http://www.intel.com/content/www/us/en/nuc/nuc-kit-dc53427hye-board-d53427rke.html
  <voidspace> dammit, I had some in my basket from amazon
  <voidspace> but too late to hit the button
  <voidspace> so I should probably cancel that order - unless I order the NUCs elsewhere at the higher price
  <voidspace> or just wait for amazon to get new stock
  <dooferlad> voidspace: amazon stock is due in weeks, right?
  <voidspace> dooferlad: allegedly
  <voidspace> dooferlad: I'm happy to wait, so long as they actually arrive
  <frobware> voidspace, I'm hoping we can make some progress with vMAAS and a smaller bundle to begin with
  <voidspace> frobware: yep, that would be good
  <mup> Bug #1510132 changed: redefinition of ânoMethods$MarshalYAML$hashâ <blocker> <ci> <ppc64el> <regression> <testing> <juju-core:Fix Released by cmars> <https://launchpad.net/bugs/1510132>
  <mup> Bug #1510944 opened: Only state-server upgrades from 1.20 <intermittent-failure> <upgrade-juju> <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1510944>
  <sinzui> cherylj: there is a regression in 1.22 https://bugs.launchpad.net/juju-core/+bug/1510952
  <mup> Bug #1510952: Upgrades broken in 1.22 tip <blocker> <ci> <regression> <upgrade-juju> <juju-core:Incomplete> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1510952>
  <cherylj> sinzui: thanks for the heads up
  <cherylj> sinzui: has that test run on 1.24 yet?
  <sinzui> cherylj: no, it hasn't
  <sinzui> It will test next I think
  <cherylj> ok, thanks
  <mup> Bug #1510951 opened: Upgrades broken in 1.22 tip <blocker> <ci> <regression> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1510951>
  <mup> Bug #1510952 opened: Upgrades broken in 1.22 tip <blocker> <ci> <regression> <upgrade-juju> <juju-core:Incomplete> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1510952>
  <natefinch> fwereade: I added setting of unit statuses in the last changes to the unit assigner review: http://reviews.vapour.ws/r/2814/
  <fwereade> natefinch, awesome, thanks
  <natefinch> er, in the last two changes, that is.
  <alexisb> dimitern, ping
  <dimitern> alexisb, oh, sorry - omw
   * fwereade needs to stop early, will be back this evening to talk to NZ
  <frobware> dooferlad, having just spoken with cherylj we should look at 1.24 first for #1510651
  <mup> Bug #1510651: Agents are "lost" after terminating a state server in an HA env <bug-squad> <ensure-availability> <juju-core:Triaged by dooferlad> <https://launchpad.net/bugs/1510651>
  <dooferlad> frobware: thanks, spotted the bug update
  <natefinch> wwitzel3: I know you're there, I can hear you typing
  <natefinch> ericsnow: you around?
  <ericsnow> natefinch: yep
  <natefinch> ericsnow: the lxd remote config value - I presume that should be an IP address or domain name?
  <ericsnow> natefinch: yep
  <ericsnow> natefinch: I'll make that more clear
  <cherylj> perrito666: got a sec?
  <natefinch> ericsnow: I'm adding an isLocal() function to the environ, since we're checking that in a couple places
  <ericsnow> natefinch: k
  <natefinch> ericsnow: (the logic is simple, but I don't want remote() == "" getting tossed around the codebase everywhere
  <ericsnow> natefinch: fine with me
  <ericsnow> natefinch: method on environConfig?
  <natefinch> ericsnow: on the environ. I don't want to have to know to look at the config to determine if it's local.  There's some law about not doing foo.bar.baz()... forget now what it's called.  Just expose it at the higher level.  The config itself doesn't need to know about local or not, I don't think.
  <ericsnow> natefinch: you're doing foo.bar.<something> either way and the config is what identifies the remote as local or not
  <mup> Bug #1507771 changed: Juju deploy fails when template container cannot be started <cdo-qa> <lxc> <maas> <juju-core:Invalid> <https://launchpad.net/bugs/1507771>
  <natefinch> ericsnow: if you're calling it on the environ, then only environ needs to know that it's getting information from the config.  Everywher else just calls the function on the environ.   And the environ is the thing that is interpreting the config.  The config is just holding the data.  Adding isLocal to the config is putting environ-level logic in the data bag that is the config
  <ericsnow> natefinch: k
  <ericsnow> natefinch: not a big deal to me either way :)
  <ericsnow> natefinch: the method is useful regardless
  <natefinch> ericsnow: agreed :)
  <natefinch> ericsnow: fwiw, I started with it on the config and then decided it was probably more appropriate on the environ... but yeah, it's not a huge deal either way
  <mup> Bug #1507771 opened: Juju deploy fails when template container cannot be started <cdo-qa> <lxc> <maas> <juju-core:Invalid> <https://launchpad.net/bugs/1507771>
  <mup> Bug #1507771 changed: Juju deploy fails when template container cannot be started <cdo-qa> <lxc> <maas> <juju-core:Invalid> <https://launchpad.net/bugs/1507771>
  <natefinch> ericsnow: your dependency injection makes my goto definition useless :/
  <ericsnow> natefinch: you're using a goto?
  <natefinch> ericsnow: yeah, godef
  <natefinch> ericsnow: which will just go to the definition of the interface.. there's no real way around it
  <natefinch> (I think the go oracle has a way to find implementors of an interface, but I haven't been able to get that working)
  <ericsnow> natefinch: ah
  <ericsnow> natefinch: yeah, bummer
  <natefinch> ericsnow: the downside of interfaces everwhere... figuring out what code is actually getting called is hard
  <ericsnow> katco: you have any ideas on a name to use instead of "clientServerMethods"?
  <perrito666> cherylj: I do now
  <cherylj> perrito666: just wanted to follow up on bug 1507867
  <mup> Bug #1507867: juju upgrade failures <canonical-bootstack> <upgrade-juju> <juju-core:Triaged by mfoord> <https://launchpad.net/bugs/1507867>
  <cherylj> it looks like there were two pieces:  the ignore machine address for containers, and the mongo issues
  <cherylj> I see mfoord took the machine addresses / containers part
  <cherylj> are you still working the mongo side?
  <cherylj> or were you going to hand that off to sapphire?
  <perrito666> indeed, on the mongo issues we are stil waiting some logs that might shed light on the mather sadly the resources of bootstck are  a bit scarce right now and they cannot run the test yet, I intended to do the handoff once these logs where in place but I might as well do it now
  <cherylj> perrito666: should we open up a new bug to track just the mongo stuff?
  <perrito666> cherylj: that would be advisable looking again at the current one it is becoming a bit of a mix
  <cherylj> perrito666: okay, I will create one and try to extract the meaningful mongo bits
   * perrito666 is wondering why he came to work to the only coffee shop without internet in town
  <katco> ericsnow: well, 3 of them have to do with certificates, so there's a cert struct/interface in there for sure
  <katco> ericsnow: the wait and config... perhaps belong in 2 separate interfaces?
  <ericsnow> katco: clientServerMethods is just there to help organize the methods better (so the Client methods aren't split across several files)
  <ericsnow> katco: in my mind the name communicates that pretty well
  <katco> ericsnow: well, first, grouping them into a struct doesn't preclue them being split across files :)
  <katco> ericsnow: what do you mean by, "organize the methods"?
  <ericsnow> katco: my point is that they were methods of Client before I moved them under their own struct (which Client now embeds)
  <ericsnow> katco: grouping the different Client methods
  <ericsnow> katco: and splitting them across multiple files
  <katco> ericsnow: ok. so i think i'm just asking that you take that to its logical conclusion and group them into types that make sense
  <katco> ericsnow: "server" methods is too generic
  <ericsnow> katco: I was following the grouping I was already using (see conn_raw.go)
  <katco> ericsnow: i think you're grouping in terms of implementation, and it probably makes more sense to group in terms of what the group of methods do
  <katco> ericsnow: you even have comments grouping them
  <ericsnow> katco: right
  <ericsnow> katco: ah, so your concern is with "Server" rather than with the whole name?
  <katco> ericsnow: well, the client should take implementations of interface it needs
  <katco> ericsnow: so no, the whole thing
  <katco> ericsnow: the fact that it's for the client is implicit because the client is embedding the types
  <ericsnow> katco: all the client needs is the raw *lxd.Client (as returned by newRawClient())
  <ericsnow> katco: I can change it back so the clientServerMethods methods are methods of Client instead if you like
  <ericsnow> katco: that may be less confusing
  <ericsnow> katco: (though it splits Client methods across multiple files)
  <katco> ericsnow: no i think what you've done is a good thing, you're doing IoC
  <katco> ericsnow: i'm just saying split things out to be a little more fine-grained
  <ericsnow> katco: but it's not meant to be IOC
  <katco> ericsnow: well it is :) and it's a good thing
  <ericsnow> katco: IOC here would be providing the raw client as an interface (or multiple)
  <katco> ericsnow: that's exactly what i'm saying
  <ericsnow> katco: right; that's orthogonal to the clientServerMethods type
  <ericsnow> katco: I wasn't planning on doing any IOC yet with the raw client
  <katco> ericsnow: you've extracted out the functionality, but haven't encapsulated it in a way that makes sense
  <ericsnow> katco: my point is that I haven't extracted out any  functionality; it was already there as-is
  <katco> ericsnow: you extracted it out into another type, right?
  <ericsnow> katco: the only thing I did was move the Client methods in a given file into their own type so that I wouldn't have Client methods spread across mutliple files
  <katco> ericsnow: that is extracting the functionality out, isn't it?
  <katco> ericsnow: it's fine for now. if you just want to collect them in the same file, leave it on client and put them all in a file together
  <ericsnow> katco: hmm, one goal is to avoid massive files like that though
  <katco> ericsnow: no i mean, same type, new file
  <ericsnow> katco: that's the way I had it in the first place :)
  <ericsnow> katco: but having a type split across multiple files is a pain point
  <katco> ericsnow: that's better than a new type that isn't encapsulating properly :)
  <katco> ericsnow: it is?
  <ericsnow> katco: it has been for me
  <natefinch> katco, ericsnow: I definitely highly dislike having a single type split across files
  <natefinch> katco, ericsnow: it makes it harder to understand the type as a whole, and it means you have to go hunting through files to find the method you're interested in.
  <cherylj> perrito666: it looks like there was another bug spun off to track the ignore-machine-addresses with containers.  Let's keep bug 1507867 for the mongo issues.
  <mup> Bug #1507867: juju upgrade failures <canonical-bootstack> <upgrade-juju> <juju-core:Triaged by mfoord> <https://launchpad.net/bugs/1507867>
  <perrito666> sounds fair
  <cherylj> perrito666: I'm going to put in an update that we are waiting on logs from bootstack and assign the bug to you
  <cherylj> once you have a chance to talk with someone from sapphire to do a hand off, please update the bug assignee
  <perrito666> ok, Ill do a handoff as soon as we get these
  <cherylj> awesome, thanks!
  <perrito666> everything is awesome
   * perrito666 sings
  <cherylj> everything is cool when you're part of a team?
  <alexisb> wwitzel3, ping
  <mup> Bug #1511090 opened: MAAS documentation on jujucharms.com incorrectly advised disable network management <juju-core:New> <https://launchpad.net/bugs/1511090>
  <wwitzel3> alexisb: pong
  <alexisb> heya wwitzel3 I was going to ask you to reach out to adam, but you did
  <alexisb> wwitzel3, also do you have documentation regarding deploying openstack on lxd provider?  or is that someone else?
  <alexisb> I heard it now exists
  <wwitzel3> alexisb: we have an email chain between James Page and myself, but we didn't have any official docs yet since it is a lot tweaking still.
  <alexisb> wwitzel3, ack
  <alexisb> thanks!
  <ericsnow> natefinch: ptal: http://reviews.vapour.ws/r/2927/
  <natefinch> ericsnow: looking
  <ericsnow> natefinch: ta
  <fwereade> thumper, waigani, just joined onyx-standup
  <mup> Bug #1511103 opened: relation-get error: permission denied <juju-core:New> <https://launchpad.net/bugs/1511103>
  <mup> Bug #1511103 changed: relation-get error: permission denied <juju-core:New> <https://launchpad.net/bugs/1511103>
  <mup> Bug #1511103 opened: relation-get error: permission denied <juju-core:New> <https://launchpad.net/bugs/1511103>
  <mup> Bug #1511135 opened: storage: add bundle support <juju-core:Triaged> <https://launchpad.net/bugs/1511135>
  <mup> Bug #1511135 changed: storage: add bundle support <juju-core:Triaged> <https://launchpad.net/bugs/1511135>
  <mup> Bug #1511135 opened: storage: add bundle support <juju-core:Triaged> <https://launchpad.net/bugs/1511135>
  <mup> Bug #1511138 opened: Bootstrap with the vSphere provider fails to log into the virtual machine <bootstrap> <cloud-init> <vsphere> <juju-core:Triaged> <https://launchpad.net/bugs/1511138>
  <mup> Bug #1511138 changed: Bootstrap with the vSphere provider fails to log into the virtual machine <bootstrap> <cloud-init> <vsphere> <juju-core:Triaged> <https://launchpad.net/bugs/1511138>
  <mup> Bug #1511138 opened: Bootstrap with the vSphere provider fails to log into the virtual machine <bootstrap> <cloud-init> <vsphere> <juju-core:Triaged> <https://launchpad.net/bugs/1511138>
  <ericsnow> cmars: so, you like my latest LXD branch, huh? :)
  <ericsnow> cmars: wwitzel3 has one up that fixes the remaining bugs you noted: http://reviews.vapour.ws/r/3015/
  <cmars> ericsnow, wwitzel3 you guys rock
  <ericsnow> cmars: hey, it's a fun one to work on :)
  <wwitzel3> indeed
  #juju-dev 2015-10-29
  <mup> Bug #1511090 changed: MAAS documentation on jujucharms.com incorrectly advised disable network management <juju-core:Invalid> <https://launchpad.net/bugs/1511090>
  <mup> Bug #1511090 opened: MAAS documentation on jujucharms.com incorrectly advised disable network management <juju-core:Invalid> <https://launchpad.net/bugs/1511090>
  <mup> Bug #1511090 changed: MAAS documentation on jujucharms.com incorrectly advised disable network management <juju-core:Invalid> <https://launchpad.net/bugs/1511090>
  <thumper> ericsnow: around?
  <thumper> menn0: http://reviews.vapour.ws/r/3018/diff/# a rename PR that I thought was up already
  <thumper> grrr... FFS
   * thumper headdesks
  <menn0> thumper: ship it
  <thumper> ta
  <menn0> thumper: that one should have come with a warning about not operating heavy machine due to drowsiness
  <menn0> :-p
  <thumper> :)
  <natefinch> ericsnow: don't suppose you're around?
  <thumper> menn0: what watches the allwatcher?
  <menn0> thumper: what do you mean? what uses it?
  <thumper> yeah...
  <thumper> It has a ServerUUID on environmenst
  <thumper> and I want to rename it :)
  <menn0> thumper: it's exposed via the apiserver ... primarily used by the GUI
  <thumper> so... probably no one is looking at the ServerUUID...
  <thumper> because it is always the same...
  <thumper> for now at least
  <menn0> thumper: the reporting of environments was only just added
  <menn0> for the allenvwatcher
   * thumper nods
  <menn0> thumper: the allwatcher (aka megawatcher) doesn't return environment entities
  <menn0> thumper: it's only the allenvwatcher
  <menn0> thumper: you might want to check with rog because he's the only potential user of that right now
  <menn0> thumper: just let him know it's changing
  <thumper> I just sent an email to rog and uros saying names are changing
  <wallyworld> menn0: is it true that when we define a new state doc for a collection, we no longer need EnvUUID   string `bson:"env-uuid"
  <menn0> wallyworld: that's correct, the underlying multi-env layer will add it
  <wallyworld> menn0: awesome, and TxnRevno ?
  <menn0> wallyworld: I have a tech debt item to remove all the existing structs and code the constructs them
  <wallyworld> ok, i won't add to your burden :-)
  <menn0> wallyworld: you don't normally put txnrevno on your structs
  <wallyworld> rightio
  <menn0> wallyworld: that's a mgo/txn field
  <wallyworld> indeed
  <wallyworld> i guess we use it in places
  <menn0> wallyworld: some places in the code do read it, but they're rare/evil cases
  <wallyworld> that's what i thought
  <wallyworld> thanks
  <mup> Bug #1511235 opened: juju-reboot --now defect in 1.24.7 <juju-core:New> <https://launchpad.net/bugs/1511235>
  <mup> Bug #1511235 changed: juju-reboot --now defect in 1.24.7 <juju-core:New> <https://launchpad.net/bugs/1511235>
  <mup> Bug #1511235 opened: juju-reboot --now defect in 1.24.7 <juju-core:New> <https://launchpad.net/bugs/1511235>
  <dimitern> jam, hey, I wanted to pass an idea by you, if you have 5m..
  <dimitern> jam, as part of fixing that d-m and d-e --force issue on maas, I'm thinking of adding a few optional methods that a provider can implement if it wants to observe containers - OnContainerIni, OnContainerCreate, OnContainerDestroy
  <dimitern> jam, those are going to be called at the apiserver side of the provisioner - at the one-time container init step (per container type), during the container broker's StartInstance, and before the broker destroys a container
  <dimitern> jam, this will give maas and other providers that want to, the ability to hook into those steps and do extra stuff - i.e. in maas, creating/removing a device for the container, allocating IPs from the provider, etc.
  <jam> dimitern: what is Init vs Create ?
  <jam> dimitern: and how does the APIServer know the ContainerBroker is doing StartInstance, isn't ContainerBroker part of the machine agent?
  <dimitern> jam, Init is the one-time container initialization the first time we need to provision a container of a given type, Create is called before actually starting the container (ideally the callback can take the cloudconfig and modify it, as well as the container entity to tweak its config)
  <dimitern> jam, the apiserver knows we're about to provision a container as the container provisioner, which uses the container broker, will call ContainerManagerConfig on Init and ProvisioningInfo before Start
  <jam> dimitern: I might call it something like "InitializeContainers" then, but maybe not. not a big deal
  <jam> dimitern: I don't really think we want side-effects from calling Config, but having the InstanceBroker send an explicit message to theAPI server that its about to start a container is fine
  <dimitern> jam, I'm ok with finding better names, but the idea is very useful I think - and can decouple provider-specific container management from the provisioner
  <jam> dimitern: sure
  <jam> for AMZ we want to be able to grab IP addresses as well
  <jam> ContainerInit might also be a time when we reconfigure bridges, etc. in the future
  <dimitern> jam, yeah, and for maas - create a device, possibly allocate an IP as well
  <dimitern> jam, this is the rough sketch - http://paste.ubuntu.com/12997865/ (in the container package) << fwereade, I'd like your thoughts on this as well - we can topic it during standup
  <dimitern> jam, fwereade, voidspace, frobware, standup?
  <jam> omw
  <frobware> dimitern, voidspace, jam: as utopic is no longer supported do we care that our 'add juju-br0' to /e/n/i does not work?
  <dimitern> frobware, I'm not sure I follow you - what's utopic specific about that?
  <frobware> dimitern, the ifup fails.
  <rogpeppe> anyone know of a tool that can merge Go coverage profiles?
  <rogpeppe> axw: ^
  <jam> frobware: I don't think we care if it is broken on U, but it matters if it doesn't work on T or V/W/X
  <rogpeppe> here's a branch that fixes two outstanding hight-priority juju-core bugs (and makes tools uploading work with macaroons too): http://reviews.vapour.ws/r/3023/; reviews appreciated.
  <rogpeppe> s/hight/high/
  <dimitern> frobware, sorry, got distracted with the iter. spreadsheet
  <dimitern> frobware, ifup juju-br0 fails only on utopic due to what?
  <frobware> dimitern, no such device
  <dimitern> frobware, even though juju-br0 is in /e/n/i ?
  <frobware> dimitern, yes
  <frobware> dimitern, I didn't look further - perhaps bridge-utils is missing.
  <dimitern> frobware, but I guess that's because even if /e/n/i has juju-br0, /run/network/ifstate doesn't
  <frobware> dimitern, possibly; but trusty, vivid, wily are OK
  <dimitern> frobware, I think we can ignore utopic then :)
  <voidspace> sounds good to me
  <frobware> dimitern, I already have.
  <voidspace> I even managed to upgrade my laptop (one of the few flawless upgrades I've ever done...)
  <frobware> voidspace, news in your INBOX
  <voidspace> frobware: I saw, I can pick it up after my current branch is done - but that will mean Monday
  <perrito666> frobware: you are making a risky bet on how voidspace manages his mail filters :p
  <frobware> perrito666, I just beame his filter. :)
  <frobware> s/beam/became/...
  <voidspace> hah
  <voidspace> perrito666: if/when I pick it up I'll probably talk to you about it
  <perrito666> mm, that would be a very interesting feature on mail "make sure this mail stays in  the inbox"
  <perrito666> voidspace: sure I have all sorts of bad things to say about
  <frobware> perrito666, about ...
  <perrito666> the task :)
  <frobware> perrito666, anything you want to get off your chest. :)
  <perrito666> nah :)
  <voidspace> :-)
  <perrito666> frobware: voidspace none of you is telling me if its off or over :p
  <frobware> perrito666, https://www.englishforums.com/English/ToHandOverAndToHandOff/bchhkz/post.htm
  <frobware> perrito666, the slang would be "chucking it over the wall" - and thanks! :)
  <perrito666> we should all switch to spanish
  <perrito666> :p
  <frobware> we should all switch to spanish beer
  <perrito666> frobware: well for what I have seen in english speakers, beer can give them the idea that they can speak spanish
  <perrito666> :p
  <frobware> voidspace, please could you put your last comment in the bug if not already captured -- seems useful info.
  <voidspace> frobware: ok
  <jam> dooferlad: dimitern: https://github.com/dooferlad/kvm_maas/pull/2 is my update to the script that is working here (at least for 1 node) with multiple subnets/interfaces
  <dimitern> jam, nice! I'll give it a try on my vmaas later
  <jam> the CLI I tested is: ./kmaas.py m1-test1 -s 10.10.0.0/24 -s 10.10.100.1/24 -t dual-nic-template.xml --debug
  <jam> where 10.10.0.0/24 and 10.10.100.0 is registered already in both virt-manager and maas
  <dimitern> and maas CC is on both of these, but managing only the first one?
  <jam> dimitern: yeah, CC is managing 10.10.0.0
  <jam> 10.10.100.0 is registerd with virt as an internal-only netwok
  <dimitern> right, ok
  <mup> Bug #1511390 opened: Setup of MAAS Server using pod-cloud-installer failing <juju-core:New> <https://launchpad.net/bugs/1511390>
  <mup> Bug #1511390 changed: Setup of MAAS Server using pod-cloud-installer failing <juju-core:New> <https://launchpad.net/bugs/1511390>
  <katco> wwitzel3: meeting time
  <fwereade> natefinch, is this the right diff range: http://reviews.vapour.ws/r/2814/diff/24-26/ ?
  <fwereade> natefinch, assuming it is, LGTM with a trivial
  <perrito666> bbl lunch with wife
  <rogpeppe> please could someone review this change (it fixes two high-priority bugs)? http://reviews.vapour.ws/r/3023/
  <rogpeppe> ericsnow: ^
  <ericsnow> rogpeppe: will do
  <rogpeppe> ericsnow: thanks
  <rogpeppe> ericsnow: and another little one: http://reviews.vapour.ws/r/3026/
  <ericsnow> rogpeppe: k
  <rogpeppe> ericsnow: ta
  <rogpeppe> ericsnow: responded to your review of http://reviews.vapour.ws/r/3023/
  <ericsnow> rogpeppe: k
  <frobware> voidspace, http://reviews.vapour.ws/r/3027/
  <voidspace> frobware: looking
  <voidspace> frobware: LGTM
  <frobware> voidspace, thanks. I forget - to merge this it is just $$merge$$ in the PR?
  <natefinch> fwereade: that's the right diff, yeah
  <voidspace> frobware: correct
  <frobware> voidspace, and because it is not master do I need some incantation with [fixes-xyz]?
  <alexisb> heh juju core team, meeting?
  <alexisb> wwitzel3, perrito666, cherylj, natefinch
  <alexisb> others who are around and it is not there eod
  <cherylj> brt
  <voidspace> frobware: no, you only need $$fixes-xyz$$  if the branch you're merging to (including master) is blocked by a critical bug
  <frobware> voidspace, thanks for the clarification
  <alexisb> ericsnow, ping
  <katco> ericsnow: wwitzel3: not coming to the core meeting?
  <frobware> cherylj, I saw that https://bugs.launchpad.net/ubuntu/+bug/1496972 was just reset to 1.26, but I had already started landing into 1.25.
  <mup> Bug #1496972: juju bootstrap fails to successfully configure the bridge juju-br0 when deploying with wily 4.2 kernel <hs-arm64> <kernel-da-key> <network> <juju-core:Triaged
  <mup> by frobware> <juju-core 1.25:Triaged by frobware> <Ubuntu:Invalid by jsalisbury> <Ubuntu Wily:Invalid by jsalisbury> <https://launchpad.net/bugs/1496972>
  <cherylj> frobware: I just created a 1.25 task
  <cherylj> frobware: so it should go into 1.25 and 1.26
  <frobware> cherylj, it's currently landing in 1.25.... ok
  <frobware> cherylj, also wondering whether this should go into 1.24
  <cherylj> frobware: how complex is the fix?
  <frobware> cherylj, trivial but significant: http://reviews.vapour.ws/r/3027/
  <ericsnow> rogpeppe: updates to both patches LGTM
  <rogpeppe> ericsnow: ta!
  <wwitzel3> katco: was away eating, forgot about it, you all still going?
  <natefinch> wwitzel3: done now
  <alexisb> wwitzel3, I figured you were out rolling tractor tires
  <rogpeppe> mgz_: just reviewed https://github.com/juju/charm/pull/164
  <mgz_> rogpeppe: they're ignored because I wanted to preserve existing behaviour to make the branch easy to land
  <rogpeppe> mgz_: ok, fair enough
  <rogpeppe> mgz_: please add a TODO, then LGTM
  <mgz_> but it got messed up anyway because callers are so confused
  <mgz_> rogpeppe: okay
  <rogpeppe> mgz_: unfortunately this commit in utils breaks some tests in juju-core: e6c1e5206ced12cd6201ad2a488e3df6a245eed4
  <mgz_> rogpeppe: that's expected, the mass dep bump will need to bring in test changes as well
  <rogpeppe> mgz_: yeah
  <rogpeppe> mgz_: i'll stick on the old utils for now then
  <voidspace> EOW
  <voidspace> o/
  <perrito666> voidspace: bye
  <katco> ericsnow: i'm ready when you are
  <jog> rogpeppe, looks like a possible regression in the latest CI run on master, related to cookies
  <rogpeppe> jog: interesting, thanks. i'll take a look.
  <jog> rogpeppe, I'll open a bug but you can see failures here: http://reports.vapour.ws/releases/rules/735
  <rogpeppe> jog: ah yes, i knew about that but thought it wouldn't be a problem for CI
  <rogpeppe> jog: the cookie format has changed
  <rogpeppe> jog: i did consider ignoring the error
  <rogpeppe> jog: but thought it was easy enough for people to delete the cookie file if they got that error
  <rogpeppe> jog: do you know if that CI run is starting with a fresh home dir?
  <rogpeppe> jog: given that the juju version using the previous format hasn't been released, i am surprised that this is an issue
  <jog> rogpeppe, we ensure jenv file are removed but I don't think we clean the entire JUJU_HOME
  <rogpeppe> jog: ah
  <rogpeppe> jog: that seems like it's an easy way to run into test pollution to me
  <rogpeppe> jog: i think you should really clean $HOME/.go-cookies before running the tests at the very least
  <mup> Bug #1511537 opened: Failed to load cookies EOF <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1511537>
  <rogpeppe> jog: or set $JUJU_COOKIEFILE to something else
  <jog> rogpeppe, I'm checking our test slaves
  <rogpeppe> jog: hmm, that bug is another interesting one.
  <jog> ses just opened that one for something he saw yesterday
  <rogpeppe> jog: this makes me more inclined to the view that failing to load cookies should just be a warning not an error
  <rogpeppe> jog: i swithered about that decision when implementing this
  <jog> where should the .go-cookies file be? In $JUJU_HOME?
  <rogpeppe> jog: usually in $HOME/.go-cookies
  <rogpeppe> jog: but $JUJU_COOKIEFILE can place them elsewhere
  <rogpeppe> jog: how can i find out the commit hash of http://reports.vapour.ws/releases/3230/job/functional-ha-backup-restore/attempt/2766 ?
  <jog> rogpeppe, there are breadcrumbs at the top of that page. Go back one to http://reports.vapour.ws/releases/3230
  <jog> you'll see 'Revision ID' with a link near the top
  <rogpeppe> jog: ah, i didn't realise that grey text was clickable
  <rogpeppe> jog: i *think* that bug (1511537) has been fixed by the dependency update landed today
  <jog> rogpeppe, ah... you said in $HOME... .go-cookies is global for the user
  <rogpeppe> jog: i think the intermittent problem was probably caused by the fact that concurrent juju client instances were stepping one one anothers' toes because the cookie logic wasn't safe for concurrent use
  <rogpeppe> jog: yes it is (very deliberately)
  <jog> so we drive multiple tests in parallel from a single slave...
  <rogpeppe> jog: ok, that makes sense
  <jog> and they can be varying version of Juju
  <rogpeppe> jog: do you use a different value of $JUJU_HOME for each running test?
  <jog> yes
  <jog> err... no
  <jog> each test has a unique environment name though
  <rogpeppe> jog: ok, well you should probably do that. and set JUJU_COOKIEFILE=$JUJU_HOME/cookies
  <rogpeppe> jog: i mean, it's *supposed* to work ok with everything concurrently, but i think it's probably best to have each test as isolated as possible (i'd probably have a different value of $HOME too)
  <jog> we have a $JUJU_HOME with an environment.yaml for each to the various clouds and substrates that we test on.
  <rogpeppe> jog: anyway, this will be more important in the future when you have CI tests that rely on macaroon authorization
  <jog> rogpeppe, with 75+ tests that's not a trivial change for CI. There is also likely CI driver code that sets $JUJU_HOME. Not that it can't be done but not at the flip of a switch.
  <rogpeppe> jog: fair enough
  <rogpeppe> jog: anyway, i hope the intermittent issue should have gone away now
  <rogpeppe> jog: and deleting the cookie file should cause the CI to succeed OK
  <rogpeppe> jog: well, at least to fix *that* issue :)
  <rogpeppe> jog: you could always set JUJU_COOKIEFILE=$(tempfile)
  <jog> that's that type of change I'm thinking of making in the short term
  <jog> also are you going to make that a Warning?
  <rogpeppe> jog: the reason i'm not sure about that is that i'm not sure of the best API for persistent-cookiejar. It seems kind of wrong to not give the caller any info about the fact that loading the cookie jar has failed, but the alternative is returning both a valid cookiejar *and* an error which also seems wrong
  <rogpeppe> jog: for the time being, i might leave it as an error but if it causes further CI issues, i'll change it
  <rogpeppe> jog: after all, if it hadn't been an error, we wouldn't have discovered this potential test cross-talk issue
  <rogpeppe> jog: so it's not all bad :)
  <mup> Bug #1511543 opened: Unhelpful error message when trying from a location with a full disk <juju-core:Triaged> <https://launchpad.net/bugs/1511543>
  <axw> rogpeppe: gocov can merge profiles
  <axw> rogpeppe: not aware of any others
  #juju-dev 2015-10-30
  <mup> Bug #1511589 opened: maas provider, hwclock out of sync means juju will not work <kanban-cross-team> <landscape> <maas-provider> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1511589>
  <rogpeppe> axw: how would i use gocov to merge coverage profiles? it's not clear from the README
  <axw> rogpeppe: sorry, IIANM you can "gocov convert profile [profile...]". but it's going to give you gocov JSON output, if it's not clear
  <rogpeppe> axw: so my aim here was to try and see the combined test coverage of juju/api when running the juju/apiserver/client and juju/api tests
  <rogpeppe> axw: and i like the nice red-green line-by-line HTML output of the usual coverage tool
  <rogpeppe> axw: can i use gocov to do that?
  <axw> rogpeppe: cmars added support to gocov for testing multiple packages, so you should be able to "gocov test juju/api juju/apiserver/client"
  <axw> rogpeppe: there's command-line "gocov annotate" for annotated source, and https://github.com/matm/gocov-html if you want HTML
  <axw> rogpeppe: (and I just noticed that doesn't do red/green, assuming the screenshot is representative; sorry)
  <rogpeppe> axw: trying it anyway. we'll see.
  <rogpeppe> axw: hmm, doesn't seem to have worked: http://paste.ubuntu.com/13007149/
  <rogpeppe> axw: and /tmp/cov.out is empty
  <axw> rogpeppe: hrm, I'll take a look
  <axw> rogpeppe: did you just fetch gocov?
  <rogpeppe> axw: hmm, maybe i should've used go test then gocov convert
  <rogpeppe> axw: yeah
  <axw> rogpeppe: that should work
  <rogpeppe> axw: um
  <rogpeppe> axw: it looks like i didn't
  <rogpeppe> % ls -l `{whatis gocov}
  <rogpeppe> -rwxrwxr-x 1 rog rog 3907964 Mar 22  2013 /home/rog/src/go/bin/gocov
  <axw> oops :)
  <rogpeppe> axw: ah!
  <rogpeppe> axw: the command isn't in the top level dir
  <rogpeppe> axw: i did go install github.com/axw/gocov
  <axw> rogpeppe: hrm, I've just realised that "gocov test <pkg>" does not tell you about dependent packages. so you'll need to specify -coverpkg
  <rogpeppe> axw: ah, i wondered about that
  <axw> rogpeppe: original gocov used to do it, but then it became a shim around "go test -cover"
  <axw> it should probably do it again, since it supports multiple packages now
  <rogpeppe> axw: hmm
  <rogpeppe> % gocov test -coverpkg ./api ./api ./apiserver/client > /tmp/cov.out
  <rogpeppe> warning: no packages being tested depend on github.com/juju/juju/api
   * axw tries
  <rogpeppe> axw: that doesn't seem right to me - both those packages depend on juju/api
  <axw> rogpeppe: maybe something to do with relative package path?
  <rogpeppe> axw: i'll try abs
  <axw> rogpeppe: I just tried it on a dummy package
  <axw> worked ok
  <rogpeppe> axw: using absolute path didn't help
  <rogpeppe> axw: i'll try go test then gocov convert
  <axw> rogpeppe: weird, putting "-coverpkg ./api" *after* the package names seems to work ...
  <axw> rogpeppe: hmm or maybe not
  <axw> ok just do it your way ;p
  <rogpeppe> axw: trying it now
   * rogpeppe wishes command line flag parsing wasn't context-sensitive
  <axw> rogpeppe: oh it does work, I was seeing warnings for another reason (I was using ./... and not all packages depend on it)
  <frobware> dimitern, please could you take a look at http://reviews.vapour.ws/r/3030/
  <axw> rogpeppe: http://paste.ubuntu.com/13007210/
  <axw> rogpeppe: I read that as: api tests 56% of api, apiserver/client tests 43.7%, together they do 77.78%
  <rogpeppe> axw: that doesn't look right to me
  <rogpeppe> axw: for example EnsureAvailability is well tested by apiserver/client but comes up as 0%
  <axw> hrm
  <dimitern> frobware, sure, looking
  <axw> rogpeppe: seems to happen regardless of whether I'm merging... checking what happens when I just use "go test -cover"
  <rogpeppe> axw: so i get the same issue when manually using gocov convert with multiple files
  <dimitern> frobware, lgtm
  <axw> rogpeppe: just tested with "go test -coverpkg github.com/juju/juju/api <etc>" and it shows 0% coverage for Client.EnsureAvailability
  <rogpeppe> axw: yeah, i just saw that too. maybe our tests are at fault
  <axw> rogpeppe: ah... it's calling the apiserver/client code directly I think
  <rogpeppe> axw: ah, so it is
  <rogpeppe> axw: ok, we're all mixed up in our testing approaches here
  <rogpeppe> axw: thanks
  <axw> rogpeppe: nps
  <axw> rogpeppe: one of these days I'd like to get coverage generated by CI
  <axw> merged for the whole tree
  <rogpeppe> axw: yes, that would be great
  <dimitern> axw, hey, since you're here - a question: do you know if there's any way in a manual environment to add a new machine and a new container on it in a single command?
  <dimitern> e.g. like in other envs you can add-machine lxc
  <dimitern> I guessed not, but wasn't completely sure
  <mup> Bug #1511659 opened: Destroyed leader, new leader not elected. <juju-core:New> <https://launchpad.net/bugs/1511659>
  <mup> Bug #1511659 changed: Destroyed leader, new leader not elected. <juju-core:New> <https://launchpad.net/bugs/1511659>
  <mup> Bug #1511659 opened: Destroyed leader, new leader not elected. <juju-core:New> <https://launchpad.net/bugs/1511659>
  <dimitern> fwereade, hey, how's that for a better UX: http://paste.ubuntu.com/13007523/
  <dimitern> I'll fix the typo
  <fwereade> dimitern, nice
  <dimitern> fwereade, I've added a PrecheckContainerHost method on the state.Policy.Prechecker
  <dimitern> fwereade, which can be called with or without the host being provisioned (the latter in the case you've done $ juju add-machine lxc)
  <fwereade> dimitern, excellent
  <dimitern> fwereade, yeah, I agree it's much cleaner like this; now just a couple of provisioner API methods are needed to allow the worker to notify the apiserver that it's about to start/stop a container, so the NICs can be created with the pre-generated MAC etc.
  <fwereade> dimitern, yeah, sgtm
  <fwereade> dimitern, lovely
  <dimitern> jam, dooferlad, standup?
  <frobware> anybody else see "cannot load cookies: json: cannot unmarshal object into Go value of type []cookiejar.entry" - this from 'juju ssh 0'
  <dimitern> frobware, you probably need to update the deps
  <frobware> dimitern, done that
  <dimitern> frobware, so that's not a build error?
  <frobware> dimitern, bootstrap works, trying to 'juju ssh 0' borks
  <dimitern> frobware, hmm I haven't seen this - try with juju ssh --debug 0
  <frobware> dimitern, just starting a git bisect to see if it is just me, or something recent.
  <dimitern> frobware, I think I noticed some sort of bug related to loading cookies and bootstrap issues today or yesterday - might be worth a quick search
   * dimitern is surprised to find out $ juju deploy mysvc --to 0,lxc:1,2,lxc works but it's not documented
  <mup> Bug #1511717 opened: Incompatible cookie format change <compatibility> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1511717>
  <mup> Bug #1511717 changed: Incompatible cookie format change <compatibility> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1511717>
  <mup> Bug #1511717 opened: Incompatible cookie format change <compatibility> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1511717>
  <mup> Bug #1511589 changed: maas provider, hwclock out of sync means juju will not work <landscape> <maas-provider> <juju-core:Invalid> <MAAS:New> <https://launchpad.net/bugs/1511589>
  <frobware> dimitern, forward port to master - http://reviews.vapour.ws/r/3031/ (do these need reviewing if they have been reviewed on 1.25?)
  <dimitern> frobware, will have a look in a bit; yes - they do need reviewing (albeit a quick one, since it's a port)
  <frobware> dimitern, well this was a cherry-pick
  <dimitern> frobware, ship it! ;)
  <katco> natefinch: sorry, what's the status on http://reviews.vapour.ws/r/2814/ ?
  <katco> ericsnow: you should have your reviews from me
  <ericsnow> katco: thanks!
  <natefinch> katco: I have to write a couple small tests and then I have a shipit from william
  <katco> natefinch: k, i know it's friday labs, but extenuating circumstances, can you land that today?
  <natefinch> katco: yeah... I think friday labs is only friday afternoons anyway, right?
  <katco> natefinch: i think we can spare a day :)
  <natefinch> katco: regardless, it should be very fast to finish up
  <katco> natefinch: cool
  <natefinch> Is there a way to make a new unit without having to invoke methods on state?  there's a state.newUnit, which is obviously not exported, and the fields of Unit are not exported...
  <natefinch> (this is for testing)
  <natefinch> fwereade, dimitern, rogpeppe, wwitzel3, ericsnow, katco, etc ^
  <katco> natefinch: there's the factory
  <ericsnow> natefinch: I've only ever used the state stuff (see the helper in state/workloads_test.go)
  <ericsnow> natefinch: but that uses ConnSuite
  <katco> natefinch: ericsnow: sec, hunting for this
  <rogpeppe> natefinch: what are you trying to do?
  <dimitern> natefinch, why without calling a state method?
  <katco> natefinch: https://github.com/juju/juju/blob/master/testing/factory/factory.go
  <natefinch> rogpeppe: test a function that takes a unit without doing a full stack test
  <katco> natefinch: but forgot it still uses state =/
  <natefinch> dimitern: because using state will make my test take 100 times longer
  <rogpeppe> natefinch: you could always use an interface type
  <katco> natefinch: don't dwell on this too long. your budget is spent
  <rogpeppe> natefinch: that's not a problem to be solved in isolation
  <rogpeppe> natefinch: someone should spend some time looking at how to speed up *all* state-based tests
  <rogpeppe> natefinch: i can't believe noone seems to have allocated any time for that
  <natefinch> rogpeppe: I don't have time to solve the problem in general. I just want to write a unit test
  <katco> rogpeppe: i think what he's saying is that he doesn't want it to be state based at all
  <rogpeppe> natefinch: so write the test the same way as all the other tests are written
  <dimitern> natefinch, ah, well - interfaces & mocking then
  <natefinch> the code in the function that I'm testing is very very simple
  <rogpeppe> natefinch: making it different will only make things harder in the long run
  <rogpeppe> natefinch: what does your function look like?
  <natefinch> rogpeppe: I think I can fix it with judicious application of a tiny special-case interface, actually
  <rogpeppe> natefinch: that's often the case
  <natefinch> instead of casting a state.Entity to *state.Unit. cast to an interface with the one function I need from Unit.
  <frobware> cherylj, ping - any chance we could quickly chat about https://bugs.launchpad.net/juju-core/+bug/1412621
  <mup> Bug #1412621: replica set EMPTYCONFIG MAAS bootstrap <adoption> <bootstrap> <bug-squad> <charmers> <cpec> <cpp> <maas-provider> <mongodb> <oil> <juju-core:Triaged by frobware> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1412621>
  <fwereade> natefinch, +100 to that, if you must cast cast to the narrowest possible interface
  <natefinch> fwereade: yep, just a single function.  And now my 3 tests take less than a millisecond each to run.
  <fwereade> natefinch, <3
  <fwereade> natefinch, and you can get nice coverage of weird errors as well, and it'll still take less than a ms :)
  <natefinch> fwereade: yep.  tested the error paths as well.
  <fwereade> natefinch, perfect :)
  <katco> natefinch: well done :)
  <mup> Bug #1511771 opened: regression setting tools-metadata-url <blocker> <ci> <regression> <set-env> <juju-core:Triaged> <https://launchpad.net/bugs/1511771>
   * wwitzel3 goes to look at those tests to remember the implementation
  <natefinch> This branch has conflicts that must be resolved
  <natefinch> noooooooo!!!
  <natefinch> I guess it's to be expected given that it's a 29 day old PR :/
  <natefinch> gah, why does git rebase master tell me I'm up to date when github says there's conflicts merging?
  <natefinch> ahh, my local master was out of date
  <natefinch> katco: merging the unitassigner stuff now.... assuming the bot is working and will pick it up, which is looking somewhat questionable currently.
  <katco> natefinch: eh? why, what's going on with the bot?
  <natefinch> katco: nevermind, it was just being slo
  <natefinch> except that master is blocked now... sonofa
  <alexisb> frobware, cherylj is out today
  <alexisb> I need a volunteer to help with a friday afternoon activity
  <alexisb> anyone around and willing?
  <natefinch> alexisb: I'm around.  I'd rather work on minjujuversion, but can do other things as needed
  <alexisb> natefinch, this shouldnt take long, I just have a couple of qs to get me unstuck
  <mup> Bug #1511812 opened: 1.25.0 cannot bootstrap local environment on wily host  <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1511812>
  <mup> Bug #1511812 changed: 1.25.0 cannot bootstrap local environment on wily host  <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1511812>
  <mup> Bug #1511812 opened: 1.25.0 cannot bootstrap local environment on wily host  <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1511812>
  <mup> Bug #1511822 opened: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <wily> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1511822>
  <mup> Bug #1511822 changed: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <wily> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1511822>
  <mup> Bug #1511822 opened: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <wily> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1511822>
  <mup> Bug #1511822 changed: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <wily> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1511822>
  <mup> Bug #1511822 opened: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <wily> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1511822>
  <mup> Bug #1511865 opened: juju run output is unusable <juju-core:New> <https://launchpad.net/bugs/1511865>
  <mup> Bug #1511865 changed: juju run output is unusable <juju-core:Invalid> <https://launchpad.net/bugs/1511865>
  #juju-dev 2015-10-31
  <mup> Bug #1511812 changed: 1.25.0 and 1.24.6 cannot bootstrap local environment on wily host <bug-squad> <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1511812>
  <mup> Bug #1511944 opened: CLI commands cannot take multiple arguments even when they are unambigious, requiring multiple calls instead <cli-ux> <juju-core:New> <https://launchpad.net/bugs/1511944>
  <mup> Bug #1511944 changed: CLI commands cannot take multiple arguments even when they are unambigious, requiring multiple calls instead <cli-ux> <juju-core:New> <https://launchpad.net/bugs/1511944>
  <mup> Bug #1511944 opened: CLI commands cannot take multiple arguments even when they are unambigious, requiring multiple calls instead <cli-ux> <juju-core:New> <https://launchpad.net/bugs/1511944>
  <mup> Bug #1511944 changed: CLI commands cannot take multiple arguments even when they are unambigious, requiring multiple calls instead <cli-ux> <juju-core:New> <https://launchpad.net/bugs/1511944>
  <mup> Bug #1511944 opened: CLI commands cannot take multiple arguments even when they are unambigious, requiring multiple calls instead <cli-ux> <juju-core:New> <https://launchpad.net/bugs/1511944>
  #juju-dev 2015-11-01
  <mup> Bug #1488245 changed: Recurring lxc issue: failed to retrieve the template to clone  <canonical-bootstack> <lxc> <juju-core:Expired> <https://launchpad.net/bugs/1488245>
  #juju-dev 2016-10-31
  <menn0> axw: review done
  <axw> menn0: thanks, I'll run up a 1.25 env and test on there before I land
  <menn0> axw: great, thanks
  <menn0> axw: thanks for the edits
  <axw> menn0: so 2.4 handles the "_id $in [...]" method well - it completed instantly. using Bulk isn't terrible though, it takes 3 seconds to remove 10000 docs
  <axw> menn0: so I'm inclined to just use Bulk for simplicity
  <menn0> axw: that SGTM
  <axw> menn0: np
  <axw> cool
  <menn0> axw: 3s for 10000 docs locally or on azure?
  <axw> menn0: on azure
  <axw> one last test with 2.0 on a fresh VM
  <menn0> axw: that seems like it should be fine to me
  <axw> menn0: cool. I'm just going to test on the exact same VM size, with mongo 3.x/wiredtiger without waiting for things to go pear shaped this time. should be a better comparison
  <axw> if it's the same I'll shipit
  <menn0> axw: +1
  <wallyworld> axw: when you have time, a small one, no rush https://github.com/juju/juju/pull/6516
  <axw> wallyworld: what is the "elsewhere" that tests the same functionality?
  <wallyworld> axw: test cases in testChangeRemoteApplications
  <axw> wallyworld: LGTM
  <wallyworld> ty
  <wallyworld> axw: this line https://github.com/juju/version/blob/master/version.go#L143 causes this bug https://bugs.launchpad.net/juju/+bug/1637079 . I think we should just remove that series check entirely from ParseBinary? The bugs happens when you have a juju agent running on a system without zesty in its distro info data and the agent tries to parse zesty tools
  <mup> Bug #1637079: Upgrade stuck due to out-of-date distro-info-data package <canonical-is> <eda> <juju:Triaged by alexis-bruemmer> <https://launchpad.net/bugs/1637079>
  <axw> wallyworld: SGTM, but does removing that get you all the way?
  <wallyworld> i think so, will check a bit more closely to try and be sure
  <axw> wallyworld: we should be changing the binary format to replace series with OS at some point. there's no need for separate agent binaries per series
  <wallyworld> axw: oh tell me about it. bootstrapping on aws is sooooooo because it uploads i think 3 copies of the tools tarballs, one for each of a few series. takes forever
  <wallyworld> maybe a 2.2 thing
  <axw> wallyworld: seriously? I thought we fixed that to just upload one, and have the controller explode it into the various series
  <axw> if we're uploading multiple times, I think we can fix that without changing the format at least
  <wallyworld> hmmm. maybe. i haven't checked in a while. maybe the tarball has grown so large it still take sages
  <axw> it does take a long time
  <wallyworld> on ADSL yeah it does :-(
  <wallyworld> axw: two small ones for whenever https://github.com/juju/utils/pull/247 https://github.com/juju/version/pull/4
  <wallyworld> axw: thanks for reviews. there's little that can be done for other usages of GetOSFromSeries() in juju - all cases require the OS to make a decision about something wrt cloud init and such. so i'll just update the deps
  <axw> okey dokey
  <wallyworld> damn, pulling in tip of utils breaks other apis, sigh
  <rogpeppe> anyone know ModelMachineInfo.Status is instance status or agent status?
  <rogpeppe> anyone around that may be able to do a review of a change to testing/checkers to ignore time zones when comparing times? https://github.com/juju/testing/pull/115
  <rogpeppe> dimitern: ^
  <dimitern> rogpeppe: looking
  <rogpeppe> dimitern: thanks!
  <rogpeppe> dimitern: are you all at the sprint this week?
  <dimitern> rogpeppe: thankfully, no :)
  <rogpeppe> dimitern: :)
  <rogpeppe> dimitern: just it seem very quiet here atm
  <dimitern> rogpeppe: LGTM, but I'm curious - why was that needed?
  <rogpeppe> dimitern: just a test failure due to two times being different only in zone (we were using UTC, bson unmarshals as local) and i got fed up. i've wanted to make this change for ages :)
  <dimitern> rogpeppe: quiet yes.. I'm tasked with making sure develop merges OK into staging though, so I haven't noticed :)
  <perrito666> morning all
  <zeestrat> Hey, when is 2.0.1 coming to the juju/stable PPA?
  <dimitern> mgz: ping
  <mgz> dimitern: yo
  <dimitern> mgz: hey, I've been watching the last few CI runs on develop
  <dimitern> mgz: any idea about the build errors?
  <mgz> the last few?
  <mgz> the earlier ones were from balloons changing the packaging branch
  <mgz> but those should be gone with the latest builds?
  <mgz> hm, something new
  <dimitern> latest one still has a few: http://reports.vapour.ws/releases/4540
  <mgz> also those results are misleading
  <mgz> hardly anything actually ran
  <mgz> there was talk about poking around with S3 on friday, I'll look and see what actually changed
  <dimitern> ok
  <dimitern> mgz: do you think we could merge develop into staging today, or we need a bless first?
  <mgz> ideally we get a bless, but of the last 10 runs most have failed due to non-juju code reasons
  <balloons> we do need a bless. I was hoping a PR would land so we'd get a testrun -- really nothing has landed?
  <balloons> ohh, no, it landed.. I see, weird
  <mgz> stuff has landed, we're stuck on the build/publish step
  <mgz> of xenial ppc64 build at least is blocking things
  <mgz> perhaps other stuff?
   * mgz goes to clean up ppc64el-slave
  <mgz> which stilson are you today...
  <mgz> hm, 05 isn't actually out of disk
  <mgz> dimitern: I deleting a bunch of things from that machine and am trying a rerun of the ppc64 build job
  <dimitern> mgz: +1
  <mgz> that should unblock various things, obviously we have some config issues as well
  <mgz> as the jobs that failed shouldn't run before publish I presume
  <SimonKLB> who's involved with the juju charm store development?
  <SimonKLB> i renamed a user to use it's username for a team instead, launchpad seem ok with it, but it's not possible to make the changes migrate to the juju charm store
  <SimonKLB> or at least im not sure how to do it
  <natefinch> rogpeppe: ^
  <mgz> ah, standup is a different hour
  <mgz> thanks clocks...
  <natefinch> haha
  <natefinch> I laugh, but ours changes next week
  <mgz> yeah, then standup time will move again
  <mgz> hm, no katco today?
  <mgz> she appeareth
  <mgz> dimitern: standup!
  <mgz> also... other people who are around?
  <katco> dimitern: macgreag1ir: voidspace: frobware: standup time
  <voidspace> katco: oh, screw - our clocks changed and I forgot that changed standup time
  <voidspace> katco: omw
  <katco> no worries!
  <katco> dooferlad: standup time
  <dimitern> katco: isn't it in an hour's time?
  <natefinch> nope, you guys had daylight saving end and we haven't yet
  <mgz> dimitern: clocks, they are a changin'
  <natefinch> next week it'll be back to the same time
  <natefinch> since we end our saving time next week
  <natefinch> huzzah
  <katco> dimitern: confusion abounds
  <dimitern> oh I see
  <dimitern> omw then
  <rogpeppe> SimonKLB: I'm not sure exactly what you mean by "i renamed a user to use it's username for a team instead,"
  <rick_h___> rogpeppe: in SSO you can change the user be but that blows up on the chsrmstore without manual intervention
  <rogpeppe> rick_h___: yeah, we discussed it by PM
  <rick_h___> Ah cool
  <rogpeppe> rick_h___: my question above was because i found the grammar hard to parse
  <SimonKLB> my bad \o
  <SimonKLB> rick_h___: would deleting the user be sufficient? because i only have some test-charms pushed
  <SimonKLB> do whatever is the easiest for you guys
  <voidspace> dimitern: so for maas 2, "not_networks" needs to be changed in gomaasapi rather than in juju
  <dimitern> voidspace: for maas 2.1 only IIRC - let me double check
  <voidspace> dimitern: according to bug 163919 it's maas 2.0
  <mup> Bug #163919: didyouknow plugin bug <Exaile:Invalid> <https://launchpad.net/bugs/163919>
  <voidspace> uhm
  <voidspace> bug 1636919
  <mup> Bug #1636919: MAAS machine selected with space in violation of constraint <ci> <jujuqa> <maas-provider> <networking> <juju:Triaged> <MAAS:Invalid> <https://launchpad.net/bugs/1636919>
  <voidspace> dimitern: ah, no - it's an api change as well
  <voidspace> dimitern: (for gomaasapi)
  <voidspace> dimitern: juju needs to pass a slice of strings not a string
  <dimitern> voidspace: according to http://maas.ubuntu.com/docs2.0/api.html#machines - not_networks is still acceptable for API 2.0
  <voidspace> dimitern: I've pinged mpontillo on #juju (canonical)
  <voidspace> dimitern: no response yet, but I'll check with him
  <voidspace> dimitern: what maas API version do you see with 2.1? I don't have 2.1 setup
  <dimitern> voidspace: I've checked the source, but as for 2.1 API docs -  http://maas.ubuntu.com/docs2.1/api.html#machines still mentions not_networks and not "not_subnets"
  <dimitern> voidspace: I guess the docs were not updated recently
  <voidspace> dimitern: well, if they're incorrect for 2.1 they could be incorrect for 2.0 as well...
  <mgz> dimitern: I filed bugs
  <voidspace> dimitern: I'll find out
  <voidspace> mgz: the maas guys are at the sprint - as far as you know is "not_subnets" correct for 2.0 or only for 2.1?
  <mgz> for both
  <dimitern> voidspace: If all else fails, use the source :)
  <voidspace> dimitern: so you're saying that according to the source, "not_networks" is correct for 2.0?
  <voidspace> we don't officially support 2.1 yet anyway ;-)
  <mgz> bug 1637009 probably what you're most interested in
  <mup> Bug #1637009: [2.0,2.1] Node acquisition constraints API documentation needs to be updated to match reality <MAAS:In Progress by mpontillo> <MAAS 1.9:In Progress by mpontillo> <MAAS 2.0:In Progress by mpontillo> <https://launchpad.net/bugs/1637009>
   * voidspace goes to checkout maas
  <dimitern> voidspace, yeah - I can confirm on the actual 2.0 vmaas here..
  <voidspace> dimitern: ah, so that's in contradiction to the bug that mgz just linked to
  <mgz> bug 1637182 and bug 1637192 also relevent
  <mup> Bug #1637182: Help and documentation 'list of unicodes' inconsistent <MAAS:In Progress by mpontillo> <MAAS 1.9:In Progress by mpontillo> <MAAS 2.0:In Progress by mpontillo> <https://launchpad.net/bugs/1637182>
  <mup> Bug #1637192: [2.0,2.1] Allocate using subnets or not_subnets with space fails <MAAS:Triaged by mpontillo> <MAAS 1.9:Triaged by mpontillo> <MAAS 2.0:Triaged by mpontillo> <https://launchpad.net/bugs/1637192>
  <mgz> and bug 1636250
  <mup> Bug #1636250: [2.1] machines allocate API returns a random machine if incorrect - parameters are used <MAAS:Fix Committed by andreserl> <https://launchpad.net/bugs/1636250>
  <mgz> not_networks is accepted and silently ignored on 2.0 and 2.1
  <dimitern> mgz: not quite: http://paste.ubuntu.com/23407376/
  <dimitern> mgz: for 2.1 - http://paste.ubuntu.com/23407383/
  <dimitern> but I'll test it with the MAAS CLI manually on both
  <mgz> what am I looking for there dimitern?
  <mgz> I only see (wrong) docstring matches for your grep
  <mgz> also, bugs document the pretty extensive testing I did for this across three maas versions
  <dimitern> mgz: I've tried `$ maas 21-root machines allocate dry_run=True not_subnets=space:0 verbose=True` on 2.1 and got `unorderable types: Subnet() < Subnet()`
  <dimitern> with not_networks there's no error, but it doesn't seem to work
  <mgz> dimitern: see bug 1637192 as linked above
  <mup> Bug #1637192: [2.0,2.1] Allocate using subnets or not_subnets with space fails <MAAS:Triaged by mpontillo> <MAAS 1.9:Triaged by mpontillo> <MAAS 2.0:Triaged by mpontillo> <https://launchpad.net/bugs/1637192>
  <mgz> there's a bit more background to that that's on a maas mailing list
  <mgz> but I think the bug has the conclusion at least
  <mpontillo> dimitern: the 'unorderable types' one is a bug in the error message. what we're trying to do is render a string that says something like "failed to match constraint: xyz", but when we go to figure out what to put in xyz we crash ;-)
  <mpontillo> dimitern: https://code.launchpad.net/~mpontillo/maas/update-constraints-docs--bugs-1637182-1637009/+merge/309617 clarifies the docs
  <dimitern> mpontillo: cheeers! :)
  <mpontillo> dimitern: but yeah, I can confirm that going from 1.x to 2.x we renamed the '[not_]networks' constraint to '[not_]subnets'. MAAS 2.1 was also recently changed to reject unknown constraints.
  <dimitern> mpontillo: so *both* 2.0 and 2.1 API only accept "not_subnets" (even if it doesn't complain in the former case) ?
  <mpontillo> dimitern: correct, when we moved to MAAS 2.x we made that change. we also dropped legacy 'connected_to' and 'not_connected_to' constraints in 2.x
  <dimitern> mpontillo: ok, thanks for confirming!
  <mpontillo> dimitern: voidspace: np. also, be aware when testing this that the corresponding *positive* constraints will probably not do what you expect. that is, 'subnets' and 'not_subnets' do not filter an inverse set of machines if multiple subnets match the specifier.
  <mpontillo> (long story short, only matching nodes *not* in a set of subnets is well-supported, if you try to pass in, for example, subnets:space:x and space x has 3 subnets, we'll try to find a node with all 3, which is probably not useful)
  <mpontillo> but that is more of a wishlist because this works as-expected when using the interfaces constraint.
  <voidspace> mpontillo: ok, thanks
  <redir> rick_h___: yt?
  <perrito666> voidspace: still working?
  <voidspace> perrito666: still around anyway
  <perrito666> voidspace: I am trying to determine an issue that is most likely unrelated to this but am getting a constant spam error because juju cant  SetLinkLayerDevices
  <perrito666> voidspace: how big of an issue is that error?
  <voidspace> perrito666: ah, I've seen that - I think it's spurious
  <perrito666> voidspace: apparently Its trying to use the same id for more than one thing
  <perrito666> voidspace: ack, I thought so
  <voidspace> perrito666: dimitern is really the one who needs to answer I'm afraid
  <perrito666> yep, blame said the same but he is not here :p
  <voidspace> perrito666: but I saw the same error when I was trying to diagnose a vsphere issue - and it had nothing to do with that error
  <perrito666> voidspace: yeah, same here
  <perrito666> :)
  <perrito666> spaces are blowing all over the place in vsphere
  <voidspace> perrito666: yeah, that needs fixing - it shouldn't error out if spaces aren't supported
  <voidspace> perrito666: if you file a bug I'll fix it ;-)
  <perrito666> voidspace: I dont have enough information, I believe this is actually causing a problem for me here
  <natefinch> perrito666: what auth types does vsphere allow?
  <natefinch> perrito666: just userpass?
  <natefinch> anyone else know?  Would be nice if our docs specified this :/
  <natefinch> looks like just userpass from the credential s chema
  <voidspace> natefinch: perrito666: I've only ever *seen* userpass...
  <voidspace> perrito666: actually, the specific issue I saw was "unrecognised type" for an interface, not id re-use
  <voidspace> perrito666: so that is "new"
  <babbageclunk> morning everyone! I'm in Hamilton!
  <perrito666> natefinch: I have the same experience as voidspace
  <perrito666> voidspace: I learned it was due to id reuse after digging into the code the error is completely misleading
  <redir> babbageclunk: congrats
  <veebers> babbageclunk: nice! Where is your final destination in NZ?
  <voidspace> perrito666: ah, ok
  <babbageclunk> veebers: uh, a grave somewhere I guess? ;)
  <babbageclunk> I mean, Welly.
  <babbageclunk> But we'll have a longish boringish interlude in Palmerston North while all our stuff is being shipped.
  <veebers> babbageclunk: wow dark. I love it
  <babbageclunk> veebers: I mean, Palmy's not that bad.
  <veebers> ^_^
  <perrito666> hey all, how do I pass availability zone to bootstrap?
  <mup> Bug #1637695 changed: Charm-guide: Openstack on LXD directions do not work <juju:New> <https://launchpad.net/bugs/1637695>
  <perrito666> Alexis wallyworld I'll be a few mins late to the standup sorry
  <wallyworld> np
  #juju-dev 2016-11-01
  <redir> how do I kill an app in error state?
  <anastasiamac> redir: do u want to resolve the error or remove the app?
  <redir> remove
  <redir> anastasiamac: ^
  <anastasiamac> redir: juju remove-application?
  <redir> yeah that doesn't get rid of it
  <anastasiamac> redir: what does it say?
  <redir> remove-app says nothing anastasiamac
  <redir> just returns
  <redir> but the app (ubuntu) is still there in an error state and the machine is still there too
  <anastasiamac> redir: do u know what's error? anything in the logs?
  <redir> ERROR juju.worker.uniter.operation hook "install" failed: exit status 127
  <redir> but is seems to keep renewing leadership lease
  <anastasiamac> redir: in the past, we'd restart jujud on the machine to make sure that leadership gets kicked too... mayb it'll help here...
  <anastasiamac> redir: i think it's worth a bug tho with repro steps...
  <redir> nope nope, not killing any machines on this maas since it is sooo hard to get the nodes up and running.
  <redir> and there's no one there to wiggle the wires for 2 weeks
  <redir> all I did was `juju deploy ubuntu --series precise --to kvm:0
  <anastasiamac> redir: not killing machine... just re-starting jujud..
  <axw> menn0: the logs I had from IS had messages for transactions we were running. they were all for lease & status. there were just a lot more entities than I thought there were
  <axw> menn0: so 9000 txn over 2 hours isn't unusual in their env
  <redir> trick-or-treaters are starting
   * redir eods
  <axw> still not not sure why there was a spike (previous period had only ~1000)
  <menn0> axw: hmm interesting
  <babbageclunk> wallyworld: I think my change would be simpler if I ripped out the blank model uuid from the api first.
  <wallyworld> babbageclunk: i din't really have enough context to comment
  <wallyworld> model uuid in 2.0 should not be blank though i don't think
  <babbageclunk> wallyworld: https://github.com/juju/juju/blob/staging/apiserver/utils.go#L47
  <babbageclunk> wallyworld: Where do I need to check? I guess all of the clients?
  <wallyworld> babbageclunk: from memory, i tried taking that check out recently and stuff failed. i can't recall exactly what though
  <wallyworld> i think right now there's still a legitimate use case for empty model uuid in that context
  <wallyworld> backup might have been one case
  <babbageclunk> wallyworld: Ah, ok - I'll leave it alone then.
  <wallyworld> i was hoping we could remove that check
  <wallyworld> it may be that there's just legacy code to fix, but am not sure now
  <axw> wallyworld menn0: monitoring is paying dividends... just found a mongo session leak
  <wallyworld> yay!
  <wallyworld> axw: where?
  <axw> wallyworld: apiserver/logsink. we've got a loop that never breaks as long as the apiserver is running
  <axw> wallyworld: so the session it copies at the start of the handler never gets closed
  <axw> wallyworld: client reconnects and starts a new one when the client conn breaks, and so we have a slow buildup of goroutines & sessions
  <wallyworld> how often does  aclient conn break?
  <wallyworld> you mean each new incoming api request?
  <axw> wallyworld: well in my env, there's been 4 new goroutines added for this in the past 90 minutes. pretty slow buildup
  <axw> wallyworld: it's not one per API request, this is the logsink thing. it's a long-lived websocket
  <wallyworld> makes sense
  <natefinch> anyone know how vsphere configuration is supposed to work?
  <natefinch> I can;t tell if my the code has a weird bug or if it's doing what it's supposed to do
  <wallyworld> nfi
  <wallyworld> horatio is your man
  <natefinch> Horatio; a fellow of infinite jest, of most excellent fancy
  <natefinch> technically the second part is talking about yorick, but it's funnier to take it out of context
  <menn0> axw: nice work. hooray for monitoring!
  <blahdeblah> natefinch: context, schmontext
  <blahdeblah> natefinch: I *may* know something about vsphere configuration also
  <natefinch> blahdeblah: cool, well, I think I figured it out, since I see code explicitly doing what I thought might be a bug.  In fact, there's like 100 lines of reflection to do it, for some odd and scary reason
  <blahdeblah> Well, in that case, I know nothing about vsphere :-)
  <natefinch> lol
  <natefinch> blahdeblah: thanks anyway :)  What it was, was that the docs say to set your yaml like the example here: https://jujucharms.com/docs/stable/help-vmware
  <natefinch> where the regions are just empty... but then when it gets saved into your clouds.yaml, the regions get endpoint set with the value from the cloud endpoint... which seemed like a very weird bug
  <natefinch> but looking at the code, it's intentional.... just seems weird to manually copy them out when they're saved rather than simply interpret them that way at runtime.
  <anastasiamac> axw: this is awesome.. i don't think we had logsink in 1.25... did we?
  <axw> anastasiamac: I think we did, I'll check
  <blahdeblah> anastasiamac: Are you thinking what I'm thinking? :-)
  <anastasiamac> axw: oooh.. it'd b great if we have a fix to put it before 1.25.7 goes out then :D
  <anastasiamac> blahdeblah: :D
  <anastasiamac> blahdeblah: always!!
  <blahdeblah> anastasiamac: you are definitely thinking what I'm thinking. :-)
  <axw> anastasiamac: it's there, but the code is different. I don't think the bug applies to 1.25
  <blahdeblah> d'oh!
  <wallyworld> axw: interesting. os.Mkdir("/some/path", 0660) only creates the directory with 0640 and there's no error
  <wallyworld> at least on my yakkety system
  <wallyworld> so unit tests fail \o/
  <axw> wallyworld: umask
  <wallyworld> ah
  <axw> wallyworld: try clearing umask and running again
  <wallyworld> ta
  <wallyworld> forgot about that
  <wallyworld> yep, that was it
  <axw> wallyworld: I guess IsolationSuite should clear it?
  <wallyworld> yeah, i think so
  <wallyworld> except not every suite embeds that sadly
  <wallyworld> the one that failed for me doesn't
  <anastasiamac> wallyworld: looks like awesome drive-by improvement :D
  <wallyworld> fix every test suite?
  <wallyworld> there's a lot of them
  <anastasiamac> if u r really keen :D but i was talking about the test (or its suite) that was brought up to u attention
  <wallyworld> it embeds BaseSuite. I'd rather approach the problem holistically
  <wallyworld> look at making BaseSuite embed IsolationSuite
  <wallyworld> but there will be consequences
  <anastasiamac> i would have prefered for this suite to embed IsolationSuite rather than Base... It looks like it needs it
  <wallyworld> piecemeal embedding IsolationSuite here and there isn't the best approach
  <wallyworld> every suite should embedd BaseSuite
  <anastasiamac> no it' not... but it gets us to a better place
  <wallyworld> that gives standard log captures, cleanup etc
  <natefinch> what does basesuite do that isolation doesn't?
  <anastasiamac> i think basesuite is more limited
  <anastasiamac> isolationsuite should b used where file systems r touched... off memor...
  <wallyworld> JujuOSEnvSuite vs OSEnvSuite
  <natefinch> if you need logging to figure out why your tests failed, it means your tests are bad
  <wallyworld> i'm not talking about that
  <wallyworld> log capture prevents logginf leaking onto the console
  <natefinch> ahh
  <anastasiamac> fun regression in tests \o/ https://bugs.launchpad.net/juju/+bug/1466520
  <mup> Bug #1466520: serviceSuite setup fails <ci> <intermittent-failure> <unit-tests> <juju:Triaged> <juju-core:Fix Released> <juju-core 1.24:Won't Fix> <juju-core 1.25:Fix Released> <https://launchpad.net/bugs/1466520>
  <mup> Bug #1466520 changed: serviceSuite setup fails <ci> <intermittent-failure> <unit-tests> <juju:Triaged> <juju-core:Fix Released> <juju-core 1.24:Won't Fix> <juju-core 1.25:Fix Released> <https://launchpad.net/bugs/1466520>
  <mup> Bug #1514922 changed: Deploying to maas ppc64le with proxies kills websocket <charmers> <juju-core:Invalid> <https://launchpad.net/bugs/1514922>
  <natefinch> if anyone is feeling ambitious, interactive add-cloud:  https://github.com/juju/juju/pull/6498   +1,670 â58
  <wallyworld> axw: i think we should move the ec2 withDefaultControllerConstraints() method and generically apply it to all providers. i can't see a reason why we would not want to ensure a minimum memory spec in all cases
  <axw> wallyworld: we already do have a min memory of 1G for all instances, see environs/instances/instancetype.go. but perhaps we should do something controller-specific
  <wallyworld> axw: yeah, that's my point. 1G is too small for controllers, eg azure gets 1.5GB. aws gets 3.75 which is what we'd want for all providers
  <axw> wallyworld: sure, sounds like a good idea. I think we'll need tests for each provider to check which instance types get used by default tho
  <wallyworld> otherwise we allow people to bootstrap with undersized controllers where steady state is say 2GB and it thrashes/pages all the time
  <wallyworld> do we care what instance types get used?
  <wallyworld> so long as the capabilities are met?
  <wallyworld> and for openstack say, we don't know ahead of time that the instance types are called
  <axw> wallyworld: IIANM, if you set a min of 3.75G on azure you get something with ~7GB
  <axw> there's one with slightly less than 3.75GB that would probably be appropriate
  <wallyworld> ok, i'll look at adding some for of sanity check or something
  <axw> wallyworld: for openstack we wouldn't do it, I'm just saying as a general sort of rule - to make sure we don't go costing people more than necessary
  <wallyworld> yep, will add it to azure etc
  <wallyworld> anastasiamac: you able to look at https://github.com/juju/juju/pull/6518 for me?
  <anastasiamac> wallyworld: that's the really big one :D
  <wallyworld> not really
  <wallyworld> a few files, but minimal changes
  <anastasiamac> wallyworld: i'll look in a sec.. i did see it coming in earlier and was posptponing the pleasure
  <wallyworld> only +192 :-)
  <wallyworld> -156
  <anastasiamac> yep agaisnt 66 files... gimme asec
  <wallyworld> most straight search and replace
  <wallyworld> would be far easier in review board :-(
  <anastasiamac> like grep? I love it when u use it :D
  <wallyworld> since yo'd have a paginated list of files
  <wallyworld> anastasiamac: i tried to explain the the PR description why Must was called. all tests use Must - that's our standard practice. the other places used Host except where it was not possible  eg existing method did not return error.
  <wallyworld> there's no one explanation
  <wallyworld> and the WIP PR will oblitarate most of this anyway
  <wallyworld> and the stuff this PR replaced all used to panic
  <wallyworld> it's just that the method in utils was incorrectly named
  <wallyworld> so all this core PR does is adapt to the new method name essentially
  <anastasiamac> agent/agent.go is nt a test and it only has .Must calls....
  <anastasiamac> in var initialisations.. if we can panic, shouldn;t it be handled differently
  <anastasiamac> ?
  <wallyworld> right, as i said in the description, for var initialisation it is needed
  <anastasiamac> the same question for api/certpool?...
  <anastasiamac> right but that means thatw e can still panic
  <wallyworld> sure, that's what it always ever did
  <wallyworld> the old HostSeries used to panic
  <anastasiamac> so if we paniced before and will panic now, what's the improvement?
  <anastasiamac> wallyworld: i'd appreciate if u got a second opinion :D
  <wallyworld> the improvement is not is this host series stuff - i am forced to adapt to use the new apis because i committed changes to tip of utils repo
  <wallyworld> work has landed there that core does not use properly let
  <wallyworld> there's a WIP PR for that
  <wallyworld> this just adapts core to use the apis from tip of utils
  <wallyworld> i have landed changes elsewhere in utils that are needed
  <wallyworld> this PR is just method renames, no bheaviour changes
  <wallyworld> except for the dep updates which do bring in desired behaviour
  <anastasiamac> wallyworld: like i said - i don;t have issues with code but m not understanding the value of the change... revving up the dependency is fine... but the change in util is the one that we are adapting to here...
  <wallyworld> but the change it not the HostSeries stuff
  <wallyworld> that is a byproduct because other wor landed in utils
  <wallyworld> my chage is to the internals of utils elsewhere
  <anastasiamac> so maybe do a depenedncy rev up as a separate PR?
  <wallyworld> i can't - tip of utils has changed
  <wallyworld> this PR *is* the dependency change
  <wallyworld> with adaoptions for the altered apis
  <anastasiamac> k... hangon m going to call royalty.. a sec
  <wallyworld> which we are now forced to consume
  <babbageclunk> how anti goto are we?
  <babbageclunk> not at all? great!
  <anastasiamac> babbageclunk: :) i thought i saw it in our codebase...
  <mup> Bug # changed: 1484105, 1506460, 1515475, 1545562, 1592609
  <babbageclunk> anastasiamac: Was only half a joke - the error flow in the code I'm working in is a bit unusual, I think goto would probably work a lot better than what I've got now. If it's not going to give people conniptions I'll try it out.
  <anastasiamac> babbageclunk: \o/
  <axw> anastasiamac: "Once the checks succeed, LGTM \o/"  <- why is that a prereq? the checks are a subset of a merge run?
  <axw> (AFAICT the checks failed because of an interimittent failure)
  <anastasiamac> axw: failed checks sometimes indicate test failures :D
  <anastasiamac> if it's intermittent... would b nice to fix but we have landed code with intermittency before.. o m leaving at ur discretion
  <anastasiamac> axw: hence, approved and lgtm'ed :D
  <axw> anastasiamac: okey dokey
  <axw> anastasiamac: thanks for the review
  <babbageclunk> yay, I think that code's much better with goto!
  <anastasiamac> axw: ur PR is merged \o/ awesome!!
  <axw> anastasiamac: :)  just upgraded my azure env, will see how it fares now
  <anastasiamac> axw: m holding my breath :D
  <axw> anastasiamac: please see my reply on https://github.com/juju/juju/pull/6520
  <axw> anastasiamac: happy for me to land now?
  <kjackal> hey everyone, is there a rest api for juju? Where can I find documentation for it?
  <hoenir> hi , could anyone explain to me why I'm getting this log messages, tring to set some custom tools on bootstrap but I'm gettings this logs
  <hoenir> http://paste.ubuntu.com/23410800/
  <hoenir> Some thoughts on this? Anyone?
  <hoenir> anyone?
  <mgz> hoenir: it's telling you your data doesn't have everything it needs
  <hoenir> mgz, so please direct me where to read on "what it needs"
  <mgz> now that is a much harder question :)
  <mgz> I think it's just that the final url for the tools results in a 404? so you need to fix/change that location
  <mgz> turning on trace logging/looking at the actual requests made may help
  <mgz> and I'd just refer to the code in environs/simplestreams for what's actually happening
  <mgz> which is pretty confusing as it's several layers of support across change versions
  <mgz> I'm going to be out from standup, summary, I have the minimal ssh fix done, looking at a more comprehesive one
  <mgz> as dash with no prompt and no control codes is still pretty annoying to work over
  <anastasiamac> voidspace: ping :D
  <voidspace> anastasiamac: pong
  <anastasiamac> voidspace: sorry, pm-ing u :D
  <perrito666> anastasiamac: do you not sleep?
  <perrito666> rick_h___: you bailed on me yesterday, I was too busy to rant at you then :p
  <anastasiamac> perrito666: not on irc :D
  <mup> Bug #1333162 opened: cannot bootstrap on openstack provider:  Multiple security_group matches found for name 'XYZ', use an ID to be more specific. <bug-squad> <charmers>
  <mup> <openstack-provider> <repeatability> <security> <uosci> <juju:Incomplete> <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1333162>
  <deanman> Any ideas how i could debug the following error 'agent is not communicating with the server' when deploying a charm using LXD ?
  <rick_h___> kjackal: https://godoc.org/github.com/juju/juju/api
  <kjackal> thanks rick_h___
  <kjackal> rick_h___: This does not look like a rest api
  <rick_h___> kjackal: it's a websocket API for the most part. There's some restful bits, but for the most part it's async with events pushed across the websocket.
  <kjackal> rick_h___: I see. thanks
  <rick_h___> kjackal: best thing might be to look at clients like the WIP libjuju https://github.com/juju/python-libjuju
  <rick_h___> kjackal: not sure what you're looking to do/etc
  <kjackal> rick_h___: I saw python-libjuju but its WIP. We have a client that wants to use its own admin console to drive juju deployments. I am exploring what would be the right way to do that
  <kjackal> rick_h___: any suggestions?
  <rick_h___> kjackal: in python or another language?
  <kjackal> Not sure about the language, I would guess python. Thats why a REST api would seem a good starting point. Do we have a reference example?
  <kjackal> rick_h___: ^
  <kjackal> rick_h___: the Juju gui on juju 1 was doing something similar (drive juju) not sure how that worked
  <katco> frobware: mgz: dooferlad: standup time?
  <voidspace> katco: just to confirm, new developer workflow, for a branch forked off staging I propose it against develop. Right?
  <katco> voidspace: correct, unless it's a fix for a revision-release (e.g. 2.0.2), then you fork the 2.0 branch and propose both there and to develop
  <voidspace> katco: ok, I thought we were doing backports - i.e. propose against develop then cherry-pick to propose against 2.0
  <katco> voidspace: i believe it's the other way around? didn't we have this conversation in a standup and rick said, "no it's more like a forward-port"?
  <voidspace> katco: we have had this conversation and my memory (which I wouldn't rely on) was that we concluded back-port... :-)
  <voidspace> katco: this one is easy, so it doesn't matter anyway
  <katco> voidspace: lol y3ep
  <mgz> voidspace: do you need review on your maas constraint changes still?
  <katco> balloons: any idea why the bot isn't getting triggered for this? https://github.com/juju/juju/pull/6414
  <hackedbellini> Hi guys! I'm using juju 2.0.1.1 here. I have the landscape-charm deployed on some machines as a subordinate charm. I removed it from one of them, but the unit still shows on "juju status". It was really removed since it doesn't appear on juju-gui. Also trying to "juju remove-unit landscape-client/3" gives me a "ERROR no units were destroyed: unit
  <hackedbellini> "landscape-client/3" is a subordinate". How can I get rid of it?
  <natefinch> you can't remove subordinates without removing the parent unit
  <natefinch> the thought is that many subordinates fundamentally change how the parent functions, and removing them could break the parent.
  <natefinch> I don't entirely agree with outright preventing people from doing it, but that's the way it works right now.
  <katco> sinzui: mgz: any ideas re. my question to balloons above?
  <mgz> is hoenir a public member of the juju devs group?
  <hackedbellini> natefinch: I see. But I removed it before, it just didn't vanish from "juju status". Because he is in an "inconsistent" state now it is on "maintainance" workload saying that it "Need computer-title and juju-info to proceed" on the message
  <mgz> seems not. so, you can either land for him, or we can sign him up.
  <katco> mgz: he's in the group. looks like he just needs to make it public
  <hackedbellini> it don't even shows on juju-gui anymore
  <katco> hoenir: can you please make your juju team status public?
  <natefinch> hackedbellini: I don't know what the GUI does, but if it's in juju-status, it exists
  <hoenir> katco, yea sure, let me search what button should I press in order to do that.
  <katco> hoenir: i always forget myself :)
  <mgz> hoenir: go to <https://github.com/orgs/juju/people?&query=hoenir>
  <katco> hoenir: after you do that, try the merge one more time
  <mgz> and mark your membership as public
  <hoenir> yeah mgz thanks ! I'm now Public
  <hackedbellini> natefinch: ahh, I remember what I did. I removed the relation that existed between the subordinate charm and the charm itself. The relation doesn't exists anymore (the reason why I can't see it on juju-gui) but the unit still shows on status
  <natefinch> ahh
  <mgz> okay, don't need to do anything else, bot will believe your comment next run
  <natefinch> hmm interesting
  <hoenir> yohooo katco it worked !
  <hackedbellini> natefinch: so the unit that is still there is useless. It doesn't do anything without the relation. That is the reason that I want to remove it
  <hackedbellini> it is just polluting my status =P
  <katco> hoenir: yay! i'm really sorry, i think you've been struggling with that for awhile haven't you?
  <natefinch> hackedbellini: can you kill the parent unit and replace it?
  <hoenir> katco, nah it was a fun experience, and thanks a lot for taking time to review my PR.
  <hackedbellini> natefinch: hrm, I'd rather not have to do that if possible
  <katco> hoenir: no worries, hth. thanks for the pr!
  <natefinch> hackedbellini: it sounds like a bug... if there's no relationship, then it shouldn't be treated like a subordinate... it's probably one of those corner cases we just hadn't really spent much time on
  <hackedbellini> natefinch: hrm that explains it. Is there any way to hack it so I can "force" the unit deletion?
  <natefinch> hackedbellini: not that I know of, let me check the code to see if there's a backdoor we can try
  <natefinch> hackedbellini: unfortunately, the code that determines whether it's a subordinate or not just reads a field in mongo, so to change it, you'd have to twiddle with mongo
  <hackedbellini> natefinch: hrm I don't have any experience with nosql databases (if it was postgresql I would try to hack it myself =P), but if you tell me what should I do there I can try it!
  <voidspace> mgz: I do, but I want dimitern to QA it as well
  <mgz> voidspace: I don't see from the diff where the 1.9/2.0+ split is
  <voidspace> mgz: are you looking at the gomaasapi PR?
  <voidspace> mgz: there's two branches - a gomaasapi that addresses both bugs for 2.0
  <mgz> voidspace: yeah, I'm not quite sure where the bits are going to fit together
  <voidspace> mgz: the gomaasapi fix is 2.0 only
  <voidspace> mgz: separate branch for 1.9 (that code is in juju - about to turn it into a PR)
  <mgz> voidspace: gotcha
  <voidspace> rick_h___:  katco: dimitern has a public holiday today which is why he wasn't at standup
  <katco> voidspace: yeah just realized that a few minutes ago :)
  <natefinch> hackedbellini: sorry, had to step out, missed your message before I left. I don't really recommend triddling with the DB.  My recommendation is to create a new unit of the parent on a different machine and kill this machine. Sorry, I think that's the best you can do.
  <hackedbellini> natefinch: yeah I did that =P. Thaks for your time anyway! :)
  <natefinch> hackedbellini: file a bug, if you haven't.  I think it's worth tracking.  Might also convince people that being able to remove a subordinate is a good idea
  <balloons> katco, sorry I never saw your pings :-(
  <balloons> katco, I will look now and see if I still have the logs on what it saw the PR as
  <katco> balloons: no worries. we got it figured out
  <balloons> katco, ohh? Why didn't it trigger?
  <katco> balloons: his juju membership wasn't public
  <balloons> ahh. I wonder if dmitern has the same problem.. menno as well
  <rogpeppe1> i've been trying to start an local lxd controller, and it's taking ages (it's been about 30 minutes so far mostly running apt-get update, upgrade and install). anyone got an idea what might be my issue?
  <rogpeppe1> just the "Installing curl" step has taken about 15 minutes so far
  <rogpeppe1> (it's still running)
  <rogpeppe1> i just bootstrapped an aws controller easily within the time that step has been taking
  <rogpeppe1> ah, seems like it's trying to connect via IPv6
  <rogpeppe1> E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/cpu-checker/cpu-checker_0.7-0ubuntu7_amd64.deb  Unable to connect to archive.ubuntu.com:http: [IP: 2001:67c:1560:8001::11 80]
  <rogpeppe1> sigh
  <natefinch> anyone have time for a big review?  Trying to get this reviewed in the next day or two so it can be mergeed by EOW: https://github.com/juju/juju/pull/6498
  <natefinch> 1600 lines, but almost all new code
  <babbageclunk> hey menn0, writing some more tests - what do you think should happen if a machine agent tries to log in to a model that's been migrated and the model's been removed?
  <menn0> babbageclunk: error I think
  <menn0> hmm
  <menn0> babbageclunk: if the model has been removed then maybe it's ok to redirect
  <babbageclunk> ok
  <menn0> babbageclunk: but really, the situation shouldn't happen
  <menn0> babbageclunk: great care is taken during the migration to make sure that all the agents are talking to the new model
  <menn0> new controller
  <menn0> babbageclunk: what is the least complicated thing to do in the redirect code?
  <babbageclunk> menn0: Ok, thanks - redirect's probably simplest for me
  <menn0> babbageclunk: as long as there's no redirects while the model data is still there
  <menn0> babbageclunk: the migrationminion still needs to be able to connect back to the source controller post SUCCESS
  <alexisb> heya babbageclunk welcome to the cool TZ :)
  <menn0> (for a little while anyway)
  <babbageclunk> thanks alexisb!
  <babbageclunk> menn0: Yup, I've got a test for that.
  <menn0> babbageclunk: sweet
  <babbageclunk> menn0: Hmm, actually at the moment (with no changes) it gives a login error because the machine doesn't exist - ok to leave it like that? Since it's something that shouldn't happen anyway.
  <menn0> babbageclunk: yep that's fine
  <babbageclunk> menn0: woot, then it's on to some manual testing!
  <menn0> babbageclunk: awesome!
  <alexisb> team having some networking issues
  <alexisb> will be on the standup as soon as I can
  <alexisb> wallyworld, ^^
  <alexisb> wallyworld, menn0 please get started without me
  <wallyworld> ok
  #juju-dev 2016-11-02
  <alexisb> ok axw ready when you are
   * redir eods. may be back later.
  <blahdeblah> hi folks - anyone available to help troubleshoot what looks like a bug in user removal?  https://pastebin.canonical.com/169528/
  <wallyworld> blahdeblah: not sure about troubleshoot - looks like a straight bug. when users are removed, they are actually marked as inactive. looks there's there's some logic issues there
  <wallyworld> axw: remind me - for bootstrap, the majority of machine config is done via ssh; cloud init is used to get the ssh keys onto the machine I think. is that also the case for workload machine provisioning ie we do the bulk of setup via ssh and not cloud init?
  <axw> wallyworld: we don't use ssh at all for non-bootstrap machine provisioning
  <wallyworld> axw: ta, i thought that was the case but wasn't certain
  <axw> wallyworld: (apart from one or two providers which may use it internally)
  <natefinch> axw, menn0: do you want me to come into the tech board meeting to talk about the dependencies I emailed about, or are you good?
  <menn0> natefinch: it might be useful if you could talk about them when we get to that bit, but we can also manage without you if you're finishing up for the day
  <natefinch> menn0: I have some time
  <menn0> natefinch: cool. i'll ping you when we get to it (or join whenever you like)
  <natefinch> menn0: cool, thanks
  <menn0> natefinch: now?
  <natefinch> menn0: omw
  <mup> Bug #1254402 changed: error bootstrapping on ec2 duplicate security group <ec2-provider> <security> <juju-core:Expired> <https://launchpad.net/bugs/1254402>
  <mup> Bug #1456725 changed: TestActionEvents fails <ci> <juju-core:Expired> <juju-core 1.22:Won't Fix> <https://launchpad.net/bugs/1456725>
  <mup> Bug #1502139 changed: Panic in TestUpgradeJuju <ci> <panic> <juju-core:Expired> <https://launchpad.net/bugs/1502139>
  <redir> anastasiamac: yt?
  <anastasiamac> redir: i m. why r u? (was doing a school pick-up)...
  <wallyworld> anastasiamac: do you recall - i think --metadata-source for bootstrap was broken around beta15 timeframe and it needed to be fixed before 2.0GA. Right?
  <anastasiamac> wallyworld: sounds about right..
  <anastasiamac> wallyworld: why r u in beta15?
  <wallyworld> i'm not
  <dimitern> voidspace: ping
  <voidspace> dimitern: pong
  <dimitern> voidspace: so I've been testing your gomaasapi change: https://github.com/juju/gomaasapi/pull/62
  <dimitern> voidspace: it looks like it's doing the right thing (compared to the maas cli at least) with one or more negative spaces constraints
  <voidspace> dimitern: great
  <dimitern> voidspace: however, I can see that 'unorderable types Subnet() < Subnet()' in some cases
  <dimitern> voidspace: is there a bug for that in maas?
  <voidspace> dimitern: I've never seen that
  <voidspace> dimitern: is that an error from MAAS or in the gomaasapi code?
  <dimitern> voidspace: from maas - e.g. when trying `jujy bootstrap --no-gui --debug vmaas-21 --config=~/.j/conf-x-fast.yaml --constraints 'arch=amd64 spaces=^compute,^external'` - the config just sets trace logging, default-series: xenial, disables apt update/upgrade
  <voidspace> dimitern: ah, so definitely sounds like a MAAS bug
  <dimitern> voidspace: there's 1 machine that matches those spaces, and here's the error we get: http://paste.ubuntu.com/23415499/
  <dimitern> voidspace: yeah, IIRC mpontillo did mention something about that and they seem to be fixing it
  <voidspace> dimitern: cool, so I don't think we should hold up this PR with it - I haven't seen a MAAS bug for it
  <voidspace> dimitern: worth filing even if it turns out to be a duplicate
  <dimitern> voidspace: I'll look into filing one later today; just testing with 2.0 now - 2.1 tested OK
  <voidspace> dimitern: I have a very short PR fixing not_networks for 1.9 I'd like you to test too if possible
  <voidspace> dimitern: will ping in a couple of minutes
  <dimitern> voidspace: sure - can do - is it about the way we serialize multiple negatives?
  <voidspace> dimitern: yep
  <voidspace> dimitern: changing from comma separated to multiple parameters
  <dimitern> voidspace: ok
  <dimitern> mgz: ping
  <dimitern> mgz: do you know which mongo version are we using for the windows CI slave running this job: http://reports.vapour.ws/releases/4546/job/run-unit-tests-win2012-amd64/attempt/3002#highlight
  <dimitern> mgz: if it's 2.6.x, the test might be failing due to this mongo bug - https://jira.mongodb.org/browse/SERVER-14840
  <dimitern> voidspace: tested on 2.0 successfully and LGTM-ed
  <voidspace> dimitern: you rock
  <dimitern> voidspace: it took me a lot longer to setup the spaces / subnets on maas than to run bootstrap OK :)
  <voidspace> dimitern: I can imagine
  <voidspace> dimitern: my maas 2 is not playing ball and won't enlist new nodes
  <voidspace> burned a few hours on it :-/
  <voidspace> dimitern: might have to reinstall from scratch
  <voidspace> got inveigled in another conversation
  <voidspace> about to create the PR for maas 1.9
  <voidspace> and I'll do PRs to bump the gomaasapi dependency in juju
  <voidspace> both for 2.0 and develop branches
  <dimitern> voidspace: I'm now using frobware's kvm-maas-add|remove-node with great success to add nodes - see https://github.com/frobware/kvm-maas/pull/1
  <dimitern> it takes <5m to create, enlist, and commission 6 KVM nodes with multiple NICs
  <voidspace> dimitern: yeah, I've used that before - don't know why I didn't think of it.
  <voidspace> dimitern: will try it now
  <voidspace> dimitern: it uses the api so should work fine
  <voidspace> dimitern: I'll just delete my existing nodes that MAAS won't recognise and try again
  <dimitern> voidspace: it works quite well, but I'd suggest trying my modified version (re that PR), as it improves quite a few things :)
  <dimitern> voidspace: like not hard-coding 4 NICs per node but as many as you've passed as args
  <voidspace> dimitern: where's your version?
  <dimitern> voidspace: this branch essentially: https://github.com/dimitern/kvm-maas/tree/multiple-networks
  <voidspace> dimitern: cool, thanks
  <voidspace> dimitern: I only have two spaces/subnets defined in maas - but that's easy to change and enough to test this anyway
  <dimitern> voidspace: +1
   * dimitern steps out for ~1h
  <rogpeppe1> does anyone know if juju restore actually works?
  <mgz> dimitern: see bug 1637294 btw
  <mup> Bug #1637294: Windows host needs a current mongodb (3.2.x) <juju-ci-tools:Triaged> <https://launchpad.net/bugs/1637294>
  <dimitern> mgz: cheers!
   * dimitern is just getting out the door - bbiab hopefully in ~1h
  <SimonKLB> is aws efs volumes something that is on the horizon or will it be a while before network shared storage will be available in juju?
  <SimonKLB> it would be really great being able to deploy multiple units that share the same filesystem
  <voidspace> dimitern: sorry for the delay I got distracted: https://github.com/juju/juju/pull/6528
  <voidspace> dimitern: bugger, I need to retarget that to develop not staging
  <voidspace> dimitern: done
  <voidspace> unfortunately means pre-check failed
  <voidspace> all unit tests pass though
  <voidspace> dimitern: and https://github.com/juju/juju/pull/6529
  <voidspace> dimitern: gah, conflicts
  <voidspace> dimitern: rick_h___: so, when a PR forked off staging but targetted against develop has conflicts
  <voidspace> dimitern: rick_h___: I assume the correct approach is to merge develop into the PR and fix the conflicts?
  <voidspace> I have both "go fmt" and build failures (probably related to go version) with current develop
  <voidspace> so I've fixed both those as part of the conflict resolution and they'll now show up in my PR
  <voidspace> running all related tests
  <rick_h___> voidspace: the best thing is to cherry pick the conflict commit and resolve the conflicts
  <rick_h___> voidspace: imo
  <voidspace> rick_h___: so fork off develop and just make the new commit
  <voidspace> rick_h___: I think that has the same end result - and I still need to fix the two issues
  <voidspace> rick_h___: thanks
  <rick_h___> voidspace: so checkout develop, update it from upstream, go to your branch, and cherry pick the conflict from develop
  <voidspace> rick_h___: ah, I see
  <voidspace> rick_h___: that would be better, yes
  <voidspace> rick_h___: I'm still concerned that we have a "go fmt" error and a build error on develop!
  <voidspace> rick_h___: (the build error is in a test)
  <voidspace> rick_h___: and is likely to be go version related - but with my fix tests all pass
  <voidspace> rick_h___: this is now my PR (very short) https://github.com/juju/juju/pull/6529/files
  <voidspace> rick_h___: (self assignment to args.Host was the build error I saw)
  <voidspace> mgz: thanks for the magic incatation on my branch - I've repeated it
  <mgz> voidspace: I'm a little surprised it didn't listen to mine
  <mgz> voidspace: also, lgtm
  <voidspace> mgz: cool, thanks
  <voidspace> mgz: I'll wait for the check and for dimitern to QA
   * dimitern is back
  <rogpeppe1> a small names change to make it easier to work with empty cloud credentials: https://github.com/juju/names/pull/77
  <rogpeppe1> reviews appreciated: ^
  <dimitern> voidspace: looking at both of your PRs, and QAing the 1.9 one
  <dimitern> actually QAing both of them as it's trivial
  <voidspace> rick_h___: is our 1:1 going to be on?
  <voidspace> rick_h___: I have no burning questions, so happy to skip this week
  <mup> Bug #1638560 opened: 1.25 broken on MacOS sierra <osx> <juju-core:Triaged> <https://launchpad.net/bugs/1638560>
  <abentley> rogpeppe1: Hi.  I've been asked to do one of the things we talked about yesterday: test the latest juju against a released version of JIMM, to ensure juju doesn't break what JIMM uses.  Do you have any tips on how to go about that?
  <rick_h___> voidspace: skip please
  <voidspace> rick_h___: kk
  <dimitern> voidspace: both PRs LGTM
  <voidspace> dimitern: you rock, you really do
  <voidspace> dimitern: about to propose same PRs against 2.0
  <dimitern> voidspace: ;) cheers
  <katco> natefinch: dimitern: frobware: standup time
  <frobware> katco: skipping, at the cloud sprint.
  <katco> frobware: ah ok... ta!
  <rick_h___> dimitern: ping for call
  <mpontillo> dimitern: the 500 error happens in lieu of a "no machine matched constraints" error. positive space constraints should happen in the interfaces constraint; they will not work properly in the 'subnets' constraint.
  <mpontillo> voidspace: ^
  <voidspace> mpontillo: ok, interesting
   * dimitern well, it's been good 4 years... take good care ;)
  <natefinch> dammit
  <natefinch> mgz, voidspace, dooferlad: any of you know dimiter's home email?
  <natefinch> got it, thanks
  <katco> ....
  <katco> what just happened?
  <mgz> I do not know.
  <voidspace> :-(
  <natefinch> check your email
  <perrito666> natefinch: I believe that their reactions are mail related
  <natefinch> ahh, right
  <mgz> okay, I think I finally have a solution I might like with this ssh stuff
  <mgz> and I can do it in two somewhat sane prs
  <rick_h___> mgz: did you see jam's reply there?
  <rick_h___> mgz: he sent an email on the ssh notes a couple of days ago I think?
  <mgz> rick_h___: yeah, that sent me off down a rabbit hole
  <rick_h___> ah ok nice
  <mgz> which, to be fair, gets to a much better final result, but involves deeper magic
  <alexisb_> changing locations all, will be back online is a few minutes
  <mup> Bug # changed: 1541482, 1597601, 1626304, 1634289
  <rick_h___> heh, that was a drop off
  <katco> yeah
  <katco> i think the event ends when the person ends it, not when the time-delayed stream ends
  <katco> i've had that happen a few times now
  <natefinch> easy review anyone? https://github.com/juju/jsonschema/pull/2
  <natefinch> +19 -19
  <mgz> natefinch: I'm in reviewing mode while waiting for build so okay
  <mgz> natefinch: heh - er, are we going to use this for actual translations ever?
  <mgz> because if so singular/plural is wrong
  <natefinch> I'm not sure what you mean.  It's used for generating interactive UI
  <mgz> natefinch: https://www.gnu.org/savannah-checkouts/gnu/gettext/manual/html_node/Plural-forms.html
  <natefinch> ahh, the old "multiple plural forms"
  <mgz> if we only ever want this to do english it's fine
  <natefinch> you uh, let me know when we plan to translate Juju, and I'll get right on making this better :)
  <perrito666> well this is great, I only have decaf and I live too far from the coffee shop... I really need a second car or a bike safe city
  <natefinch> perrito666: I feel you, I ran out of coffee today too. luckily I have caffeine pills in the cupboard which let me function until I could get to a coffee shop
  <perrito666> natefinch: I live way closer to "civilization" than you do, I just have one car only which my wife takes to work so even though I live closer its not all that close
  <natefinch> mgz: anyway... yes, for now, I think it's sufficient.  At some point, if we want to localize juju, we'll need to update this.  But that's a much bigger job.  This is just so you can take jsonschema and produce reasonable english-looking queries out of it
  <mgz> natefinch: code changes approved.
  <natefinch> mgz: thanks :)
  <mgz> perrito666: I don't really believe that you can't use the internet to cause a man on a scooter to bring you coffee
  <perrito666> mgz: never been to south america eh?
  <mgz> perrito666: I simply trust your ingenuity
  <perrito666> well I guess I could kidnap one of mi neighbours and ask their family to bring me coffee but its a bit excessive
  <mgz> :D
  <natefinch> lol
  <katco> dooferlad: hey are you still around?
  <alexisb> perrito666, nice job on the quick turnaround on the backup/restore test failure
  <perrito666> thank me when CI passes, jenkins does not always agree with me ;)
  <perrito666> anyway, ill step out for a moment
  <redir> natefinch: yt?
  <menn0> wallyworld: unit tests appear to failing on non-ubuntu test hosts and it could be a recent change by you
  <wallyworld> damn ok, will look
  <menn0> wallyworld: http://reports.vapour.ws/releases/4551/job/run-unit-tests-win2012-amd64/attempt/3007#highlight
  <wallyworld> how did it land i wonder
  <wallyworld> i have meeting for the next hour will look after
  <menn0> wallyworld: b/c the windows tests for merge only seem to test packages under cmd/
  <menn0> which is a bit shit
  <wallyworld> well that sucks
  <perrito666> menn0: that sucks a lot
  <perrito666> and by that I dont mean wallyworld but the limited scope of windows tests
  <wallyworld> oh i suck too
  <wallyworld> menn0: luckily looks like it will just be a skip
  <menn0> wallyworld: sweet
  <babbageclunk> menn0: To do a migration, do I need to have bootstrapped both controllers with the feature flag?
  <menn0> yep
  <babbageclunk> that would explain it then!
  <menn0> I have this in my shell config:
  <babbageclunk> okthx
  <menn0> export JUJU_DEV_FEATURE_FLAGS=migration,developer-mode
  <babbageclunk> cool - what's developer-mode?
  <menn0> it enables commands like dump-db and dump-model
  <menn0> I *think* it also enables the introspection worker
  <menn0> or maybe that runs all the time, not sure
  <perrito666> babbageclunk: the world becomes pixelated around you and you hear 8 bit music, a bit like what happens in tron :p
  <babbageclunk> perrito666: so it's like DMT?
  <perrito666> wouldnt know :) but I would adventure things are a bit more nerdy
  <perrito666> like this https://www.youtube.com/watch?v=KEkrWRHCDQU
  <babbageclunk> I'm pretty sure that's how I learned to program!
  <perrito666> babbageclunk: to be more accurate https://www.youtube.com/watch?v=KEkrWRHCDQU&feature=youtu.be&t=134
  <babbageclunk> perrito666: I had to have a one minute cad chill-out to recover https://youtu.be/mNTj05I4k5I?t=6m9s
  #juju-dev 2016-11-03
  <alexisb> anastasiamac, I will be right there
  <anastasiamac> alexisb: me 2
  <perrito666> alexisb: ill grab a bite and be back is that ok?
  <alexisb> yes perrito666
  <babbageclunk> anyone have a script for quickly opening a mongo shell on a controller?
  <veebers> babbageclunk: I think menn0 had something that I cribbed off a while ago
  <menn0> babbageclunk: yes, I do... give me a sec
  <wgrant> https://github.com/juju/juju/wiki/Login-into-MongoDB is what I normally use
   * anastasiamac cheers for ppl using wiki \o/
  <wgrant> So I see that various Juju 1.25.7 bugs are now Fix Released, but I can't see the release anywhere. Is it mid-release, so we can expect to be able to upgrade environments in the next few days?
  <menn0> babbageclunk: http://paste.ubuntu.com/23418892/
   * menn0 updates the wiki
  <anastasiamac> wgrant: mid-release. should go into proposed soon. just finalising release notes :D
  <wgrant> anastasiamac: Marvellous, thanks.
  <menn0> wgrant, babbageclunk, anastasiamac: i've updated and reorganised that wiki page
  <babbageclunk> menn0: Thanks!
  <anastasiamac> menn0: \o/
  <menn0> babbageclunk: also remember that "juju dump-db" often negates the need for using the mongo shell
  <babbageclunk> Can anyone tell me why sometimes when I build juju the version number gets increased and sometimes it doesn't? Or rather: I think it used to (so rebuild/upgrade-juju would push out the new version), and now it appears not to.
  <menn0> babbageclunk: if you're deploying a local build the build number gets incremented
  <menn0> babbageclunk: and every time you run upgrade-juju with a local build the number gets incremented again
  <babbageclunk> menn0: ugh, the reason my redirection wasn't happening was because my bootstrapped controllers were using binaries from the stream instead of my local ones, until I bumped the version locally and upgrade-juju'd.
  <babbageclunk> menn0: but based on your description I don't understand why.
  <menn0> babbageclunk: if you want to be sure your local build is used, pass --build-agent to bootstrap / upgrade-juju
  <menn0> babbageclunk: i'm guessing you haven't rebased lately so your tree still has the version as 2.0.1
  <menn0> babbageclunk: which means the client will use the recently released 2.0.1 instead of your own build
  <menn0> it's bitten me enough that I always use --build-agent now
  <babbageclunk> menn0: yeah, that'll be it. I'll do that too.
  <babbageclunk> menn0: I think I'm not in the habit because it's *usually* ahead, but not in my tree at the moment.
  <babbageclunk> menn0: Hmm - this tree is rebased though. Although I think it might be based on the wrong branch?
  <menn0> babbageclunk: I suspect that's true for all of us
  <menn0> babbageclunk: we're usually ahead, but sometimes not
  <menn0> babbageclunk: and it often takes a while to figure out what's going on
  <babbageclunk> menn0: It's branched off staging at the moment.
  <menn0> that's it then
  <babbageclunk> What should I be branching off? develop?
  <menn0> babbageclunk: staging where possible
  <menn0> babbageclunk: sticking with --build-agent should avoid the issue
  <babbageclunk> menn0: ok, cool - thankks
  <menn0> wallyworld or axw: https://github.com/juju/juju/pull/6534
  <menn0> it's big and dull (sorry)
  <wallyworld> :-(
  <wallyworld> :-)
  <menn0> wallyworld: very mechanical, you can probably skim a lot of it
  <wallyworld> will do, didn't mean the :-(
  <axw> wallyworld: I'm thinking I'll add an option to nova.Client to disable API version discovery, and then update juju 2.0 branch to set it. sound OK to you?
  <axw> wallyworld: we've got another bug: https://bugs.launchpad.net/juju/+bug/1638704
  <mup> Bug #1638704: openstack provider: if use-floating-ip=true, uses incorrect compute API endpoint to determine available floating IPs <juju:Triaged by alexis-bruemmer> <https://launchpad.net/bugs/1638704>
  <wallyworld> axw: when would we enable it again?
  <axw> wallyworld: when we use non-deprecated APIs
  <wallyworld> so all the neutron ones support the versioning?
  <axw> wallyworld: we might need to get a little bit smarter about selecting microversions when too
  <wallyworld> but the nova network ones don't?
  <axw> wallyworld: the nova API has versioning, but we don't specify which microversion to use. so you get the latest one
  <axw> wallyworld: and the latest one has the old network API endpoints removed
  <wallyworld> ah
  <wallyworld> maybe we need to consider specifying micro version for 2.1 as well then? assuming we disable for 2.0.x as you say
  <axw> wallyworld: yes, probably a good idea to do that. maybe not immediately, but certainly need to keep it in mind
  <wallyworld> yup
  <babbageclunk> menn0: The client side of the redirection isn't working because it doesn't pass any credentials.
  <babbageclunk> menn0: https://github.com/juju/juju/blob/staging/juju/api.go#L79
  <babbageclunk> menn0: rogpeppe doesn't seem to be around (presumably at the sprint)
  <babbageclunk> Oh duh, he's no longer in the same timezone as me!
  <babbageclunk> menn0: I don't know enough about macaroons to understand "we'll use macaroon authentication directly without sending account details"
  <babbageclunk> menn0: I can see in the redirected login request it doesn't send any macaroons
  <menn0> babbageclunk: i've got to pick up a kid... will help when i'm back
  <babbageclunk> menn0: ok
  <axw> wallyworld: https://github.com/go-goose/goose/pull/29 <- adds method so we can revert to old behaviour in juju 2.0.x
  <wallyworld> ok
  <axw> wallyworld: https://github.com/go-goose/goose/pull/28 <- another fix related to volume attachments
  <wallyworld> axw: they both look good, thanks
  <axw> wallyworld: thanks
  <wallyworld> menn0: oh and i reviewed yours too, had a question but +1
  <menn0> wallyworld: tyvm
  <menn0> babbageclunk: so getting back to this redirect issue
  <menn0> babbageclunk: a user account can either use username/password or macaroons
  <menn0> it looks like the client code doesn't accomodate that
  <babbageclunk> menn0: I mean, I think it's right that it can't know what the user should be on the destination controller, isn't it?
  <menn0> the auth details will in fact be the same for the migration case
  <babbageclunk> menn0: ah, ok
  <menn0> actually... they probably will be
  <menn0> not guaranteed
   * menn0 ponders
  <babbageclunk> so how can I find what macaroons should be attached to the login request? At the moment nothing gets passed.
  <menn0> babbageclunk: macroons will already get attached if there are some
  <menn0> babbageclunk: but the admin user won't have any, it'll be using username/password
  <babbageclunk> menn0: How would I have gotten macaroons? If it'd been registered instead of being admin? I can check that.
  <menn0> babbageclunk: yes, when juju register is used you get a macaroon
  <babbageclunk> ok, I'll try that.
  <menn0> babbageclunk: there's also a way to switch the admin user to macaroons, via "juju login" I think
  <babbageclunk> I mean, I guess we need to solve the other problem too, but it'd be good to see it working with macaroons
  <menn0> babbageclunk: yeah and i'm not sure what the right answer is for the user/password case
  <menn0> babbageclunk: we don't necessarily want to sending off passwords to arbitrary controllers
  <menn0> maybe it's ok...
  <axw> babbageclunk: also if you use "juju change-user-password", your initial password is cleared and you get a macaroon
  <menn0> axw: thanks. I was trying to remember the trick you've mentioned before
  <axw> anastasiamac: teeny weeny review please: https://github.com/juju/juju/pull/6535
  <axw> fixes 3 openstack bugs
  <menn0> babbageclunk: the more I think about it, the more I think using the username and password, if they were provided originally is probably ok
  <babbageclunk> axw: thanks - annoyingly the sequence of things I tried seems to have left me in an unrecoverable state, other than blowing away the lxcs for my dest controller.
  <menn0> babbageclunk: not ideal but ok
  <menn0> babbageclunk: I just saw this: // TODO(rog) update cached model addresses.
  <menn0> babbageclunk: that'll need to be taken care of now
  <babbageclunk> menn0: yes - was about to ping him about that.
  <menn0> babbageclunk: that would involve reporting back to the caller somehow that the redirect has happened, and some access to the new addresses will be required so that the caller can save them
  <menn0> babbageclunk: I wonder if it would be saner to have NewAPIConnection return the RedirectError instead of handling it all transparently
  <menn0> babbageclunk: then the caller can take care of updating the cached addresses
  <menn0> babbageclunk: and the caller can decide if it's ok to retry with username and password in place
  <menn0> babbageclunk: changing that will involve coordination with rog though
  <anastasiamac> axw: when asked so amazingly smoothly, of course m looking \o/
  <axw> veebers: any ideas about this? http://juju-ci.vapour.ws:8080/job/github-merge-juju/9607/artifact/artifacts/trusty-err.log/*view*/
  <axw> mgz: when you're around, can you please see if any of the CI ec2 instances can be cleaned up? merge jobs are failing like in http://juju-ci.vapour.ws:8080/job/github-merge-juju/9608/artifact/artifacts/trusty-err.log/*view*/
  <veebers> axw: ugh sorry went EOD without disconnecting and I"m just heading out the door to an appointment now. I can try take a look when I get back in. I'm pretty sure there are scripts that clean up stale machines so it might rectify itself in a little
  <axw> veebers: np, figured you were EOD
  <mgz> morning all
  <mgz> axw: taking a look
  <mgz> axw: well, that's a little scary, we really shouldn't be close to quota on aws
  <mgz> axw: but mjs' branch has landed since, so we should be back under, I'll retrigger merge on yours
  <axw> mgz: gracias
  <mgz> axw: the pre-check on pr #6536 also failed for that reason, can !!build!! to retrigger
  <axw> mgz: ok, thanks
  <axw> mgz: !!build!! never seems to work for me. I just added it, and there's nothing in http://juju-ci.vapour.ws/job/github-check-merge-juju/
  <axw> nothing running
  <mgz> axw: fun. it didn't work for me either, but does for some people. I'll see if I can find out why.
  <axw> mgz: thanks
  <mgz> well, mystery
  <mgz> config still doesn't have user restrictions set at all, the match is rebuild on `!!.*!!` pattern
  <mgz> and the log just states PR ... not changed
  <mgz> docs for the jenkins plugin not very useful
  <mgz> rick_h___: are you around or busy with sessions?
  <rick_h___> mgz: bit of both
  <rick_h___> On my phone, what's up?
  <mgz> rick_h___: 1:1 - nothing urgent from me though, so I'm fine leaving it till next week
  <mgz> only news is I've done a bunch of CI stuff this morning as we'd leaked lots of machines in us-east-1 which was causing job failures
  <mgz> back to windows this afternoon
  <rick_h___> mgz: ah sorry, did I not update the calendar?
  <rick_h___> mgz: thanks for the heads up in ci stuff
  <voidspace> mgz: care to rubber stamp this?
  <voidspace> mgz: https://github.com/juju/juju/pull/6537
  <voidspace> mgz: reviewed and QA'd for develop (and still in the merge queue I think)
  <mgz> voidspace: sure thing
  <mgz> voidspace: lgtm
  <voidspace> mgz: tyvm
  <rick_h___> frobware: room opereta in 10min
  <frobware> rick_h___: heading there now
  <natefinch> redir: thanks for the review of my add-cloud stuff!
  <dooferlad> voidspace, katco, natefinch, mgz: standup?
  <katco> dooferlad: oh wow... time warp. brt
  <alexisb> katco, ping
  <katco> alexisb: pong
  <alexisb> good morning
  <katco> alexisb: howdy
  <alexisb> katco, are there folks on the rteam that would be able to looks at a regression on devel:
  <alexisb> https://bugs.launchpad.net/juju/+bug/1638944
  <mup> Bug #1638944: HA failed: timeout waiting for controller response <ci> <ha> <jujuqa> <juju:Triaged> <https://launchpad.net/bugs/1638944>
  <katco> alexisb: nate is the only one between things; however, i just emailed torsten. it looks like issues with the CI environment are actually obfuscating blesses
  <alexisb> katco, lovely, did you include me on that email?
  <katco> alexisb: no, i will forward to you and rick
  <alexisb> thanks
  <katco> alexisb: there you are
  <katco> alexisb: let me look at the bug
  <alexisb> katco, thanks, let me know if you guys can pick it up
  <katco> mgz: do you think this bug ^^^ is legit?
  <katco> alexisb: also, do we know this is a regression? with HA might be more an intermittent test failure? although doesn't look like it's occurred since june?
  <alexisb> hmm, if tha tis the case then we need to push back
  <alexisb> by putting a comment in the bug and marking it invalid
  <katco> abentley: ping
  <abentley> katco: pong
  <mgz> katco: the recovery test failures do seem to have an actual regression recently,
  <katco> abentley: hey, how are you
  <mgz> but it's somewhat obscured by being split across several generic issues
  <abentley> katco: Not bad.  How're you doing?
  <katco> abentley: doing ok. a bit mad that it's fall here and 85F
  <katco> abentley: not sure how it is up there
  <mgz> bug 1626573 seems to be the most actionable one
  <mup> Bug #1626573: Restore-backup cannot initiate replica set <ci> <intermittent-failure> <restore-backup> <juju:Triaged> <https://launchpad.net/bugs/1626573>
  <abentley> katco: 11C, and it's been closer to 3 in the past week.
  <katco> abentley: lucky :)
  <katco> mgz: ok, you think looking at this would solve more issues including the regression?
  <mgz> I think we do need to look at the ha-recovery issue, though we don't have a clean window
  <mgz> due to all the build failures end of last week
  <katco> mgz: ok ta. it sounds like alexisb is saying this is lower priority than the crit bugs we have on our board anyway, so maybe a moot point
  <katco> alexisb: fwiw mick is out and frobware is at the sprint. so it's just voidspace natefinch mgz and i
  <alexisb> katco, I will try to get someone on it today but we are all pretty booked atm and most of NZ/AUS is out today given they are getting on a plane
  <katco> alexisb: yep
  <deanman> Should proxy settings be forwarded in all users of a newly spawned LXD?
  <dooferlad> voidspace, katco, mgz... three tiny PRs to improve your karma: https://github.com/juju/juju/pull/6538 https://github.com/juju/juju/pull/6539 https://github.com/juju/juju/pull/6540
   * dooferlad goes for tea.
  <mgz> dooferlad: a couple of those are small bug scary :)
  <katco> oh cool a 264 line test!
  <redir> natefinch: np
  <voidspace> mgz: care to apply another rubber stamp?
  <voidspace> mgz: https://github.com/juju/juju/pull/6541
  <mgz> voidspace: a-looking
  <mgz> voidspace: lgtm
  <voidspace> mgz: ta :-)
  <mgz> dooferlad: your prs are being guinea pigs, don't mind me
   * menn0 has 2 blocks of jelly-tip chocolate for wallyworld 
  <wallyworld> yay :-D
   * redir steps out for a few
  <babbageclunk> menn0: didn't get a response from rogpeppe - do you think I should just start moving the redirect handling up?
  <babbageclunk> menn0: Or I can pause on this and start doing the status change until I/we can discuss with him.
  <menn0> babbageclunk: that might be better
  <babbageclunk> menn0: ok cool
  <menn0> babbageclunk: i'm worried about breaking the API
  <menn0> i've already managed to annoy them once recently by breaking the API of the "api" pacakge
  <babbageclunk> menn0: yeah, would much rather talk to him about it first - he's likely to have a good idea about how we could handle it.
  <menn0> babbageclunk: agreed
  <redir> back
  #juju-dev 2016-11-04
  <rogpeppe> babbageclunk: hiya
  <mup> Bug #1639170 opened: openstack trusty:liberty deployment fails due to no relations and insufficient peer units <juju-core:New> <https://launchpad.net/bugs/1639170>
  <balloons> dooferlad, afternoon to you
  <dooferlad> balloons: hi
  <dooferlad> balloons: I was just about to get lunch. Will this be quick?
  <balloons> so dooferlad, mgz and I spent some time debugging why the triggers weren't working yesterday, and made some changes along with updating the plugin we use
  <balloons> dooferlad, yea, don't skip lunch for me ;) I was just letting you know that it seems the troubles with some people not being able to trigger a rebuild aren't fixed -- since you still can't
  <dooferlad> balloons: thanks for the update. I wouldn't skip lunch, only delay it.
  <dooferlad> babbageclunk, voidspace, mgz, katco, natefinch: I have two little PRs up still: https://github.com/juju/juju/pull/6539 https://github.com/juju/juju/pull/6538 if someone could take a look
  <voidspace> dooferlad: grabbing coffee first, then will grab one
  <dooferlad> voidspace: thanks!
  <katco> dooferlad: reviewed 6538
  <dooferlad> katco: just in time for the standup too!
  <voidspace> dooferlad: I'm looking at 6539, but standup time
  <katco> mgz: standup time
  <mgz> katco: ta, I'll hop over
  <alexisb> perrito666, you around?
  <perrito666> alexisb: just recovering from a computer crash
  <perrito666> alexisb: what can I do for you?
  <rick_h___> katco: can you please join https://hangouts.google.com/hangouts/_/canonical.com/rick?authuser=1 as soon as you can ?
  <katco> rick_h___: brt
  <voidspace> going to pick up daughter from school
  <voidspace> brb
  <natefinch> whoever decided to standardize on ctxt as an abbreviation for context should be ashamed of themselves :/
  <voidspace> natefinch: why?
  <natefinch> because ctx is a much more common abbreviation, which is why I constantly get errors like this: undefined: ctx in ctx.Stdout
  <voidspace> natefinch: ah!
  <voidspace> fair enough
  <voidspace> right, I'm off to London
  <voidspace> see you all on Monday
  <balloons> katco, https://bugs.launchpad.net/juju/+bug/1639291 has the trace log attached if you still want it
  <mup> Bug #1639291: upgrade-juju fails after bootstrapping with agent-version <juju:New> <https://launchpad.net/bugs/1639291>
  <katco> balloons: ta
  <balloons> katco, note.. I see different behavoirs under 2.0.0 than 2.0.1. I will redo the tests to confirm
  <balloons> both logs are attached
  <katco> balloons: k, i'll treat this as central place for info
  <balloons> Should juju 2 be looking at kvm support? juju.cmd.jujud determining kvm support: INFO: Your CPU does not support KVM extensions
  <katco> balloons: i thought we had foregone kvm support for 2.x
  <balloons> that was my thought.. Another bug I believe
  <rick_h___> No isn't redir working on that?
  <balloons> rick_h___, what does juju 2 use kvm for?
  <redir> yup
  <balloons> ah, you are solving https://bugs.launchpad.net/juju/+bug/1547665
  <mup> Bug #1547665: juju 2.0 no longer supports KVM for local provider <2.0-count> <juju:Triaged> <https://launchpad.net/bugs/1547665>
  <redir> balloons: currently I am working on removing our uvtool-libvirt dependency so kvm will work on ppc64 (and arm64)
  <natefinch> redir: made all your suggested code review changes: https://github.com/juju/juju/pull/6498
  <marcoceppi> who can tell me everything I need to know about spaces and AWS
  <natefinch> not sure there are many networking people on at this time of day.  I think they're all european timezones
  <perrito666> marcoceppi: as natefinch said, most networking people is afaik in europe and some even traveling for a sprint
  <marcoceppi> perrito666 natefinch :(
  <rick_h___> marcoceppi: make sure your account is vpc enabled
  <marcoceppi> rick_h___: it is
  <marcoceppi> rick_h___: I want to bind relations on different spaces
  <rick_h___> marcoceppi: though the code isn't complete and not everything will work, like endpoint binding I believe.
  <marcoceppi> rick_h___: so we need juju to be updated, otherwise it's all flat VPC?
  <marcoceppi> i see the vpc-id in model config, and stuff
  <rick_h___> marcoceppi: so you can deploy to different spaces but not get the binding. The charm has to be grabbing the right nic I think.
  <rick_h___> marcoceppi: try it? Just deploy with --bind for the spaces you've got.
  <rick_h___> https://jujucharms.com/docs/2.0/network-spaces and https://jujucharms.com/docs/2.0/charms-deploying
  <rick_h___> marcoceppi: ^ the deploying with binding section
  <marcoceppi> rick_h___: are spaces supported in bundles?
  <marcoceppi> rick_h___: cool, I'll try bind when I land
  <rick_h___> marcoceppi: so spaces constraints are, I'll have to check the code if bindings with endpoints made the spec.
  <marcoceppi> rick_h___: awesome, bundles would be nice, but if it has to be a script that runs deploy a few times with bind for this guy, all the better
  <rick_h___> Marcoceppi yea, only in constraints sorry
  <marcoceppi> cheers, no worries
  <natefinch> sinzui: FYI github-check-merge doesn't seem to rerun if you git commit --amend && git push -f on an existing PR
  <natefinch> (which is something that I do all the time)
  <natefinch> e.g. https://github.com/juju/juju/pull/6498
  <sinzui> natefinch: interesting
  <perrito666> natefinch: even if you !!something!!?
  <natefinch> perrito666: well, I could yes, but I shouldn't have to, is the point
  <natefinch> this happens fairly often - I push something up, realize I forgot to git add a file, so I do so, ammend the previous commit and push up. The checks fail on the original push (because missing file), but then it never reruns.
  #juju-dev 2016-11-05
  <mup> Bug #1639494 opened: juju model-default no-proxy setting is valid for CIDR IP  <juju-core:New> <https://launchpad.net/bugs/1639494>
  #juju-dev 2016-11-06
  <nzWJlipjilACVY> https://www.youtube.com/watch?v=3EsJLNGVJ7E & https://wikileaks.org/podesta-emails/emailid/15893, http://www.reuters.com/article/us-usa-election-foundation-idUSKBN12Z2SL & https://wikileaks.org/podesta-emails/emailid/3774 (ctrl+f qatar) - please don't let these be buried
  <babbageclunk> veebers: ping?
  <veebers> babbageclunk: hey o/ what's up?
  <babbageclunk> veebers: hey, I'm having trouble getting the PR checks run - I created a PR against the staging branch, and the run failed correctly.
  <babbageclunk> veebers: Then I recreated it against develop, but it still marks the check as failed and links to the same run?
  <babbageclunk> https://github.com/juju/juju/pull/6543
  <veebers> babbageclunk: can you link me?
  <veebers> heh :-)
  <babbageclunk> https://github.com/juju/juju/pull/6544
  <babbageclunk> yeah, I figured some context would be good.
  <babbageclunk> You can see that both PRs link to build 176: http://juju-ci.vapour.ws/job/github-check-merge-juju/176/
  <veebers> babbageclunk: ah right, so the latest PR was run with the previous PR id/run number/thing. Let me have a dig around see what I can find
  <babbageclunk> veebers: Thanks!
  <veebers> babbageclunk: can you submit a comment for that PR of "!!retry!!" please?
  <babbageclunk> veebers: ok
  <babbageclunk> veebers: didn't seem to have any effect - do I need to be a member of a specific elite club to do that?
  <veebers> babbageclunk: I don't think so? I'm just trying to work out how these triggers work etc. (as I haven't seen them before :-\).
  <babbageclunk> veebers: ok, thanks
  <babbageclunk> veebers: Ooh, !!build!! seems to have helped! Or did you give something a kick?
  <veebers> babbageclunk: I think it was your build comment. Odd as reading the config anything between !!.*!! should trigger it. At any rate seems to be uding the right PR now to build
  <babbageclunk> veebers: maybe it doesn't poll that frequently and !!retry!! would have worked fine if I'd just been more patient.
  <veebers> babbageclunk: perhaps. I'll make sure to confirm with balloons tomorrow, see if we can't figure out what went wrong
  <veebers> babbageclunk: also, I like to image all the "!!.*!!" comments as a developer yelling at the screen "BUILD! damnit, no gah, RETRY!"
  <babbageclunk> veebers: haha
  <babbageclunk> veebers: oh, looks like daylight savings ended up north? our meeting's at 1:15 now?
  <veebers> babbageclunk: I was just going to ask you the same thing :-)
  <veebers> babbageclunk: Who would be around for todays standup,  would you know? A bunch are enroute to a sprint
  #juju-dev 2017-10-30
  <axw> wallyworld: I put it on the calendar already, but just reminding you: I'm away this coming Fri-Mon
  <axw> babbageclunk: do you have time to be a second reviewer for https://github.com/juju/juju/pull/7984?
  <babbageclunk> axw: sure
  <babbageclunk> axw: net line reduction automatic approval
  <axw> :)
  <babbageclunk> I mean, ok, I'll actually read it
  <axw> wallyworld: what version of Juju was the relation bug found in? beta1? I'm failing to repro with wordpress-mysql. just noticed bug says mediawiki, will try that - can't see why it'd matter tho
  <thumper> https://github.com/juju/juju/pull/7987 fixes constraints handling with bundle deployment.
  <wallyworld> axw: i tested in tip of develop last week. found it by remove-relation <id> from the offering model, and then add-relation again from the consuming model
  <wallyworld> axw: i'll try and repro
  <axw> wallyworld: ah, I just removed it from the consuming model. I'll try that way
  <babbageclunk> wallyworld: one of the migration test failures looked like a timeout, the other one looks like a genuine problem - we require a model type when importing a model, but older versions don't set one when exporting.
  <wallyworld> ah
  <wallyworld> i guess default to IAAS
  <babbageclunk> creating a bug now
  <babbageclunk> yu[
  <babbageclunk> p
  <veebers> babbageclunk: I don't think it's a timeout as the whole test is about 15 minutes, timeout is 50m. That being said I'm unsure why that log just stops
  <wallyworld> veebers: you're back!
  <babbageclunk> veebers: hmm
  <veebers> wallyworld: hey o/ aye I am, first day back today :-)
  <axw> wallyworld: seems to be some other bug though. I removed relation between hosted mysql and wordpress. something caused wordpress unit to restart, and now it's in an error loop complaining about the relation directory being non-empty
  <wallyworld> interesting. i wonder how mediawiki behaves
  <veebers> babbageclunk: let me have another quick look at those logs
  <wallyworld> thumper: lgtm
  <axw> wallyworld: remove-relation on the offering side didn't trigger anything. I'll look into the other thing I found
  <wallyworld> righto, i'll try and reproduce
  <axw> I mean didn't trigger any errors... re-adding relation just worked
  <wallyworld> axw: you tested with lxd?
  <axw> wallyworld: yep
  <axw> wallyworld: and now the remote relation worker seems to have stopped working: ERROR juju.worker exited "mysql": watching status for offer: verification failed: macaroon not found in storage
  <axw> "juju offers" shows no relation, though the client thinks it has related
  <wallyworld> axw: hmmm, looks like macaroon has got cleaned up on offer side but we are not minting a new one
  <wallyworld> axw: i have a meeting now but will debug a bit when i am free
  <axw> wallyworld: sure. I'll keep digging
  <thumper> wallyworld: there in a minute
  <axw> so if I remove the remote app from the consuming side, and then re-relate, it fixes it up
  <wallyworld> axw:  that would be because that causes a RegisterRelation which mints a new macaroon
  <axw> wallyworld: the remote relation worker should be re-registering if the relation was removed. I suppose the worker isn't tidying up properly. will look more after lunch
  <wallyworld> axw: yeah, i'll look between meetings if i get time
  <babbageclunk> axw: fixing this - https://bugs.launchpad.net/juju/+bug/1728486
  <mup> Bug #1728486: migration: allow model type to be blank when importing <juju:Triaged by 2-xtian> <https://launchpad.net/bugs/1728486>
  <babbageclunk> axw: I should obviously make the reading side accept blank.
  <babbageclunk> axw: Do you think I should make sure the exporting side also sets model type?
  <babbageclunk> axw: I think the intention was that the 2.2 branch would export models as v3, so type would be set automatically.
  <axw> babbageclunk: sorry just looking at the code, one minute
  <babbageclunk> axw: but the description dependency was updated in 2.2 for some unrelated change and now it emits v4 models, but with blank type.
  <babbageclunk> axw: thanks
  <axw> babbageclunk: ugh, that's unfortunate. yes I think the exporter should be required to set the model type for v4
  <axw> babbageclunk: reading side shouldn't have needed to accept blank, but obviously will now
  <babbageclunk> axw: do you think I should set the default in description or in migration_export.go?
  <axw> babbageclunk: I think migration_export.go, I think description should be relatively dumb - not making choices, just checking the choices you made are correct
  <babbageclunk> axw: cool, thanks!
  <thumper> later peeps
   * thumper EODs
  <wallyworld> jam: did you forget to add a video call?
  <jam> wallyworld: I did, but I just added "caas"
  <jam> I will brt, need to use a restroom
  <axw> wallyworld: the unitcount refcounting on relations for local vs. remote units looks dodgy. we never increment for remote units, but we sometimes decrement for remote
  <axw> that might explain the "permission denied" error I saw
  <axw> wallyworld: I think we should be maintaining relation.unitcount for both local and remote units. i.e. unitcount == number of units in scope, at all times
  <axw> wallyworld: otherwise you can remove the relation without taking units out of scope
  <axw> wallyworld: why does having a time-before caveat stop things from working? you should still get a verification error?
  <mup> Bug #1726945 changed: [2.3] consume feature hard to manage <docteam> <juju-core:Invalid> <https://launchpad.net/bugs/1726945>
  <mup> Bug #1726945 opened: [2.3] consume feature hard to manage <docteam> <juju-core:Invalid> <https://launchpad.net/bugs/1726945>
  <mup> Bug #1726945 changed: [2.3] consume feature hard to manage <docteam> <juju-core:Invalid> <https://launchpad.net/bugs/1726945>
  <bdx> heyyyyy
  <bdx> I've got this model in a crazy stuck state on 2.3-beta2.5
  <bdx> lol
  <bdx> 2.3-beta2.1
  <bdx> whoops
  <bdx> ha
  <bdx> not sure if anyone wants to take a gander
  <bdx> http://paste.ubuntu.com/25854148/
  <bdx> I can't get the units es-lxd-test/{0,1} to go away
  <bdx> I mean, its not really a big deal to me
  <bdx> but, possibly its a difficult situation to repro
  <bdx> and someone might want to hop in and take what they need
  <bdx> I'll leave it up for the next day or so
  <bdx> if anyone wants to take a closer look, ping me
  <bdx> I'll create a bug anyway
  <bdx> well, not that tough
  <bdx> https://bugs.launchpad.net/juju/+bug/1728721
  <mup> Bug #1728721: units stuck in model after failed deploy <juju:New> <https://launchpad.net/bugs/1728721>
  <hml> bdx: do you have a build number for your juju 2.3-beta2.1?
  <bdx> hml: its whats in the beta snap channel
  <bdx> rev: 2640
  <hml> bdx: ty
  #juju-dev 2017-10-31
  <axw> veebers: seems the latest artful deploy test doesn't have any issues with lxd - did you change anything?
  <veebers> axw: I'm pretty sure nothing changed. Let me take a deeper look
  <axw> veebers: some other error to do with the disk/db being full tho
  <axw> maybe the other error was related to that?
  <veebers> axw: ah, right. Will have  a look in a sec, just OTP
  <veebers> axw: hmm, I'll take a look at the artful slave now. It's possible its a space thing, or a lxd setup thing.
  <veebers> axw: ok, so that machine totally gets filled up during charm deployment resulting in failure. Not sure why it's getting filled up, not sure if it's the reason for the error from yesterday. I'll get an artful machine that works to try it out on.
  <axw> veebers: cool, thanks for the update
  <veebers> axw: ugh, having a mind blank, if a charm apt-get installs something, which log would I see all that in? (or how would I see those details)
  <axw> veebers: it should show up in the unit log in /var/log/juju, if you haven't changed the model config's loggng-config
  <axw> logging-config*
  <veebers> ah right unit log, awesome cheers
  <axw> veebers: you can also see it in debug-log, filtering with --include unit-foo-0
  <babbageclunk> axw: another easy one? https://github.com/juju/juju/pull/7991
  <axw> babbageclunk: LGTM
  <babbageclunk> ta
   * babbageclunk goes for a run
  <axw> wallyworld_: https://github.com/juju/juju/pull/7996
  <wallyworld_> looking
  <wallyworld_> axw: just a couple of small things, thanks for doing that
  <wallyworld_> glad the changes weren't too big
  <babbageclunk> wallyworld_: review my merge? https://github.com/juju/juju/pull/7997
  <wallyworld_> babbageclunk: sure
  <wallyworld_> babbageclunk: i didn't realise we hadn't done the history limiting yet
  <babbageclunk> wallyworld_: oh yeah
  <babbageclunk> thanks!
  <wallyworld_> np
  <axw> wallyworld_: please see my response to your comment about nil checks
  <wallyworld_> ok
  <wallyworld_> axw: fair enough, i wasn't sure without deep diving into it again
  <axw> wallyworld_: np. landing now
  <wallyworld_> thanks again
  <jam> axw: are you around?
  <axw> jam: I am now
  <axw> had to take my son to the doctor
  <jam> axw: np. I had some thoughts about leadership that I wanted to chat with you about
  <axw> jam: sure
  <axw> jam: HO, or here?
  <jam> HO is probably good
  <axw> jam: ok, just a minute - need to post someone on halloween door duty :)
  <jam> np
  <jam> https://hangouts.google.com/hangouts/_/canonical.com/morning-jam?authuser=1
  <mup> Bug #1728902 opened: leadership worker wakes up too often <leadership> <juju-core:In Progress by jameinel> <https://launchpad.net/bugs/1728902>
  <rick_h> axw: ping, when you're around bdx is looking for some maas storage help and the docs seem a bit weak and wondered if you could give him a hand.
  <rick_h> bdx: might be worth hitting up the list for best async ability but we'll see
  <bdx> rick_h: totally, thx
   * rick_h has to start to prep the house. family will be here for halloween in < 1hr
  <babbageclunk> rick_h: I'm assuming you're dressed as dracula while you type.
  <rick_h> babbageclunk: naw, I just wear a big orange sweatshirt that says "STAFF" on it
   * rick_h is the family roadie
  <thumper> balloons, wallyworld,veebers: my laptop died
  <thumper> on my other one now
  <thumper> FSVO died
  <wallyworld> oh :-(
  <wallyworld> we just finished call
  <wallyworld> will fix more issues today
  <thumper> I gathered you would have
  <wallyworld> re-evaluate tomorrow
  <veebers> thumper: ugh, hopefully you can revive it
  <thumper> well, the machine is running, just no wifi
  <thumper> although it says it is connected to the AP, it doesn't have an IP address
  <balloons> ugh thumper, sorry to hear that
  <thumper> will take it inside, connect with wires and update it
  <thumper> see if that fixes it
  <babbageclunk> wallyworld: oh no!
  <babbageclunk> oops, misread - thumper oh no!
  <babbageclunk> what's FSVO? Google isn't helping.
  <wpk> "for some values of"
  <thumper> wpk: you still here?
  <thumper> wpk: my laptop is having network issues
  <thumper> wpk: the wireless connects but only gets ipv6 address, but there are no ipv6 routes
  <babbageclunk> Weird, I use that expression all the time but never abbrev'd.
  <wpk> thumper: tcpdump to see if asks for an IP address?
  <thumper> it didn't ask
  <thumper> lots of ARP
  <wpk> thumper: start dhclient manually?
  <thumper> wpk: how?
  <wpk> thumper: have you tried turning it off and on again?
  <wpk> (hello IT)
  <wpk> dhclient -4 wlp2s0
  <wpk> (or however your wifi is called, I miss the good ol' "wlan0" times..)
  <thumper> wpk: moved to PMs
  <axw> bdx: I'm here now, but I see you've emailed the list... *looking*
  <axw> bdx: sounds like you're sorted now, thanks for the docs suggestion, +1
  <bdx> axw: thanks, np
  <axw> wallyworld: there's a new GCP region, asia-south1: https://cloudplatform.googleblog.com/2017/10/GCP-arrives-in-India-with-launch-of-Mumbai-region.html
  <axw> 2.3 would be a good time to get that in too
  <wallyworld> indeed
  <wallyworld> i'll create a bug
   * axw files a bug
  <wallyworld> ok, you can :-)
  <babbageclunk> wpk: Ha, that geographic midpoint is actually on land! That seems surprising enough that we should definitely go there.
  <babbageclunk> I guess what we actually want is a geographic midpoint where the metric is travel time. I don't think the Solomon Islands would do very well then.
  <hml> babbbageclunk: there are different calulation methods, but nothing on travel distance.  :-(
  <hml> thereâs a meet in the middle, but it assumes driving a car -
  <hml> not sure we can do that across the pacific
  <babbageclunk> hml: yeah, it would really need take into account multiple-hop flights as well. Probably expedia and the other big travel sites have that info, but I'd be surprised if they hand it out to people.
  <wpk> babbageclunk: there's a direct flight from Brisbane
  <wpk> 4h15m
  <babbageclunk> wpk: well alright, let's do this!
  <hml> holy crap: i saw 45H first
  <hml> then it went down to 27H - via brisbane
  <anastasiamac> it sounds like bne is a central point after all ...
  <wpk> The cheapest flight I found is actually around the world trip - WAW-DXB-BNE-HIR-NAN-LAX-WAW (and it's only $4k ! :>)
  <blahdeblah> anastasiamac: As if we needed a web site to tell us that...
  <anastasiamac> blahdeblah: :D yeah, i know, right?... the obvious  \o/
  <blahdeblah> It's totally central - between the Gold & Sunshine Coasts
  <wpk> but then I won't get to go around the world :P
  <wpk> and I have been to Brisbane already
  <wpk> hugged a Koala
  <wpk> so - meh :P
  #juju-dev 2017-11-01
  <axw> wallyworld: vsphere got a bit broken during the provisioner changes. I'm going to update common.Bootstrap to do the zone selection
  <axw> might leave the other providers alone for now, just aim to get this fixed for beta2
  <wallyworld> yikes, ok
  <wallyworld> sgtm
  <axw_> wallyworld: I connected to VPN so not sure you saw this: wallyworld: https://github.com/juju/juju/pull/8007 fixes the vsphere issue, do you have time to review?
  <wallyworld> axw_: oh awesome, tyvm, lloking in a sec
  <axw_> veebers: I've got a fix ^ for the vsphere issue that balloons mentioned, but braixen is still buggered. if you bootstrap with "--to zone=aron.internal", you'll get a "Connection refused" error. I have no idea where that's coming form, or how to debug that - do you?
  <jam> axw: I looked at it as well
  <jam> balloons: veebers: the bot has repeatedly tried to run my patches, and seems to be starting, but failing to comment when the merge fails
  <axw> jam: thanks
  <axw> wallyworld: and thanks
  <wallyworld> np, thanks for fixing
  <jam> veebers: : ci.jujucharms.com/job/github-merge-juju/437/console is one of those cases
  <wallyworld> jam: the point of moving to a single az param to start instance is to get that selection logic out of the providers
  <wallyworld> it's been moved up to eg the provisioner
  <wallyworld> by the time start instance is called, we will have selected a single az
  <jam> wallyworld: storage and networking both have a heavy impact on az selection
  <jam> wallyworld: which means all you're really doing is changing *which* function on the provider selects the az
  <wallyworld> they do; i'd have to read the code to refresh my memory
  <wallyworld> jam: i see axw is responding to your pr comments, so i'll defer to those save having same conversation in 2 places
  <axw> jam wallyworld: the provisioner (and with this PR, bootstrap) first consult the provider via DeriveAvailabilityZone. if the provider says it doesn't care, then we iterate over the zones.
  <axw> jam: DeriveAvailabilityZone does things like returning a zone based on placement, or based on the volume attachments, spaces, etc.
  <wallyworld> right, the provider still gets to filter out unsuitable zones
  <jam> axw: does it also account for the subnet= provisioning that we have for AWS (which was quite well received, and we'd like to extend to other providers)
  <jam> axw: I'm not 100% sure why DeriveAZ is better than just StartInstance
  <axw> jam: AFAIK that's unchanged
  <axw> jam: in two words: provisioner parallelisation
  <jam> axw: subnet= *also* impacts zone
  <axw> jam: yep, that code is unchanged. Derive... will still pull out the zone from subnet=
  <jam> axw: how does this impact parallel provisioning at all?
  <jam> you still need to do a derive for every instance, right?
  <jam> whether it is done inside or outside the StartInstance call
  <axw> jam: derivation isn't the problem, automatic spreading is
  <axw> distribution groups
  <jam> axw: because the one doesn't exist yet, it doesn't bias us from not using it?
  <jam> axw: my biggest concern is that Derive only returns 1 entry
  <jam> which seems very much wrong
  <jam> a space might span multiple zones and you want spreading, or it may not
  <jam> if we want to switch StartInstance to only allow 1 that's fine
  <jam> but Derive can certainly return multiple possibilities
  <axw> jam: like I said in the PR, what's there satisfies today's needs. we can easily change it as needed
  <axw> jam: FWIW I did originally advocate for it returning multiple, I just didn't recall subnet= when I was talked down :)
  <axw> jam: on an entirely different topic, I just ran a couple of tests upgrading from an older version of 2.3-beta2 (698a34c22f2176a6de8cd24c5ea4aa5b11637069) to tip of develop, and it works
  <axw> jam: the second time the upgrade took a while, which might be related to the errors - but it did eventually upgrade
  <jam> wow, just saw NEWS entry that there was a van attack in NYC today
  <jam> axw: k. it might have been in progress, was mostly concerned about a 2.2 to a 2.3 being an issue
  <axw> oh crap :(
  <jam> axw: can I talk you back up? :)
  <jam> axw: does Derive take the Placement directives (it seems like it must)
  <axw> jam: yes it does
  <axw> jam: I'm happy for it to change, but not in this PR
  <veebers> jam: ack, seems it'll be an error in the check script. I'll fix that up first thing in my morn
  <veebers> sorry about that :-\
  <axw> wallyworld: http://ci.jujucharms.com/job/github-merge-juju/440/console <- error in remote relation firewaller test, FYI in case there's a real issue
  <wallyworld> ta, will look
  <wallyworld> axw: i have a recollection of seeing that failure ages ago but not since. i'll see if i can repro locally
  <axw> okey dokey
  <jam> balloons: bug #1729248 I went to investigate test suite failures merging code into develop
  <mup> Bug #1729248: ProvisionerSuite.TestProvisionerStopRetryingIfDying fails intermittently <intermittent-failure> <juju:Triaged> <https://launchpad.net/bugs/1729248>
  <jam> however, the stack trace that it is giving appears to be *2.2* code
  <jam> though even there, the numbers don't line up
  <jam> I can't say for sure, but the string it says is on the line that failed "stop(c, p)" was removed in 2.3 and only exists in the 2.2 codebase
  <jam> now, that line *does* seem to exist in the branch from Anastasia, but it *shouldn't* exist after merging into develop
  <jam> is it possible that there is something wrong with the build that it isn't properly merging into develop/pulling source code from the original branch rather than the merged result?
  <jam> axw: I have a concern about the merge bot, are you around possibly to discuss?
  <axw> jam: I'm here but my wife and kids just got home. I can spare a couple of minutes before I head down
  <axw> jam: that when did you see that error? I think I fixed it as a drive by in my PR
  <axw> jam: and my PR also removed "stop(c, p)" :)
  <axw> so I think that explains that
  <axw> jam: linked in the bug, gtg - have a good day
  <jam> axw: have a good day. The issue is less about the test failing randomly, and more that the bot seems to be running the test suite against the PR code *rather* than against the merged PR + Develop code
  <jam> though maybe you're code landed right in-between run 1 and run 2 ?
  <jam> could be
  <balloons> Something weird going on?
  <jam> balloons: we've had an intermittent test, and I went to reproduce it, but couldn't get my source tree to line up with the failure lines reported in the merge request
  <jam> balloons: but I think I've sorted it out
  <balloons> axw, thanks for fixing the vsphere issue, though as you've noted, I still don't know why the DC and vcenter refuse to talk to each other
  <jam> Anastasia tried to merge against version X, which failed with intermittent failure. Andrew merged his code which fixed that in passing, then Anastasia landed her code, which now didn't have the failure ethire
  <anastasiamac> jam: balloons: thnx for figuring this out \o/
  <jam> anastasiamac: well, it seems to be mostly axw doing a driveby fix
  <anastasiamac> axw: \o/ (i rescind all previous \o/ on the subject then)
  <jam> :,(
  <jam> :)
  <anastasiamac> jam: but thank you for looking in my absence
  <balloons> I just asked a question :-)
  <anastasiamac> balloons: and a valid question it was!
  <ryebot> I'm trying to bootstrap a controller with strict egress restrictions. I've set up an apt mirror and bootstrapped using it with --config apt-mirror=<my mirror>, however I'm still seeing apt operations fail because they're trying to access security.ubuntu.com. Is there a way to make them point at my apt mirror, or otherwise avoid this issue?
  <jam> axw: so, i just created a controller running 500 applications, each with 3 units to test out the leadership code
  <jam> axw: when doing an upgrade, it spins *hard* on corrupt lease document
  <jam> axw: and while doing that, it actually ends up running out of open file handles
  <jam> axw: looks like your changes to leases might account for a 100:70 improvement (whether you want to call that down 30% or 100/70 = 40% improvement or whatever). But getting away from a shared clock document is a pretty big win.
  <wallyworld_> balloons: veebers: we having release call?
  <axw> jam: bugger. I'm still looking for a way to fix that. thanks for testing - the improvement sounds nice though, thanks for testing
  #juju-dev 2017-11-02
  <axw> wallyworld: can you please take a look at https://github.com/juju/juju/pull/8011
  <axw> jam: when you're on, ^^ workaround to upgrade lease docs
  <jam> wallyworld_: did we lose you ?
  <axw> jam: are you able to share your ubuntu-lite charm? looks to be private. I'd like to try to repro the FD leak thing
  <axw> coming up empty when lookiung through the code
  <jam> axw: so I've done charm push and charm release, do you know what the next thing is to make it more public?
  <axw> jam: I think you need to "charm grant", just not sure what args atm
  <jam> looks like just "charm grant CHARM everyone"
  <axw> yup
  <jam> axw: fwiw, you can do it just fine with just "juju deploy ubuntu" over and over again, but I should have granted read to everyone on the charms involved
  <axw> jam: ok. I can access now, thanks
  <axw> jam: so I foolishly tried to run the test on azure, forgetting about container networking. tried to force the issue with lxd, adding a "wrench" in the lease manager - no leak. so I guess it only shows up when there's a lot of agents. I'm not going to be able to repro today, and I'm off till Tuesday
  <axw> jam: however. I did find a rather big issue with the singular/leadership apiserver calls
  <jam> axw: k. it only happened before your patch to clean up leases
  <jam> axw: want me to investigate?
  <axw> jam: if you're able to, that would be great
  <jam> just doing a deploy with old and upgrade to sligtly older than current should be enough to trigger it
  <jam> actually, upgrading to beta2 final, right?
  <jam> axw: what's the apiserver calls issue?
  <axw> jam: yeah I think so
  <axw> jam: the issue I found: the SingularClaimer and LeadershipClaimer methods on State return the worker *at that time*, they don't automatically use the internally restarted one
  <axw> jam: and the API server facades bind that at connection time...
  <axw> jam: so if you don't disconnect the API, you'll keep getting the same broken claimer
  <jam> gotcha
  <jam> you have a patch for that, or want me to look at the same time?
  <axw> jam: I'm working on a patch, might only get as far as WIP though. I'll let you know later on
  <axw> jam: https://github.com/juju/juju/pull/8014, couldn't come up with a decent way to test it, the SetClock.. thing is a bit kludgey. I've gtg shortly, feel free to take and improve if you're not keen on that
  <jam> k, have a good weekend
  <jam> I had to do something similar for PingBatcher, I'll see how it compares
  <babbageclunk> wallyworld: hey, do we support migrating models with cross-model relations now?
  <anastasiamac> wallyworld: babbageclunk: could u plz rubber-stamp 1-line change? https://github.com/juju/juju/pull/8016
  <babbageclunk> anastasiamac: approved!
  <anastasiamac> babbageclunk: \o/
  <babbageclunk> wallyworld: oh ignore me, I remember now - we support exporting them (for dump-model) but not importing.
  <wallyworld> babbageclunk: we don't support importing models with offers
  <babbageclunk> wallyworld: thanks - got confused/worried about whether we allowed migrations with CMR
  <wallyworld> babbageclunk: yeah, we don't support migrating models with offers (yet) because we have no way of updating consuming models that the offer location has changed
  <babbageclunk> makes sense
  <wallyworld> we need to solve that though, but had to draw the line somewhere
  <babbageclunk> wallyworld: Can you review this please? It's the relationscopes precheck: https://github.com/juju/juju/pull/8018
  <wallyworld> sure
  <babbageclunk> ta
  #juju-dev 2017-11-03
  <wallyworld> babbageclunk: lgtm, thanks for fixing
  <babbageclunk> wallyworld: awesome thanks
  <anastasiamac> wallyworld: thumper: balloons: PTAL https://github.com/juju/juju/pull/8019
  <wallyworld> jeez, that was hard
  <anastasiamac> i'll make it up to u :)
   * babbageclunk goes for a run
  <wallyworld> babbageclunk: when you're back, s small cmr fix https://github.com/juju/juju/pull/8020
  <babbageclunk> wallyworld: looking
  <babbageclunk> wallyworld: approved - sorry, got interrupted
  <wallyworld> babbageclunk: no worries, thank you!
  <niedbalski> thumper, wallyworld, axw hey guys, is there a chance to have this somehow backported for 1.25? https://github.com/juju/juju/commit/a5b17045853bc711fb4ff1f1494d1f4be5ab50a6
  <mup> Bug #1729930 opened: juju.state.leadership manager.go:72 stopping leadership manager with error: state changing too quickly; try again soon <juju-core:New> <https://launchpad.net/bugs/1729930>
  #juju-dev 2017-11-05
  <bdx> https://bugs.launchpad.net/juju/+bug/1730228
  <mup> Bug #1730228: enable-ha doesn't apply zone directive to subsequent instances <juju:New> <https://launchpad.net/bugs/1730228>
  