#maas 2012-11-01 <jtv> jam: thanks for the lint review â I was going to do that myself once I'd eyeballed the lint. I guess I didn't claim it in time. :) <rvba> roaksoax: I'm running the image downloading script in the daily package and I'm getting the following error: http://paste.ubuntu.com/1323535/ . Does it ring a bell? <roaksoax> lol <roaksoax> rvba: yeah they seems to have moved the image <roaksoax> Daviey: ^^ <roaksoax> Daviey: did you remove the squashfs from cdimage to releases? <roaksoax> Daviey: can we temporarily have it back until we SRU the fix to use releases instead? <rvba> roaksoax: When I open the link in my browser it works fine. <roaksoax> rvba: it doens't work for me <rvba> Weird, wget http://cdimage.ubuntu.com//ubuntu-server/daily/current/quantal-server-amd64.squashfs <roaksoax> rvba: The requested URL /ubuntu-server/daily/current/quantal-server-amd64.squashfs was not found on this server. <rvba> roaksoax: http://paste.ubuntu.com/1323571/ <roaksoax> rvba: maybe some kind of dns/storage issue <roaksoax> rvba: maybe they have mirrors being loadbalanced and one doesn'y have all the files? <roaksoax> rvba: routing problem? <roaksoax> rvba: this seems to be a server issue <cr3> hi folks! how can I configure an installed maas instance with my own power templates directory as ultimately returned by app_or_default().conf.POWER_TEMPLATES_DIR <cr3> I tried to add POWER_TEMPLATES_DIR=/my/path to /etc/maas/maas_local_celeryconfig.py, but my template is not being called and the default one is being called instead #maas 2012-11-02 <cr3> is there a way to define my own kernel parameters to generate the pxelinux.cfg file returned by the maas tftp server? #maas 2012-11-04 <dingus9> hey anyone around? #maas 2012-11-05 <jam> hi mgz <mgz> hey jam! <dimitern> jam, mgz: hey guys :) <jam> /wave <mgz> bug 1068086 is what I'd like to know what actually needs fixing <ubot5> Launchpad bug 1068086 in juju "juju fails to deploy with MAAS by default" [Undecided,Confirmed] https://launchpad.net/bugs/1068086 <jtv> Hi jam, hi mgz :) <jam> hi jtv <jam> good to see someone else has made it safely home <jtv> Most of Red & friends did, actually. <jtv> Only 2 MIA. * jtv is looking on the bright side today <jam> jtv: I saw that thumper only made it back to NZ about 4 hours ago, so i'm imagining that bigjools is similarly occupied. <jtv> He may be there, but he's not here. :) <jtv> I believe I heard he arrived at 01:00 local, today. <jam> rvba: hey raphael, we're poking around the kernel parameters stuff this week. We were wondering if there is an obvious DB table to put "global config" <jam> (command line parameters that by default should apply to every node) <rvba> jam: well, the Config table is certainly an option. <jam> sounds reasonable <rvba> Also, if you make this a Config item, it's easy to expose it on the Settings page if you need to. <jtv> Hi matsubara -- did you hear that our jenkins instance has gone read-only again? <jtv> Oh, looks like it was fixed a few hours back. <jtv> No, it wasn't. Grrr. :( <Lele_> Hi guys, anyone can lend me a hand on a maas setup ? <Lele_> we already have everything installed <Lele_> but when the server boots from pxe <Lele_> and timeouts the 20 secs on the boot menu that has all the options (local, maas-i386, comissioning, etc) <Lele_> doest boot anything to commision itself as DECLARED <Lele_> in the dashboard <Lele_> im missing anything ? <matsubara> did you run maas-import-pxe-files? <matsubara> what error do you get in the node trying to enlist? <Lele_> yep matsubara <Lele_> i have all the profiles <Lele_> if i run cobbler profile list <Lele_> and all the images are stored on the images directory <matsubara> hmm maas is not using cobbler anymore <matsubara> what version of maas are you running? <Lele_> i installed the 12.04 version <Lele_> from the main repo <Lele_> should i run the 12.10 ? <matsubara> ah, there's an updated version without cobbler. you probably want to use that one <Lele_> can i deploy 12.04 from that version ? <matsubara> Lele_, you can probably use the precise package in this PPA: https://code.launchpad.net/~maas-maintainers/+archive/dailybuilds <Lele_> awesome matsubara <Lele_> ill get you back if i continue with issues <Lele_> thanks for the support <bigjools> morning #maas 2012-11-06 <jam> morning all <jam> glad you made it back safe bigjools <bigjools> morning jam <bigjools> me too ;) <jtv> Hi jam, hi bigjools <mgz> did anyone catch smoser yesterday? <mgz> he posted a why-what? comment in bug 1044503 which is a bit suprising as I thought we'd asked him about it last week <ubot5> Launchpad bug 1044503 in maas (Ubuntu Raring) "kernel command line is not easily customizable" [High,Triaged] https://launchpad.net/bugs/1044503 <mgz> jam, dimitern: have put up a very simple migration branch, will commit some smarts around model and fallback now <Daviey> mgz: i did, i did! <mgz> Daviey: and the upshot was? :) <mgz> I'm still not really sure how this is going to be used, beyond maybe some simple customisations for specific hardware (like the highbank console thing currently hardcoded in the arch method) <Daviey> mgz: No upshot.. i just happend to catch him. <Daviey> now i think about it, it's not that helpful. <mgz> well, if you catch him again, don't let him go! <Daviey> because he bit your finger so? <mgz> hm, jenkins is unhappy, "Failed to create a temp file on /home/ubuntu/launchpad-jobs/workspace/maas-merger-quantal" <jam> mgz: it has been unhappy fora few days, it is either a perms issue, or a out-of-disk space, IMO <jam> rvba: ^^ you have direct access to jenkins, right? <jam> mgz: commented on https://code.launchpad.net/~gz/maas/1.2_add_db_kernel_params/+merge/133044 <mgz> thanks jam <rvba> jam: yes, the started yesterday, the whole filesystem of the vm went read only. matsubara is aware of the problem and worked on it yesterday; apparently he did not manage to fix the problem yet. I'm waiting for him or Larry to come online to see what they think. <rvba> s/yes, the/yes, that problem/ <matsubara> rvba, hey, I'm around now. I waiting on Larry as well since I couldn't get thta read-only problem yesterday <rvba> matsubara: all right. Please keep us posted when you'll have news, this problem is really blocking us now. <matsubara> rvba, will do. sorry about that <rvba> ta <roaksoax> rvba: hey man! how's everything? how was your flight back <rvba> roaksoax: Hey Andres, everything all right. How are you? <roaksoax> rvba: long :) <roaksoax> rvba: so anyways, where are we standing towards an SRU <rvba> Indeed. <roaksoax> rvba: are there things ready for me to start preparing for SRU? <rvba> roaksoax: you're talking about the precise SRU right? <gwd-laptop> I've got a question: Does MaaS require IPMI to function, or can it work with just DHCP / PXE / &c? <roaksoax> rvba: quantal <roaksoax> rvba: i need to catch up with the security team on the precise SRU <rvba> roaksoax: ok <roaksoax> rvba: but for 12.10 we has half of the bugs in the list already fixed, so I think we need to start SRU'ing by chunks <roaksoax> rvba: i'm also gonna be trying to reproduce the upgrade bug again and see if it shows up <rvba> roaksoax: we've committed most of the fix which will go in the SRU already https://launchpad.net/maas/+milestone/12.10-stabilization <rvba> fixes* even <rvba> roaksoax: 2 bugs are assigned to you in that list. Not sure what's the status for these twoâ¦? <roaksoax> rvba: well 1 is pretty hard to fix as there's no real way to find out whether it is a VM or real HW. and the second one is mostly IPMI issue itself, but will handle it by waiting more time before optaining an IP address, and dealing with the 0.0..0.0 as a failure to obtain IPMI address <mgz> huh. we haven't backported our other changes to 1.2 yet so basing on 1.2 was maybe not helpful. * mgz adjusts the test <mgz> jam, dimitern: how goes? <dimitern> mgz: I saw your change and I'm adapting my tests and code to match what you added <dimitern> I'll be ready later today with my changes, but still need a ui review <gwd-laptop> Ping? Anyone have an answer to my question? <mgz> gwd-laptop: what ever it was, it's not in my scrollback <gwd-laptop> mgz: Thanks -- the question was: "Does MaaS require IPMI to function, or can it work with just DHCP / PXE / &c?" <roaksoax> gwd-laptop: no it doesn't require IPMI to function, but then you would need to manually turn on machines <gwd-laptop> mgz: I think that's probably OK -- in this case they'll actually be VMs. :-) * mgz takes all the credit <gwd-laptop> Oops, sorry, didn't look closely. :-) <roaksoax> gwd-laptop: yeah, there's an option to use virsh as a power method <gwd-laptop> I work for Citrix on the Xen open-source team. I want to write a charm to install Xen / XCP on MaaS, so that they can be tested with the OpenStack testing that the Server team does. But we don't use MaaS for our test infrastructure at Citrix, so I was thinking of trying to use a virtualized environment. <mgz> that sounds like a good plan. not sure what docs we have currently on testing like that. <gwd-laptop> roaksoax: Would doing it manually work as well? I'm a lot more familiar with setting up XenServer than with setting up libvirt -- although, it's probably not bad to get a bit more practice with the latter... <smoser> mgz, the screen shots didn't make a lot of sense to me. <roaksoax> gwd-laptop: hold on, you want to deploy Xen based Vm's with MAAS? <roaksoax> gwd-laptop: or you simply want ot deploy Xen/XCP on machines using a charm? <gwd-laptop> roaksoax: What I actually want is a charm to install Ubuntu + Xen in a MaaS environment. <gwd-laptop> roaksoax: But I don't actually have a real MaaS environment, so I'm thinking of trying to set up a virtualized environment, just to test the charm. <gwd-laptop> roaksoax: If it's not really feasible, I can probably get a phsyical MaaS system set up here, but it's just more work. <roaksoax> gwd-laptop: right, so yeah a virtualized environment should be enough. The power management in maas is simply to start/stop machines/vm's when juju deploying something <gwd-laptop> roaksoax: So if I'm willing to babysit the installation and flip the virtual switch myself, I should be able to test the charm. Great -- I'll give that a try. <mgz> smoser: sorry, screenshots? customising the kernel params is pretty much just and admin level config thing to avoid some hardcoding of stuff <smoser> screenshots on the merge proposal <dimitern> smoser: that ones I added? <smoser> and i was never clear on what "tags" meant. i just wante dto see some example. <roaksoax> gwd-laptop: yeah, power management is not really required :) <gwd-laptop> roaksoax: Cool -- thanks for your help! <roaksoax> gwd-laptop: anytime! <mgz> dimitern: you seem to have proposed a 1.2 based branch against lp:maas rather than lp:maas/1.2 maybe? <dimitern> mgz: it's agains 1.2, the MP is there now just for ui review <dimitern> so it's on trunk (by mistake) <smoser> dimitern, yeah, i suppose. <smoser> i jus wanted an example description of how tags will be rendered. <mgz> smoser, okay, so the concrete stuff I can see this doing is changing stuff like the compose_arch_opts function in provisioning.kernel_opts into an admin level command like 'add-tag --kernel-params="console=ttAMA0" --definition="contains(/node/product, 'Highbank')"' <dimitern> smoser: so the tags are things you can attach to nodes, and the kernel opts can be part of the tag info, an in addition there are the global params in settings <mgz> so, rather than needing a provisioning server code change to alter the params for a specific set of hardware, you define something that matches against the lshw output, then the final stage boot uses the params attached to that tag <roaksoax> rvba: alright, so we are SRU'ing quantal MAAS into precise <mgz> roaksoax suggested this would be enough for what you guys want to do, hopefully that's the case? <smoser> hm.. <roaksoax> well yes, I suggested based on the fact that this was decided to be resolved using tags <smoser> mgz, so --definition= says 'attach tag named kernel-params with value "console=...." to all machines that are Highbank' <rvba> roaksoax: yeah, but that is going to require some extra work because of the dependencies (Django 1.4, etc.) <roaksoax> rvba: yeah, i guess we could simply SRU the minimal change as well? <smoser> and 'kernel-param' is some special tag that would even be identified as such in the maas-cli ? <roaksoax> rvba: that'[s the only blocker really <mgz> definition just specifies which nodes have that tag, based on their reported lshw output <mgz> by attaching the kernel params to tags, we hope to make the boot after that admin customisable based on the hardware <rvba> roaksoax: not sure it's the only blocker, I need to check the new dependencies that we use in the quantal version. <roaksoax> rvba: ok cool that would be helpful <roaksoax> mgz: right, so i think that regardless of the definition, it's still necessary to be able to assign kernel params to any given system <roaksoax> mgz: and might be required to assigned based on the name of the system <mgz> but what I'm still not clear on is which boots matter <smoser> well, any boot in which you want to do something matters <smoser> :) <mgz> with the ability to manually add a tag to any name, you can give one (or more) boxes completely custom params <smoser> so what happens if multiple tags named 'kernel-params' match a node? <mgz> what are the names for the three steps again? enlistment, commissioning, something else? <smoser> and are admin specified appended after MAAS required (ie, iscsi_target=....) <mgz> smoser: the easiest thing for us is to just take the params from one matching tag, in a deterministic fashion <jtv> mgz: you mean deployment? <smoser> that is the easiest thing, yes. <mgz> basically, tag-level customisation can't kick in till after we have the lshw output, which happens on the comissioning step, right? <roaksoax> right <smoser> so if i did like you suggested above with but with a different value with different criteria <smoser> what would happen? <smoser> which match would "win" <mgz> the globally configured ones could maybe be used before that, but there still needs to be a way for the nodes to talk to the maasserver to get that out <smoser> is that determinable? <mgz> smoser: one idea is to just take the first tag lexicographically, but we've not coded it yet, so what ever you like :) <mgz> the UI dimitern is working on aims to make this clear <smoser> that word is way to big for me. <mgz> by displaying the params used and where they came from <mgz> smoser: A comes before Z and so on <smoser> i'm confused. <smoser> which is why i just wanted some example that showed what would happen <nveitch> i think thats not a great idea <dimitern> smoser: you have tags, named "big-gpu", "large-mem", "0-special-tag", etc., so when sorted the 3rd one will be applied first, even if more than one of them match a single node <nveitch> chronologically would be better <mgz> the problem with chronologically is that would either need tracking seperately or would be unstable in the face of minor tweaks to tag definitions <smoser> right. <dimitern> smoser: and the other ones matching will be ignored, if none matches - use the global params from settings <smoser> so tag has name and value <smoser> right? <smoser> this is confusing me <dimitern> nveitch: ordering them chronologically poses problem with how to control the order and fine-tune it <mgz> they have a name, and a definition, which specifies what they match <dimitern> smoser: name, description (comment) and optionally, kernel_params <smoser> because it seems like your implying that tag_name="0-special-tag" and value="console=ttyS0" <mgz> (and a comment, which is just fluff, and shortly also an optional list of kernel params) <smoser> but then i dont know where the string 'kernel_params' would fit. <smoser> oh <smoser> that is odd <dimitern> smoser: the kernel_params field of the tag is the complete boot command line for the kernel, as I understand it - all params as a string <smoser> so you're suggesting that names have, importantly, a name and a 'kernel_params' <mgz> no, tag (name='0-special-tag', definition='true', kernel_params='console=ttyS0) <smoser> dimitern, well, it can't have the entire set of parameters, because that is not knowable outside of maas. <smoser> that is wierd. <dimitern> tag names are used just for sorting <smoser> dont you think this is weird? <smoser> we're promoting "kernel_params" to some sort of super item that is even then part of a tag? <dimitern> smoser: maybe I don't get it well enough to tell why it's weird? <mgz> maas will still supply some internally (needs to for various things that take arguments), but will allow you one extra set of params to stick on the end <smoser> if you told me "you can apply tags to nodes or groups of nodes" <mgz> smoser: yes. but I don't think it really matters. <smoser> i would think that a "tag" is either a key:value pair, or (less usefully) a 'name' <smoser> but i would probably not expect that it is a key:value:kernel_params triplet <dimitern> smoser: a tag is a few extra properties, which can be attached to a node, one of them can be the kernel params <mgz> it complicates the tag is name+definition thing, but sticking tag associated kernel params elsewhere wouldn't really change much <dimitern> smoser: it's not just key/value <mgz> I would prefer if the apis were kept clean of kernel stuff, but jam disagreed on the basis that this is the simplest thing <mgz> it would be perfectly possible to stick the actual data whereever, the main question is if getting that stuff based on one tag would be okay for what you'll want to do with it <mgz> the main issue with fancier things seems to be that kernel params are pretty opaque to us, we can't easilly tell what overrides what, what combines with what, so merging multiple params is non-trivial <smoser> right. <smoser> so i think its not at all unreasonable to just throw up your hands. and not attempt to merge or join in any way, but just let some definition "win" <smoser> but with tags, that seems somewhat arbitrary. <smoser> so i suspect thats why the lexical ordering based on 'name'. <mgz> right, and then we document that you name your tags specially for kernel params with 001- prefixes if you want clarity <smoser> but if we're playing the funny name game ... <smoser> then why not <smoser> 001-kernel-params="console=ttyS0" <mgz> because how do you then associate 001-kernel-params with a particular set of machines? <smoser> however you do that anyway. <smoser> i had assumed the --definition thing was not specific <smoser> to kernel params <mgz> or you mean pickling the params into the tag name? <smoser> if i have both key and value for a "tag" (which 'im still confused by) <smoser> then no pickling needed. <mgz> tags are just names, with some fancy logic attached <mgz> the fancy logic is optional <mgz> rather than needing to manually look at each machine and say whether a tag applies to it or not (node triage...) <mgz> you can give a definition that we then use to automatically apply the tag if appropriatte to any new machines as they show up <smoser> ok. that at least maks sense. <mgz> the kernel params are again just a bit of extra logic bundled with the tag, to say when booting a machine that matches this tag, add in these params <smoser> then yeah, my solution would require pickling name/value <smoser> which i dont think is beautiful <smoser> but i dont think the extra special attribute is either. <smoser> but then my suggestion was that you could just have a name of "001-kernel-params=console=ttyS0" <smoser> which would probably be improved to move kernel-params to the beginning. <smoser> but either way. <smoser> http://pad.daviey.com/maas-kernel-params <smoser> i put 3 points there that i'd like to at least have considered. <smoser> and know how i would address them <smoser> i think it'll work, but i just iddn't wan tto create somethign that wouldn't solve anyone's actual problems <mgz> thanks, extra details are useful <mgz> the -- is interesting... I wonder if we can leave that just to the user supplied params? <mgz> or will some of the ones that maas needs to add in itself have to be mixed before and after? <mgz> ...or can we assume user supplied ones should always be carried over to the installed system? <smoser> mgz, well, the -- is annoyhing. <smoser> and i ran into this personall a bit ago. <smoser> the BOOTIF= is the most annoying bit <smoser> as it gets copied over :) <smoser> as IPAPPEDN is "append" <rvba> jam: jtv mgz: matsubara fixed the Jenkins problem (apparently, rebooting the vm many times fixed it). <jtv> rvba: \o/ thanks! <rvba> Diogo will investigate further but in the meantime so that it does not happen again but we can land our branches. <jtv> And thanks to matsubara for the work... what fixes it for me is a manual âfsck -f -y /dev/<partition>â from a boot shell. <mgz> go matsubara <jtv> rvba: you landed my lint branch already? <rvba> jtv: yes, I used that branch as a test :) <jtv> I was so looking forward to landing that one. :) <roaksoax> rvba: jtv so when do you guys have a catchup call <jtv> roaksoax: normally around 08:00 UTC <rvba> roaksoax: 08:30 UTC <jtv> (I'm being vague because I believe it was just being changed due to DST) <jtv> Ah yes, the half hour too. <roaksoax> uh, that's early for me <roaksoax> :) <roaksoax> rvba: ok so we need to have a catch up call to talk about the upgrade path <roaksoax> and so on <rvba> roaksoax: ok, we can do that tomorrow. <roaksoax> alright cool <jtv> roaksoax: thanks for the review... I'm out too. :) <mgz> matsubara: tried merging again, but it failed once more alas... <matsubara> mgz, which merge proposal? <mgz> jenkins job 252, merge for lp:~gz/maas/1.2_fix_duplicated_039_migration <matsubara> job #253 passed <mgz> was that against trunk or against 1.2? <matsubara> 1.2 <matsubara> there are separate jobs for 1.2 and trunk <mgz> shall I try again and hope? <matsubara> I just did <matsubara> let's see <matsubara> mgz, the eagle has landed <mgz> woho! <mgz> clearly jenkins likes you better :) #maas 2012-11-07 <jam> so mgz, what pieces can I pick up so we can finish up the kernel params stuff, since we got Jenkins running again <mgz> so, what I still don't know how we're going to do is get the kernel options to the provisioningserver <mgz> also, did you see the notes from scott in last nights log? <jam> mgz: I did. I'm pretty confident about the specific setup, the most interesting part was the '--' issue. <jam> I think for a first pass, we can pick a spot to put it <jam> and if it is important, we can detect '--' in the parameters being set, and split it to that side of a '--' in the commandline. <mgz> that makes sense <jam> mgz: though maas itself doesn't seem to be setting '--', wh <jam> which means that in the immediate term, we don't have to do anything. <jam> if they need to pass "opts more opts -- extra_opts after doubledash" <mgz> it perhaps should, but we can probably punt on it then. <jam> it would just work <dimitern> jam, mgz: do we have the call this morning? <jam> dimitern: not explicitly, though I'm happy to chat with you guys. Shall we just meet up on mumble? <dimitern> sure, I have a headset now :) noise cancelling <jam> mgz: ^^ <mgz> er, let me install that, I did the g+ stuff <mgz> er... that won't be happening any time soon <mgz> can we do hangout instead? <jam> mgz: sure, though what happened to you being able to use mumble? (out of curiosity) <mgz> not using the special laptop, due to the screen borkage <mgz> and this borrowed one lacks mumble... or the qt libraries <jam> mgz: sure, I was wondering more about "that won't be happenign any time soon" <mgz> all will become clear :) <jam> dimitern: I seem to have 3 google identities for you, should I use the one with you picture associated with it? <jam> mgz: dimitern: https://plus.google.com/hangouts/_/216f679bd782c69f54ec4960241fd8cedef4340f?authuser=2&hl=en# <dimitern> jam: well, I have the @canonical one - that's it <jam> dimitern: when Martin joined, you suddenly disconnected <mgz> sorry, mistouchpadded the close button <jam> allenap: poke about the tftp server, I'm trying to figure out if it ever talks to the maas server <jam> Specifically, I realize now why mgz was worried about getting the provisioning server code to get information from the maas server. <jam> Because tftp is where the kernel commandline is getting set <jam> and AFAICT it never talks to the MAAS_URL <jam> so I can import 'auth' and use that stuff, but I want to make sure that is a sane thing to do. <jam> (I don't quite understand the layout given that tftp sits right next to the other worker tasks, maybe they run in the same process?) <jam> rvba: maybe you have an idea about how the tftp service is set up? <rbasak> Are we still supposed to be using quantal for development, or have we switched to raring? <rbasak> I ask because I just tried on quantal and maas-import-squashfs appears broken (in a way that the daily ppa probably fixes) <jam> rbasak: the lp:maas/1.2 branch is meant to be the stable release series for Quantal, I'm not sure about the ppas <rbasak> jam: what about trunk? <jam> rbasak: I don't really know. I would think that it isn't worth breaking Q compatibility yet, but I don't know if there is a reason to diverge. <rbasak> OK <rbasak> I just had to add multiverse for python-selenium, which wasn't a requirement previously, but fair enough <rbasak> Next import-squashfs failed :-( <dimitern> jam, mgz: how is it going guys? <jam> d <jam> dimitern: I belive mgz is finishing up his lunch. I've gotten the apis exposed, which should unblock your work. <jam> Though I have now hit the tftp wall, and while I can force my way through, I was hoping for feedback from red squad to make sure that was ok to do. <dimitern> jam: I see also your initial MP I depended on is approved now, how about landing it? <jam> dimitern: I marked it as such, jenkins should pick it up and land it shortly <dimitern> jam: ok, sweet - i'll get it from there so I can finish my tests <rvba> jam: I'm not really familiar with the tftp stuff. But AFAIK, it's a separated twisted process. It creates 'readers', one reader reads from the local filesystem to serve the images, the other uses the API to generated PXE files on the fly (the url for this is read from pserv.yaml). <jam> rvba: right, I roughly see the code that does it. The problem is it doesn't seem to use any MAAS logic to actually generate the pxe files. <jam> (it doesn't talk to the maasserver for anything, it just grabs the request parameters and uses it to fill out a template file) <jam> at least that I can see. <rvba> jam: it is supposed to talk to the API to generated the PXE file. <jam> rvba: k, it might be, I see a 'generator_url' in there. <jam> ok, found it, I think we can just poke data onto the KernelParameters, which should make things reasonable for us. Yay! <jam> rvba: thanks! <rvba> np <jam> untangling how it was all talking to eachother was a bit tricky <rvba> Indeed, that twisted code isâ¦ twisted. <jam> rvba: stuff like having a 'context' object that you can 'get()' data out of <jam> rather than passing parameters <jam> and just sort of magical 'get()' calls. <rvba> Yeah, that makes following what's going on harder. <jam> something seems fishy about the 1.2 series config. 'make run' is failing to startup again, which seems to be an '/etc/bin/rndc.key' failure-to-read <mgz> hm, have not tried make run in 1.2 yet <jam> mgz: I get the feeling people generally don't :). <jam> mgz: https://code.launchpad.net/~jameinel/maas/1.2-pxeconfig-includes-kernel-opts/+merge/133255 is closing the loop to the tftp server <jam> mgz: and now I'm 1 hr past EOD, and my son is right in getting me to stop. <mgz> the son is right <mgz> play time! see you tomorrow :) * mgz gets on with some branches <roaksoax> robbiew: hwody! is trunk fully operational? <roaksoax> err <roaksoax> sorry :) <roaksoax> rvba: ^^ <robbiew> heh <rvba> robbiew: hi, we have to fix bug 1075597 but that's underway. <ubot5> Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597 <rvba> err <rvba> roaksoax: ^ <rvba> Sorry again Robbie :) <roaksoax> rvba: alright, I'd like to upload to raring so I can start preparing the SRU for quantal <roaksoax> rvba: but now I'll look into the precise -> quantal upgrade to see whether i can reproduce the issue or not <rvba> All right. <rvba> jtv is working on a fix for bug 1075597. <ubot5> Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597 <roaksoax> ok cool <roaksoax> rvba: what were the consequences of upgrade failure? <roaksoax> rvba: not being able to deploy? <rvba> roaksoax: if you're talking about 1072772, the main symptom was that the DNS config was not being written. So yes, unable to deploy anything. <roaksoax> rvba: ok cool i'm upgrading now so we'll see what's the outcome <roaksoax> rvba: what else does that affect (without having to enable DNS/DHCP?) <rvba> roaksoax: no task can be fired so: no power_on/power_off task can be fired, the reporting of the boot images cannot happen. <roaksoax> rvba: ack! <roaksoax> rvba: alright, so I just upgraded and the issue didn't appear <roaksoax> rvba: message about images dissappear correctly <roaksoax> rvba: testing power management now <roaksoax> if it works, then DNS should also <roaksoax> rvba: alright, what seems to be the problem now is that tftpd-hpa does not stop <roaksoax> rvba: hence maas-pserv doesn't start <rvba> roaksoax: that looks like another problem. <roaksoax> rvba: python-maas-provisioningserver should probably conflict with tftpd-hpa... but that's weird because maas already does <roaksoax> err maas-cluster-controller <roaksoax> rvba: ok so I upgraded again <roaksoax> rvba: seems to be working just fine <roaksoax> rvba: but the logfile for cluster-celery wasn't created <rvba> roaksoax: and the celery process for the cluster is running? <roaksoax> rvba: nope <rvba> roaksoax: any error in the upstart log? <roaksoax> rvba: there's no log <rvba> Weird. <roaksoax> rvba: you can access the instance <rvba> Yep, I'm in. <roaksoax> rvba: so nodes enlist and commission <rvba> Nov 7 17:17:47 server-0fd520a0-454d-43b7-8e86-a7d5a4661e8e kernel: [ 1507.296247] init: maas-cluster-celery main process (1229) terminated with status 1 <roaksoax> yeah there's no error log though <roaksoax> rvba: what's the logfile that it should have? celery.log? <roaksoax> or celery-cluster.log <rvba> /var/log/maas/celery.log <rvba> roaksoax: I wonder how nodes could be enlisted if pserv is not running. <roaksoax> rvba: i mnanually restarted pserv <rvba> ok <roaksoax> rvba: so I stopped tftpd-hpa and started pserv <roaksoax> rvba: ok so the maas-pserv issue is definitely becuase tftpd-hpa is running <roaksoax> rvba: but maas-cluster-celery is the issue <rvba> roaksoax: I tried to restart maas-cluster-celery and it crashed. maas-cluster-celery main process (8707) terminated with status 1 <roaksoax> rvba: ok i think i got it <roaksoax> rvba: i think it is the way how it is sourcing MAAS_URL <roaksoax> or the file that containst it <roaksoax> rvba: ah no <rvba> roaksoax: I ran what the upstart job runs and I got this <rvba> https://pastebin.canonical.com/77847/ <roaksoax> rvba: yep, exaclt what i get <rvba> /usr/share/maas/celeryconfig_cluster.py is not on the path. <roaksoax> rvba: yeah I think i saw a bug fixing that <roaksoax> as it was sourcing something else instead <rvba> This must have been caused by a very recent change because it was working fine last week. <roaksoax> indeed <roaksoax> rvba: IIRC, we looked at it last week and it worked <roaksoax> rvba: i mean, we found the root cause <roaksoax> rvba: could this be it? "Another thing that came up is that maas-provision complains about not finding celeryconfig.py, a configuration module that we only use on the region controller. In a separate branch I'll make the import script select cluster_celeryconfig.py instead." <rvba> No, I don't think this is related. <roaksoax> rvba: i'm pretty sure you debug this couple weeks ago, I came to you and you found the problem <rvba> roaksoax: yeah, this rings a bell indeedâ¦ but I can't find what the problem is this time :( <roaksoax> rvba: wasn't it something in /usr/sbin/maas-provision? <rvba> roaksoax: I don't really remember tbh. Did you manually changed that file? Looks like it's changed since I checked. <roaksoax> rvba: yeah but set it back to default <roaksoax> rvba: it is not that file, i can't remember what was it <rvba> roaksoax: I think I found the pb. <roaksoax> rvba: what is it? <rvba> roaksoax: /etc/maas/maas_cluster.conf contains the wrong URL. <rvba> MAAS_URL=http://+1-905-866-7687/ <rvba> No, I don't think this is related.I changed it to: <rvba> MAAS_URL=http://+1-905-866-7687/MAAS/ <rvba> Then I restarted the service. <roaksoax> hmmm <rvba> If you look in /var/log/apache2/access.log you will see that it was desperately trying to register. <roaksoax> yeah <roaksoax> let me test this locally real quick <roaksoax> rvba: yeah that seems to be the solution <rvba> roaksoax: We really need to improve the error logging here :) <roaksoax> indeed :) <rvba> roaksoax: filed bug 1076080. <ubot5> Launchpad bug 1076080 in MAAS "No error message is printed anywhere when the cluster controller fails to register itself." [Low,Triaged] https://launchpad.net/bugs/1076080 <roaksoax> cool, I filed bug 1076075 <ubot5> Launchpad bug 1076075 in maas (Ubuntu) "maas_cluster.conf ends up with the wrong URL after upgrade." [Critical,Confirmed] https://launchpad.net/bugs/1076075 <rvba> Cool. <Lele_> hi guys, im running maas from the ppa dailybuiilds on precise <Lele_> when im trying to install maas-dhcp <Lele_> i get <Lele_> Setting up maas-dhcp (0.1+bzr971+dfsg-0+998+75~ppa0~precise1) ... <Lele_> Usage: /usr/bin/django-admin config_master_dhcp [options] <Lele_> Initialize master DHCP settings. <Lele_> a <Lele_> usr/bin/django-admin: error: no such option: --interface <Lele_> dpkg: error processing maas-dhcp (--configure): <Lele_> subprocess installed post-installation script returned error exit status 2 <Lele_> Errors were encountered while processing: <Lele_> maas-dhcp <Lele_> and indeed --interface its not a valid django-admin option <Lele_> is this a BUG ? <Lele_> a <Lele_> anyone guys ? <Lele__> i submited a bug for the dailybuild https://bugs.launchpad.net/maas/+bug/1076092 <ubot5> Ubuntu bug 1076092 in MAAS "maas-dhcp fails to install on Precise - dailybuild ppa" [Undecided,New] <Lele__> hope someone can take a look at it <bigjools> morning <bigjools> roaksoax: hi, want to talk about SRU? <roaksoax> bigjools: yes <roaksoax> but im coming back from lunch <bigjools> roaksoax: np, I am free for an hour <roaksoax> sonif you can give a few minuted will be grear <bigjools> are you drunk? :) <roaksoax> lol i wish it was friday night for me to do so <roaksoax> but im from the ohone <roaksoax> phone <melmoth> on 12.04, is there an easy way to tell maas to use a specific preseed file for _some_ nodes ? <melmoth> the idea would be to use lvm for computes nodes that have 2 drives. <melmoth> right now, maas install computes nodes without touching the 2 drive. So i ended up with 14 drive not being used <melmoth> (actually, 7 drive not being used) <bigjools> melmoth: no, that feature will be in 13.04 though <melmoth> thanks. <roaksoax> melmoth: u would have to ply witth cobbler <roaksoax> bigjools: ready when you are <bigjools> roaksoax: don't encourage people to play with cobbler <bigjools> since we're getting rid of it :) <bigjools> roaksoax: ok so is there anything else special needed for the SRU? <roaksoax> bigjools: I'm not encouraging :) i'm just saying... maybe costumer needs it <roaksoax> bigjools: ok so, yes. <bigjools> so far we have: Backport the Django 1.4 feature we're using and any other new dependencies (need to check those) <bigjools> and test like hell :) <roaksoax> yes <roaksoax> bigjools: so, the plan is basically, finish with those 12.10 bugs <bigjools> and the SRU team agree to this already? <roaksoax> bigjools: and implement the features the qa requrested, plus those missing such as kernel parameters <bigjools> I will make a 12.04 PPA build toda to get the ball rolling <roaksoax> and SRU that to both, quantal, and 12.04 <roaksoax> bigjools: i already have precise in PPA <roaksoax> its in experimental, same version we have in quantal <bigjools> roaksoax: the 1.2 branch? <bigjools> ah <bigjools> nice <bigjools> what did you backport? <roaksoax> bigjools: nothing yet, I simply needed that to test upgrade failures <roaksoax> but so far is the django thing <roaksoax> the problem <bigjools> ok, it won't work then :) <roaksoax> and dependencies in universe <roaksoax> bigjools: now, I did want to discuss this <bigjools> universe in 12.04 and main in 13.10 I presume <roaksoax> bigjools: when are you guys looking to finish the SRU-able features? <bigjools> so the "features" were 13.04 only really <roaksoax> bigjools: right, but kernel parameters are needed to be SRUable <bigjools> however given the QA team wanted it in 12.04 I think we should backport those separately <roaksoax> bigjools: uhmmmm maybe <bigjools> so the initial backport should be the bugs in the stabilization milestone <roaksoax> bigjools: well the thing is that we discussed this with smoser and jamespage <bigjools> yeah <bigjools> so I want to minimise risk <roaksoax> bigjools: and we agreed that the best approach was to SRU the QA team features as well <bigjools> if we change too much at once it's not minimised <roaksoax> bigjools: our mplan is to have precise SRU'd by 12.04.2 <bigjools> ok <smoser> bigjools, its not minimised :) <bigjools> that's January, plenty of time <smoser> that ship has sailed <roaksoax> yeah <bigjools> smoser: there's relative levels of minimising :) <roaksoax> bigjools: however, note that we will have to make incremental SRU's, so I'm gonna start with quantal <bigjools> so my suggestion would be to SRU the quantal stabilization stuff first, then backport that to precise, then SRU the new features to both <roaksoax> bigjools: my plan is that, sru stabilization, and then sru to precise <roaksoax> bigjools: not backport, SRU <bigjools> roaksoax: perfect <bigjools> yes, I meant SRU <bigjools> but it involves backporting dependencies of course <roaksoax> bigjools: we can't introduce new dependencies <roaksoax> bigjools: so they will have to get shipped with the maas source <bigjools> gaaahhhhhh <bigjools> really? <smoser> what dependencies are we talking about here? <roaksoax> bigjools: i have that covered since yui3, and python-tx-tfpt are <roaksoax> smoser: yui3, python-tx-tftp <smoser> as in those are not available in 12.04 ? <bigjools> correct <smoser> is it just a matter of pulling versions in quantal to 12.04 ? <roaksoax> smoser: both yui3 and python-tx-tftp are two new sources not in 12.04 <bigjools> this is also a matter for the security team <bigjools> and who will win? :) <roaksoax> bigjools: us... we have no other option than doing it :) <smoser> roaksoax, given the source of this request, i suggest that new packages to 12.04 is at least possibly negotiable. <roaksoax> smoser: i wasn't aware, nor never heard of being able to introduce new sources into the archive for an older release <smoser> and from security team perspective, if you put yourself in their shoes, and you were going to have to suppor this one way or another... <bigjools> smoser: +1 <smoser> which would you rather hvae? "real packages" or "all shoved inside maas" <bigjools> exactly - security team will need to track stuff shoved into maas <roaksoax> smoser: that's out of the question, definitely real packages, but in terms of policies, I don't think that's covered, is it? <smoser> https://launchpad.net/ubuntu/+source/walinuxagent <smoser> roaksoax, note, that package is not present in 12.04 "release" pocket. <roaksoax> smoser: right, alright, if it is possible <smoser> also note, that i'm not terribly happy with using it as an example, but i'm not aware of others. <roaksoax> then we should do that <smoser> ie, i'm not happy with that crappy package . but it is an example of something that went from "not packaged" to "main" in a SRU <roaksoax> smoser: right, <roaksoax> smoser: i don't really have issues in doing so, i just wasn't aware it was even possible <roaksoax> bigjools: ok so, the plan would be to finish 12.10 stabilization, SRU to quantal, then start looking at precise, sru that <roaksoax> and then SRU raring <roaksoax> as a second mayor SRU <roaksoax> smoser: agreed? <smoser> roaksoax, its not *really* possible <smoser> :) <roaksoax> smoser: right :) <bigjools> roaksoax: yes <smoser> roaksoax, yeah, i think that is a reasonable path. the only change i woudl suggest is <smoser> that you do not wait until "finish 12.10 stabilization" before "start looking at precise" <roaksoax> smoser: oh definitely not, I already have precise packaged <smoser> ie, lets get a ppa functional with what is in quantal now. <smoser> right. <roaksoax> smoser: just need to pull in django fix <smoser> bigjools, are you just awake because you're stuck on europe time? <roaksoax> and now split the other dependencies into their own packages <bigjools> smoser: pretty much <bigjools> smoser: body clock is screwed to hell <bigjools> smoser: was up at 4am <bigjools> sun comes up at 5 <smoser> roaksoax, we should consult security team to see what they suggest on the new dependencies. <roaksoax> smoser: will do <bigjools> then my twins wake up ... not much hope of sleep after that :) <roaksoax> smoser: the only real problem would be yui3, which is a new source that really replaces yui <smoser> and point out to them that a.) we really, really want SRU of 12.10 maas to 12.04, and b.) walinux did it, so maybe we can too <roaksoax> so they won't be co-installable IIRC <smoser> roaksoax, hm.. yeah, i dont know. probably can be sorted out. <roaksoax> alright, i'll talk to them and then sort things out <roaksoax> them = security team <bigjools> roaksoax: great, I'll catch up with you next week about that then (I am off work Friday) <roaksoax> bigjools: same here #maas 2012-11-08 <jam> bigjools: ping if you are still around <bigjools> jam: I am here <jam> bigjools: /wave. <bigjools> surprisingly awake too :) <jam> We've been landing our changes into the 1.2 branch <jam> but I'm realizing they should also get merged into trunk, right? <bigjools> I saw <jam> I tried doing the obvious merge, but there are a handful of conflicts. <bigjools> ideally they need to start in trunk because I wanted to do a bug-only SRU into quantal :( <jam> bigjools: I can revert the changes, but the bug was on the 12.10-stabilization <jam> which I thought meant it needed to be in the 1.2 branch. <bigjools> It does. I had intended to SRU features later but I realise that the bugtask was a bit misleading :( <jam> bigjools: so, if you want I can revert what I've landed so far, retarget them onto trunk. <jam> There is still the fact that 1.2 doesn't merge cleanly into trunk, should it? <jam> or are they intended to diverge? <jam> (So in the *immediate* term, I can put together my teams changes as a single branch, which you can land once you have done the bugfix SRU, and are ready to do a feature SRU) <jam> though I imagine any branch will bitrot if not landed soon <jam> bigjools: is it worth doing that effort? <bigjools> they are diverged <bigjools> (sorry had to deal with a large insect) <bigjools> you need to cherry pick revisions <jam> so, they *have* diverged, but it would be possible to converge them. s there something you don't want in trunk that is in 1.2? <bigjools> hmmmm let me think more about the feature branch <bigjools> there's stuff in trunk we don't want in 1.2 <jam> bigjools: right, I'm talking about the other direction <jam> I was thinking to land our code in 1.2 then merge up into trunk <bigjools> there *might* be stuff in 1.2 - jtv has been landing there first and forward porting <jam> but it isn't possible today because of divergence on both side. <jam> mostly simple textual conflicts that are fairly easy to resolve <bigjools> you should be able to CP stuff though? <jam> bigjools: we can, but it seems a waste to do so unless there is stuff you really don't want in trunk <bigjools> if you can work out which revisions are missing we can decide <bigjools> but there should be nothing that's not in trunk <bigjools> unless there was a separate fix landed just for a bug in 1.2 <jam> I'm willing to do the conflict cleanup. The main problem with specific revisions is that cherrypicking makes it hard to see where things specifically come from. <bigjools> ok thanks <jam> bigjools: so one diff between 1.2 and trunk that I think you worked on. There is a pserv task "import_pxe_files" in 1.2, and "import_boot_images" in trunk. <jam> I'm not sure which name you want to use <jam> . <bigjools> jam: rvba renamed the former to the latter; he needs to backport it <jam> k <jam> I'll make sure to use boot_images after the merge. <bigjools> rightyo <bigjools> jam: I think you need to revert the 1.2 features and land them on trunk <jam> bigjools: i'm willing to do so, do you know an ETA when 1.2 would be opened again. <jam> I guess at this point, we would essentially just hand it off to your team for that, though. <jam> /wave mgz, dimitern <dimitern> jam: /wave :) <jam> I'm sitting in mumble if you guys would care to join. <dimitern> ok <jam> w7z: poke <mgz> hey jam <mgz> ...I wonder if I can find an extra network cable <jam> mgz: I've got about 10 of them here. <jam> If you feel like dropping by :) <jam> mgz: care to join us on mumble? <jam> bigjools: https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133424 <jam> and now https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133425 to properly target 1.2 <bigjools> jam: if you make a branch targeted to 1.2, we can land it after the first SRU <jam> bigjools: sure <jam> the patch is essentially revert this revision :) <bigjools> ok :) * bigjools heading off shortly, tired.com <jam> mgz: https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133425 <jam> mgz, dimitern: https://code.launchpad.net/~jameinel/maas/merge-1.2/+merge/133418 <jam> mgz: DatabaseError: column maasserver_tag.kernel_opts does not exist <jam> LINE 1: ...er_tag"."definition", "maasserver_tag"."comment", "maasserve... <jam> evilnickveitch: just to let you know, I'm landing a 1.2 merge => trunk which includes your doc fixes <evilnickveitch> jam, thanks for the headsup <jam> mgz, dimitern: https://code.launchpad.net/~jameinel/maas/land-kernel-opts-in-trunk/+merge/133434 <w7z> failure is https://jenkins.qa.ubuntu.com/job/maas-merger-quantal/264/console <jam> mgz w7z: Did you see bug #1073090 ? <ubot5> Launchpad bug 1073090 in MAAS "Using MAAS and --constraints "maas-name=$SPECIFIC_HOST" juju will deploy when that host is in commissioning or some other non-ready state" [Critical,Triaged] https://launchpad.net/bugs/1073090 <jam> I didn't think this was specific to something we've changed, but I could be wrong. <w7z> looking <jam> w7z: from what I can see, we already have: 'available_nodes.filter(status=NODE_STATUS.READY)' <w7z> hm, we probably inherit the same dodgy logic for other constraints though <jam> so get_available_ shouldn't be allowing it if the status isn't actually ready. <jam> though maybe juju keeps poking at us for a specific node, and the status goes to READY before some action actually finishes? <jam> w7z: is it a possible data race between multiple juju requests to acquire a node, and the backend having a specific node ready? <w7z> hm, right, only accepting READY seems fine <w7z> I wonder if a symptom is being misdiagnosed and it's actually some other issue with using maas-name <w7z> when does READY get changed to something else... <jam> w7z: when it is "damn well read <jam> ready" <jam> :) <w7z> in acquire, which is before the api returns <jam> w7z: I think you mean in RELEASE <w7z> so, no obvious chance for maas to mislead juju here <jam> but note, any HTTP level request is inside a transaction that doesn't commit until it returns. <w7z> right, READY->ALLOCATED in acquire, ...->READY in release <w7z> I think the bug description is just wrong <jam> w7z: right, and the API level is "get_available(), node.acquire()" <jam> so we shouldn't return a node without it already being marked acquired. <w7z> the issue is that if you try to allocate more than one machine with the same maas-name constraint, juju will get no possible options for the second, and be stuck in pending <w7z> the fix being, don't use that constraint then <dimitern> jam, mgz: my branch is ready for review https://code.launchpad.net/~dimitern/maas/1.2-node-view-shows-kernel-params/+merge/133449 <jam> w7z: it sounds more like if you do a deploy before the machine has finished being ready, juju will just stop trying <w7z> I don't think he's actually confirmed that, though release does seem optimistic <jam> becoming ready <w7z> sending reboot then immediately putting in READY may not be good enough <w7z> I don't think that's what he's observing though <jam> w7z: from what I can pull out of the bug comments, the issue is that you do a 'juju deploy' and it doesn't do anything, because there are no nodes matching the constraint that are READY <jam> then juju doesn't wake up and realize once the machine is actually ready. <w7z> I think what he's seeing is just the known issue with trying to deploy with a set
{'channel': 'maas', 'month': 11}