#maas 2013-03-01 <roaksoax> rvba: howdy!! I was wondering if you guys already have a ETA for the multiple environments storage fix? I just wanna have on idea on when we can do the SRU <rvba> roaksoax: Hi. We just finished sketching out a plan for this. Now we're working on the code. End of next week seems like a reasonable ETA. <- allenap, what do you say? <allenap> rvba: Yeah, that sounds about right. <roaksoax> rvba: awesome, thanks! <roaksoax> rvba: so what are the reasons why trunk maas can't run in precise? <roaksoax> other than dependencies <rvba> roaksoax: there is only the dependencies issue AFAIK <rvba> The only reason we have 2 branches (trunk and 1.2) is because we added 2 new features in trunk that were considered "too much" to be SRUed. <roaksoax> rvba: which ones are those? <rvba> roaksoax: offline operation and user commissioning scripts <roaksoax> rvba: ack! thanks <rvba> np <roaksoax> rvba: and kernel params too <roaksoax> rvba: what are the dependencies that differ? <rvba> roaksoax: I think kernel params have been backported to 1.2 <roaksoax> rvba: nah <roaksoax> rvba: what are the dependencies that trunk has that make it impossible to work on precise? <rvba> roaksoax: I don't know offhand <rvba> roaksoax: 1.2 and trunk are so similar that I even wonder how the dependencies differâ¦ <roaksoax> rvba: iirc bigjools did ssy that trunk wont eork in prrcise <roaksoax> rvba: ok this it what i'll do <roaksoax> rvba: i'll upload 1.2 to raring, then to precise-proposed and quantal-proposed before FF <roaksoax> rvba: once I do that, I'll upload trunk to raring <roaksoax> before FF <roaksoax> rvba: but then when your fix is really, will upload it to -precise-proposed and quantal-proposed <rvba> roaksoax: do we really have to upload now? <roaksoax> rvba: i', not really concerned with raring TBH <roaksoax> rvba: and the upload to -proposed will sit there for a while <roaksoax> rvba: and once it gets approved it will be there for SRU verification <roaksoax> so it is not really gonna be released right after upload <roaksoax> rvba: as far as raring, nobody will be upgrading current environments right? since raring is still in development <rvba> roaksoax: if yes, then I suggest you don't upload the last version. But instead r 1438 in trunk and r 1361 in 1.2 <roaksoax> rvba: the plan is to get multiple environments fix rather than a new feature for the SRU <rvba> Because we've changed the way the file storage works and that change will only be properly complete when the fix we're working on will be released. <roaksoax> smoser: ^^ <roaksoax> rvba: right but upgrading from 1363 to 136X should be fixed right? <rvba> roaksoax: what do you mean? <smoser> ugh. <smoser> roaksoax, unless daviey or sru team told you otherwise, just upload something to raring. its a pre-relesae snapshot. thats fine. it has bugs. <roaksoax> rvba: so if I release 1363 (1.2) to raring, then I release 136X (which contains your fix), then upgrading will fix the problem right? <smoser> whether or not we wait for the multiple-environment bug fix or not to upload to -proposed (quantal and precise) is un-related. <roaksoax> smoser: yeah that's what I'm planning to do <smoser> and i'd just upload a trunk snapshot to raring. <smoser> somethign that passed basic test. <roaksoax> smoser: i'll upload lp:maas/1.2 to raring, then to {precise,quantal}-proposed, and since it is gonna sit there for a while we'll wait for the bug to be released in trunk and update -proposed . But in the meantime, upload a trunk snapshot <roaksoax> do I make sense? <rvba> roaksoax: it will, but it you use 1363 you might encounter serious bugs. <smoser> why do you bother uploading maas/1.2 to raring ? <roaksoax> rvba: yes which is why it will be sitting in -proposed :) <smoser> ugh. <smoser> don't upload something to -proposed that we're not happy to release. <rvba> +1 <rvba> The bug we're fixing is really ugly. <smoser> but i'm do not think that this maas multi-user thing is significant. <smoser> no one is using that <smoser> and its not a regression <rvba> It is. <smoser> it is a regression from something in ubuntu ? <roaksoax> ok then I'll upload trunk to -raring <smoser> rvba, ? <roaksoax> trunk will be released as 1.3 <roaksoax> and SRU will have 1.2~bzr136X, which is nowhere in the archives <smoser> wait to see if rvba says this is a regression, but otherwise, i'm fine with that. <rvba> smoser: this is really tricky tbh. * rvba thinks <smoser> "tricky" isn't answering the question :) <roaksoax> if it gets rejected by the SRU team because we are backpoirting a new upstream which is nowhere in ubuntu, then we are screwed :) <rvba> smoser: That's why I need to think :) <smoser> roaksoax, honestly, if it does, then that is silly. <smoser> because, as you've suggested, there is a very stupid workaround for that process requirement. <roaksoax> smoser: well I came across a situation like that long time ago and was told "you can only backport something that is in the archive" <smoser> roaksoax, then if you want to, upload a 1.2 to raring. i'm fine with that too. <smoser> but i tihnk that it wont be a sticking issue on sru <rvba> smoser: I think it's a regression. If you have 2 users, you'll definitely see a regression. <smoser> remember, you're also not allowed to entirely new packages <smoser> or features. <smoser> rvba, you see that if you have 2 users on current quantal mas and you upgrade to 1.2 ? <smoser> is that what your'e saying ? <rvba> Yes. <smoser> if only we had unit tests <smoser> then we'd neer see bugs like this. <smoser> roaksoax, i say hold off on it then. i dont want to upload a regression to sru. <roaksoax> smoser: right but this time we got a TB exeption <smoser> but imo upload whatever you want to raring. we're already in major exception mode. <smoser> this little thing is a very minor exception. <rvba> smoser: this problem is not because of the lack of unit tests. It;s because we had to rush something to meet a crazy deadline at some point, and now we're paying the price for it. <roaksoax> i will just upload trunk to raring <smoser> and doing the dance to avoid that exception will cause you to make another exception (FFE) <roaksoax> i'll continue as suggested, upload trunk (1.3) to raring, and SRU 1.2 <smoser> roaksoax, right. but dont bother sruing without the fix for bug 11239986 <ubot5> Error: Launchpad bug 11239986 could not be found <smoser> 1123986 <roaksoax> smoser: i won't SRU until it gets fixed <rvba> roaksoax: why don't you upload 1.2 revision 1361 and trunk revision 1438 ? <roaksoax> rvba: 1263 has IPMI fixes <roaksoax> err 1363 <rvba> The problem is only that right now, we're half way through fixing the multiple env bug. <roaksoax> and I need to make available those IPMI fixes to raring too <roaksoax> but we need tests for them <roaksoax> so that will take me couple days maybe <roaksoax> and then i'll upload trunk to raring <roaksoax> wait for your fix <roaksoax> and upload 1.2 tyo -proposed <rvba> Ok, I take it no one is using raring in prod now so it does not really matter. <roaksoax> exactly <rvba> Just don't upload the current tip of trunk or 1.2 before next week's fix to an important place :) <roaksoax> raring is fine to have bugs so raring is not really a concern <rvba> Right. <rvba> So you'll wait until next week for the other uploads right? <roaksoax> rvba: yeah <rvba> roaksoax: ok, thanks. <roaksoax> rvba: no worries :) just fix that bug i'll take care of the rest :) <rvba> roaksoax: we're on it, you can trust me. <roaksoax> i do :) <roaksoax> you've never disappointed so I'm really not concerned :) <rvba> Good :) #maas 2013-03-04 <roaksoax> allenap: ping #maas 2013-03-05 <roaksoax> allenap: howdy!! so the cluster_host thing didn't really work unfortunately :( <allenap> roaksoax: Hi there. Because of the get_managed_interface() thing? <roaksoax> allenap: yeah, it wont get the address of the cluster <allenap> roaksoax: Look at the implementation of get_managed_interface(). You can use a similar approach to getting the address of the cluster, i.e. get any interface of the cluster. <roaksoax> allenap: right, but the thing is that it seems that if we are not managing any interface, it wont return the address <roaksoax> allenap: so from what I udnerstood, if we dont configure DHCP/DNS, then it wont return the address <roaksoax> but we still need itr <roaksoax> regardless of whether we manage it or not <allenap> roaksoax: http://pastebin.ubuntu.com/5588071/ <roaksoax> allenap: ack, i'll look into that <allenap> roaksoax: Yeah, get_managed_interface() won't work then, but I meant for you to follow its approach. Afaik, all interfaces are recorded. <roaksoax> allenap: yeah, the difficulty is finding out what interface it PXE booted from <roaksoax> allenap: but the difficulty is really finding out what interface it PXE booted from. <roaksoax> allenap: and thanks btw :) <allenap> roaksoax: Okay, so I could update lp:~allenap/maas/preseed-cluster-host to find an interface on the cluster that's on the same network as the node. I guess that would be better? <roaksoax> bigjools: let me know when around :) <bigjools> roaksoax: o/ <roaksoax> bigjools: howdy!! so I think i found an issue with maas and nodegroups :( <bigjools> go on ... <roaksoax> bigjools: so while trying to work on the cluster_host thing, I first did this:http://pastebin.ubuntu.com/5589024/ to basically obtain all the interfaces of a cluster, instead of the 'managed' interface <roaksoax> (extending what allenap did) <bigjools> roaksoax: that won't do that <bigjools> it's returning the first <roaksoax> bigjools: right :) <roaksoax> bigjools: i guess I should have said get any of the interfaces <bigjools> there's a better way of doing it as well :) <bigjools> but go on <roaksoax> bigjools: but anyways, that's not really the issue though <roaksoax> bigjools: the thing is that I installed 2 cluster controllers <roaksoax> one in the region, and one external <roaksoax> 1 first enlisted/commissioned node01 with external cluster in 502.619.2016. Then node02 with local (in region) cluster in 502.619.2016 <roaksoax> bigjools: and then I noticed this: http://pastebin.ubuntu.com/5589031/ <roaksoax> bigjools: both nodes seem to be in the same nodegroup, when they enlisted/commissioned in a different nodegroup <roaksoax> bigjools: note that the local cluster running in the region was registered to maas *after* the external <roaksoax> but I;ve tested the other way around (local cluster registered first, then external), and the adddress returned there is 502.619.2016. <bigjools> let me lool, I think your code is buggy <bigjools> look <roaksoax> this leads me to believe that all nodes are said to belong to the same nodegroup <bigjools> check in the DB <roaksoax> bigjools: http://pastebin.ubuntu.com/5589038/ <roaksoax> bigjools: unless my logic is incorrect, how else would I check to what nodegroup the node belogs to <bigjools> hmmm <roaksoax> bigjools: this is what I did to get the cluster_host btw: http://paste.ubuntu.com/5589041/ (in get_node_preseed_context) <bigjools> roaksoax: multi clusters were tested in the lab and worked ok, so either something regressed recently or your setup is not doing what you think it is <bigjools> are you sure your nodes are on separate lans? <bigjools> and cluster controllers <roaksoax> bigjools: my nodes are in the same lan, but they do register themselves with a different cluster controller. I tell it where to PXE boot from so I know they are pxe booting from different clusters <bigjools> roaksoax: are you sure? :) <roaksoax> bigjools: 100%. It is a controlled environment and DHCP is managed by libvirt, which tells it where to pxe boot from <roaksoax> let me re-test <bigjools> roaksoax: if you disable the DHCP that you think it is using, does it still boot? <roaksoax> bigjools: nope <bigjools> ok <roaksoax> bigjools: it is a clean environment with no DNS/DHCP being managed by maas <roaksoax> nor dhcp is running in any nore <bigjools> roaksoax: ok file a bug <bigjools> not sure we'll get to it right away quite frankly <roaksoax> right <roaksoax> this does affect the FPI though <bigjools> we're getting the FPI done <bigjools> roaksoax: what's the cutoff for raring? <roaksoax> bigjools: FF is this thursday :) <bigjools> yay <bigjools> when you say "final" ... <roaksoax> bigjools: so I'm planning on uploading and then just doing FFe's for what's important <roaksoax> bigjools: so yeah we have like 1 extra month to request FFe;s <bigjools> ok #maas 2013-03-06 <shang> bigjools: ping <roaksoax> win 18 <roaksoax> rvba: howdy!! <rvba> roaksoax: Hi <roaksoax> rvba: dont mean to bother you but any updates on the critical bug for multi environment juju <rvba> roaksoax: branch is about to be landed. I've just built a package with it (yes, *before* it's actually landed) and I'm testing it. <roaksoax> rvba: awesome!!! <roaksoax> rvba: so re on the nodegroup, if you havent eeen my comments yet, there are not managed intrfaces on any of the cluster. but that doesn't change the fact that a node belongs to a cluster/nodegrop <rvba> roaksoax: I've seen your comments. I've even replied :). We need to think about it a little. One of us will get back to you shortly about this. <roaksoax> rvba: cool thanks. either way we still have time for this one :) <rvba> Good :) <roaksoax> rvba: let meknow when he storage fixes and please <roaksoax> thanks :) <rvba> roaksoax: I will. <roaksoax> rvba: still around? <roaksoax> allenap: around? <allenap> roaksoax: Hello. <roaksoax> allenap: howdy! so I just saw a branch landing to trunk, does that contain all the fixes for the storage issue? <allenap> roaksoax: I saw your comment, and I'm thinking about it now. Is my latest comment in https://bugs.launchpad.net/maas/+bug/1085823 relevant? <ubot5> Launchpad bug 1085823 in MAAS 1.2 "The cluster controller detection algorithm considers all the interfaces (and not only the managed interfaces)." [Critical,Fix committed] <allenap> roaksoax: Yes, should do. <roaksoax> allenap: are you guys planning to backport that to 1.2? <allenap> roaksoax: We may need to update the documentation too, but that should be it for documentation. We have not finished QA yet though. <allenap> roaksoax: Yes. <roaksoax> allenap: ack! please let me know as soon as it gets backported so I can start preparing the packages <allenap> roaksoax: Should I wait until we're happy with QA in trunk first? <roaksoax> allenap: what ever works best for you guys, if you want to wait fro trunk QA first <allenap> roaksoax: I'm asking rvba, but I think he's gone for the day now. I'll prepare proposals for 1.2 anyway, so we can land quickly. <roaksoax> allenap: cool thanks. If they are ready that would be helpful for me as I could work on the SRU packages today, and have them uploaded tonight/tomorrow morning #maas 2013-03-07 <Bobby_V> Just a quick reminder - the NebOps Admins will be participating in a team outing tomorrow 11:00-18:30. The CI Engineers have graciously offered to staff the phones so our Huntgroups will be covered. We will return to normal coverage at 22:00 when 3rd Shift starts. As always, we will be available should an incident arise needed our attention. <rvba> roaksoax: Hi. We've fixed the bug. Code is landed on trunk and 1.2. allenap will give you a tiny text that we would like to see included in the changelog or someplace else where the user will be able to find it. <roaksoax> rvba: cool thanks <allenap> roaksoax: I've emailed it. <roaksoax> allenap: cool. i can include that in the changelog, but tht wouldn't be seen bypeoplewho just dist-upgrade <roaksoax> we do need to provide an upgradepath documentation which this should be specified <allenap> roaksoax: Right, I wasn't aware of that. Is that text sufficient, or do you need something with more depth to it? <roaksoax> allenap: is there and upstream readme,probablyt that's a better place <roaksoax> allenap: but for the upgrade path documentation should be enouhg,which is something we wil have to take care of in the upcoming days after this gets accepted for SRU <allenap> roaksoax: Okay. Let one of us know if you need some more text about this. <allenap> rvba: Are we happy now that the 1.2 branch is ready for SRU? <roaksoax> allenap: sure thing. thanks <rvba> allenap: very much so :) <allenap> roaksoax: What rvba said :) <roaksoax> allenap: cool thanks guys <roaksoax> preparing packages now <jderose1> so i'm just getting started with MAAS... i'm wonder where the code is that generates the ephemeral images like the ones here: http://maas.ubuntu.com/images/ephemeral/daily/raring/20130307/ <jderose1> is that found in the maas source tree, or elsewhere? <jderose1> smoser: is the code for the fast path installer available publicly? <smoser> jderose1, fast path installer is lp:~smoser/+junk/xinstall.maas <smoser> code that generates daily images is at https://code.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral <roaksoax> jderose1: https://code.launchpad.net/~andreserl/maas/trunk-fpi <jderose1> smoser: okay, thank you very much. so System76 interested in using maas to provision desktop systems, and it making that something useful to others well. and then internally, we plan to use the same foundations for system imaging at our distribution centers <jderose1> (also, thanks roaksoax) <roaksoax> jderose1: smoser's branch is what he initially did, my branch is the actual work to get it integrated with maas. However, this is very early support. We have only tested deployments of server not desktop though (for the FPI_ <jderose1> smoser: you're fast installer work caught our eye... for us, it's important to image a system as quickly as possible :) <jderose1> roaksoax: okay, thank you... i'll poor over it. i just jumped into this, so fire hose and all :) <smoser> jderose1, well, the fast path installer will really be targetted at putting a root filesystme tar file donw, and thne doing whatever necessary to make it boot. <smoser> the key is in having the tar root filesystem that you want ready to go <jderose1> smoser: right, that's the exact same thing we want to do, just it's going to be a bigger rootfs based on our desktop golden image <smoser> right. <jderose1> how does the idea of provisioning desktop systems with maas strike everyone? seems like stuff like provisioning a room full of workstations at a button push, enrolling them with landscape would be a good show :D <jderose1> right now i'm just trying to figure out where the common ground is, if this makes sense. obviously it would need to be done with zero impact on standard server provisioning <racedo> hi guys, question: juju deploy --constraints maas-name=node.domain is trying to deploy in a new non-existent machine that gets created <racedo> plus i get all the time ProviderError: No matching node is available. from the juju logs <roaksoax> racedo: and I'm guessing that if you don't use the .domain, only node it works? #maas 2013-03-08 <haitao> I have some questions related to setting up MAAS on virtual box, is this the right place? <lifeless> haitao: yes <racedo> roaksoax: no, I've seen what you say in the past (not specifying the domain in the maas-name constraint fixing it) <racedo> roaksoax: in this case it never matches and creates an additional machine that doesn't exist <AskUbuntu> Why doesn't the nova-cloud-controller charm properly set up openstack networking? | http://askubuntu.com/q/265489 <roaksoax> racedo: has the backport of django in ppa fixed your issue? <racedo> hey roaksoax <racedo> i did it before updating <racedo> roaksoax: i ended up not using constraints as i needed it up and running but i think i can reproduce it easily <racedo> i can try now if you want and have it 15 mins <roaksoax> racedo: nothing in the logs? <racedo> yeah, i'll show you <racedo> i need to rebuild it now because i destroyed everything <racedo> roaksoax: i did it today and it just works, the only difference is that i have updated python-django <racedo> roaksoax: so i fortunately cannot reproduce it... that's weird, yesterday i had the old python-django from the ppa anyway <roaksoax> racedo: that's probably it <roaksoax> racedo: the problem really was that a security update landed with the same ubuntu revision as the package we had prepared for maas, and since it hasn't yet been released, it obviously took the version of the security fix <roaksoax> so since Daviey already backported it, we should be good to go <roaksoax> rvba: howdy!! So I just saw two more branches were landed in lp:maas/1.2 <roaksoax> rvba: are those realted to the file storage thing? <rvba> roaksoax: Hi! Yes, these are cleanup branches. If you can include them in the SRU great, if not, that's not the end of the world. <roaksoax> rvba: ok cool :) <roaksoax> i think i might be able to <rvba> Cool :) <racedo> roaksoax: yeah i noticed, thanks for having fixed it so quick, maas-developers/stable is now the one we use <roaksoax> racedo: cool! the sru should be comming soon too <Bobby_V> REMINDER: At 11:00 CT, the NebOps admins will be departing for their team outing. Our huntgroup will be monitored by the CI Engineers. We will be having team fun until 18:30 CT. Our On Call admin will cover from then until 3rd shift is on line at 22:30. As always, should emergencies arise, we will be available to handle incidents/outages. <mross> Hi, I'm trying to setup maas-dhcp to pixeboot a cluster <mross> My nodes aren't picking up the dhcp server <mross> [mezeoadmin@maas]~â¯ service maas-dhcp-server status <mross> maas-dhcp-server start/running, process 2747 <mross> So it is running, but what would prevent the nodes from finding the dhcp server <mross> ?? #maas 2013-03-09 <physicalsrv> hello everybody #maas 2013-03-10 <mattrae> hi, when i destroy-environment and then redeploy to the same machines they aren't reinstalled. I have to delete the machines and re-enlist to get a fresh installation. <mattrae> i couldn't find a bug for this. sounds like a bug though <mattrae> i think whenever i terminate a machine, it should be reinstalled when that hardware is used again <bigjools> mattrae: I have seen one other person say this too, however I can't re-create it. Which version of maas are you using? <mattrae> hi bigjools i'm using 1.2+bzr1360+dfsg-0ubuntu1~ppa1 from http://ppa.launchpad.net/maas-maintainers/stable/ubuntu/ <bigjools> mattrae: which version of juju, and what do you see happening on the machines' consoles that's different to normal? <mattrae> bigjools: i'm using juju 502.619.2016+bzr618-0juju2~precise1. for example i do juju destroy-environment. then when i do juju bootstrap ubuntu is not reinstalled. i can do juju status and see my old environment <bigjools> sounds like a bug in juju? <mattrae> i see the machines being returned to 'ready' when i terminate the environment <bigjools> can you see the destroy request in the maas log? <mattrae> sure i'll check for the request in the log <bigjools> so the machines are ready, yet when you do a status, you see them in use? <mattrae> bigjools: i'm looking in maas.log and i see a number of these errors from around the time i destroyed the environment: "PermissionDenied: Not authenticated as a known node." <bigjools> ew <bigjools> can you paste the log somewhere for me plesae <bigjools> please <mattrae> the machines are ready, then i do juju bootstrap, the machine starts but doesn't reinstall. then i do juju status and i see my previous environment <bigjools> what does it do instead of re-installing? <mattrae> it appears to just boot the machine with whatever was on it previously <bigjools> what state does maas show it in at that point? <mattrae> since i see my old environment <mattrae> at that point its allocated <bigjools> the machine will do a local boot if anything goes wrong with the pxe boot <bigjools> so I suspect your PermissionDenied errors have got something to do with this <mattrae> here's the error i see a few times i the log. yeah i don't know whether it is related or not http://pastebin.com/zu16ETvW <mattrae> that error could be related to me trying to re-enlist the machines. re-enlisting doesn't seem to be working <mattrae> only worked for one node :/ <bigjools> yeah it's a metadata server error <mattrae> sounds like i need to reinstall maas :/ <bigjools> the metadata server IDs the requesting node so it knows which data to send it <bigjools> don't re-install, let's investigate <mattrae> ok cool :) <bigjools> ok this will take a while, sorry, but can you remove all your nodes, shut down maas, wipe your logs and start again <bigjools> then re-enlist, bootstrap <mattrae> now it seems that even though i deleted the machines from maas. then rebooted them, they aren't re-inlisting. only one node re-inlisted.. the rest just booted up with whatever they had on them previously <bigjools> send me the log <mattrae> i'll try rebooting them again <bigjools> then destroy-env and send me the log again <mattrae> ok sounds good <bigjools> this will get me a clean log <mattrae> bigjools: hrm, so i deleted all nodes from maas and shut down the machines. i power on one machine and it boots up with what it had previously and I get that same "permissionDenied: Not authenticated as a known node" error <mattrae> the nodes never show up in the maas web interface <bigjools> hmm <bigjools> so there's 0 nodes registered? <mattrae> yeah "0 nodes in maas" <bigjools> ok <bigjools> can you send me the pserv log <mattrae> sure, want the whole thing? the most recent message in pserv.log is 40 min ago <bigjools> oh <bigjools> darn <bigjools> that's odd <bigjools> you don't have more than one dhcp server on your network do you? <mattrae> would it make sense to try restarting the maas server? <mattrae> or is there a way to check the health? <mattrae> i see maas-pserv, maas-cluster-celery, maas-txlongpoll, maas-region-celery, and maas-dhcp-server are running <mattrae> i can try reinstalling maas too and report if i see this issue again <mattrae> normally deleting nodes works, so i wonder if something got corrupted <mattrae> deleting/re-inlisting works i mean <mattrae> nope <bigjools> when you boot the node, can you see it pxe booting from maas? <bigjools> or does it time out? <bigjools> it looks to me like it is not pxe booting, and the previous installation boots and tries to contact the metadata server with predictable results <mattrae> hmm i'll check <bigjools> the most obvious reason for that is usually that there's another dhcp server <mattrae> bigjools: ahh yeah looks like my vm's are set to boot from the hd. that is weird because i'm not sure how i would have got them deployed previously <bigjools> aha <mattrae> these are libvirt vms.. does maas set a node to not pxe once there is an installation? <bigjools> yes <bigjools> oh wait <bigjools> no, sorry <mattrae> or maybe it was pxeing because there was no installation <mattrae> cool, i should at least be able to set these back to pxe <bigjools> the tftp server gives a different config based on what state we think it's in <bigjools> quite likely, yes <bigjools> maas does not touch bios boot order <bigjools> you need to make sure pxe is first <bigjools> great, good luckj <mattrae> great, thanks for the help <bigjools> welcome #maas 2013-03-11 <carl_memsql> anyone around? I am looking for some help with finding a avahi boot image <carl_memsql> the url here is dead: https://wiki.ubuntu.com/ServerTeam/MAAS/AvahiBoot <carl_memsql> not sure if it is just a ubuntu live image that is preconfigured with avahi or if it's something else <carl_memsql> any help would be very appreciated <AskUbuntu> MaaS Enlistment through 12.10 installation CD fails (SIGKILL problem) | http://askubuntu.com/q/266510 <mattrae> hi, i have nodes that didn't get added to the dns zone file (maas 1.2). I add the records manually but they are removed whenever maas regenerates the zone file. i suspect there is a db table where the the zone file gets generated from that wasn't updated when the machines were installed. <mattrae> do i need to update the database? one suggestion was to rename the node and name it back, but these nodes are already deployed with services <mattrae> also sometimes bind is not being reloaded when the zone file is updated and a manual reload fixes that <mattrae> carl_memsql: the image that maas uses should have avahi. i believe you just need to set your domain to .local for it to work. i've found it to be kinda unreliable though <mattrae> oops he's gone <mattrae> bigjools, roaksoax: around? maas is intermittantly not adding records to the zone file. seems like the nodes must not be in the table that the zone file is populated from. each time the zone file is regenerated the same hosts are missing <mattrae> maas 1.2. other than that the nodes seem fine <mattrae> is there some table i can add those nodes to so that the zone file gets generated correctly? <AskUbuntu> Does MaaS have any log? | http://askubuntu.com/q/266629 <roaksoax> mattrae: what interface are you using? <roaksoax> for DNS/DHCP management <roaksoax> mattrae: is there anything on the logs? anything on syslog? (such as apparmor?) #maas 2013-03-12 <bigjools> mattrae: that usually indicates a dhcp leases parsing problem <mattrae> bigjools: thx, yeah i was seeing the leases file still show the hostname as maas-enlist for hosts that were deployed.. unless i was confused <bigjools> the leases file doesn't have hostnames, what do you mean? <mattrae> bigjools, roaksoax here's some of the different errors i've received. i think i saw a validation error regarding a mac address as well. http://pastebin.com/WKcBv9zt <bigjools> mattrae: that paste means nothing to me, I need to see actual logs <mattrae> bigjools: those are the exceptions raised. i had to write them down. i'll see if i can get better logs <mattrae> bigjools: regarding the hostname in the leases file, its 'client-hostname'. looks like thats sent by the client.. i'm just trying to find other symptoms that could possibly be related to why some nodes are not being added to the zone file. thats my main issue, and <bigjools> mattrae: the primary reason in my experience is that there's something funky in the leases file that prevents maas from parsing it <bigjools> the DNS entries are all driven from there <bigjools> so first check that your expected IP/MAC is in there and then check the logs <bigjools> the leases file format is fragile :/ <mattrae> ah sweet, so if i'm able to fix the leases file that may fix the zone file? <bigjools> no, it might be a parsing bug <mattrae> cool well that gives me some hints.. i'll look at the leases file <bigjools> or it could just be that the leases file is not getting parsed at all <mattrae> ahh ok, i'll keep that in mind when i look at it again tomorrow <bigjools> make sure that the celery log shows the job running <AskUbuntu> How to delete a MaaS node which is in 'commissioning' state? | http://askubuntu.com/q/266924 <mattrae> roaksoax: i'll try to get more logs from maas/libvirt/syslog today. right now we're working around the dns issue by manually editing the zone file to add records it forgot to add. if i can continue doing that I will to get the rest of the nodes deployed. <mattrae> roaksoax: we were running into additional problems deploying to nodes though, juju could not find machines that were 'Ready'. I tried different constraints and removing constraints. it could have been a typo so we're trying to add those nodes again today <roaksoax> mattrae: ok remember to clearout constraints as they seem to be permanent <roaksoax> or "global" <mgz> `juju set-constraints arch= cpu= mem=` should be enough if you're reusing a bootstrapped environment but don't have any services <mgz> ah, sorry, =any, not = <roaksoax> mattrae: ^^ <mgz> generally you want to do `juju destroy-environment` if you've made maas config changes, which will blow away most juju settings except those you have locally in ~/.juju/environments.yaml <mattrae> roaksoax, mgz: sweet that will help. i'll make sure there are no constraints <mattrae> roaksoax: right now we are enlisting a node and getting an exception of "one or more MAC addresses is invalid". I have a printout of the exception and the lease <mattrae> roaksoax: the maas interface shows that the node is declared. is this exception something to worry about? <roaksoax> uhmm if it got enlisted dont think so <mattrae> cool sounds good <mattrae> roaksoax, bigjools: i am leaning strongly on the issue of nodes not getting added to dns being related to the leases file. i see that the leases file has different info for nodes that are added to dns and nodes that aren't. i'll work on getting the leases file to examine <roaksoax> mattrae: that might be related to an issue in isc-dhcp <mattrae> there is a private lp bug opened to track the issue https://bugs.launchpad.net/ubuntu-advantage/+bug/1154184 <ubot5> Error: launchpad bug 1154184 not found <bigjools> roaksoax: hey <bigjools> when is the sru planned? #maas 2013-03-13 <roaksoax> bigjools: o/ <bigjools> yo <roaksoax> bigjools: yo! <roaksoax> whats up? :) <bigjools> should I ask what time it is there? :) <roaksoax> 1.10 am :) <bigjools> nice <roaksoax> i drank colombian coffee and cant sleep :/ <bigjools> I am only 14h ahead of you then :) <roaksoax> heh:) <bigjools> when's the SRU going out mate? <roaksoax> bigjools: i uploaded on friday <roaksoax> gonna ping ppl tomorrow <bigjools> ah it's in -proposed <roaksoax> (or later today) <bigjools> there's another fix to go in, sadly, but I think you know. It can wait for later anyway. <roaksoax> bigjools: i can still upload i think <bigjools> that'd be good <roaksoax> will look into that tomorrow/later today <bigjools> just need to check the "fix committed" stuff on 1.2 that is not milestoned <bigjools> I'll release the milestone once it's all in updates <roaksoax> ok cool * bigjools high-fives roaksoax <roaksoax> bigjools: yeah finally we are almost done \o/ <bigjools> it's been a long road ... <roaksoax> indeed <roaksoax> but almost there :) <AskUbuntu> Howto perform OpenStack upgrade from Essex to Folsom on juju maas | http://askubuntu.com/q/267256 <AskUbuntu> UBUNTU MaaS Region and Cluster Controller in One Box | http://askubuntu.com/q/267284 <melmoth> is there any doc on how to use /usr/sbin/maas-import-pxe-files in an environment not connectetr to the internet ? <melmoth> like, downloading the iso manually, and feeding them into maas ? <roaksoax> melmoth: no... you need internet connection. it is a requirement <melmoth> racedo, here i am :-) <racedo> hey melmoth <racedo> hi roaksoax melmoth is trying to setup an offline environment with maas using mirrors <melmoth> so, i have a problem, i m trying to use maas (1.2+bzr1360+dfsg-0ubuntu1~ppa1) on a lab not connected to internet <melmoth> i have used reprepro to mirror archives.ubuntu.com (main updates security restricted and universe) <melmoth> i also have mirrored in a separate subdir the ppa for juju and for maas (so i can install maas) <melmoth> now, i have maas installed, but when i start a node, it try to contact archives.ubuntu.com <racedo> melmoth: when enlisting the nodes, do they try to go online to pull the packages? <melmoth> yes <racedo> is it cloud-init configuring sources.list? <melmoth> i want to be able to tell the machines where are my repos, and not to use the real upstream one <melmoth> i have no idea <melmoth> i never had to add anything related to cloud init anywhere before <melmoth> btw, this is the step by step doc http://bazaar.launchpad.net/~pierre-amadio/+junk/c6100-isolated-maas-deployment/view/head:/README.txt <melmoth> basically just how to set reprepro <melmoth> i know some people workarouned this by trying to play with name resoltuion <melmoth> and have archive.ubuntu.com point to the local repo, but i m not sure this is a good idea. <melmoth> (plus i m not sure where to do that anyway ;-) ) <melmoth> i would rather have the preseed file changed so they use _my_ mirrors. <racedo> melmoth: the images may already contain this, we may have to modify them <racedo> check /var/lib/maas/ephemeral/precise/ephemeral/amd64/20121008 <melmoth> hmmm <racedo> hep <melmoth> that s a good idea <racedo> sudo mount -o loop disk.img <racedo> and check the sources.list <melmoth> yeh, i m trying that <melmoth> looks good <roaksoax> melmoth: its probably easier to makecloud init do the repo change <roaksoax> smoser: ^^ <melmoth> i dont know, i never used any mirror tool. I first try with apt-mirror, then i have been told to use reprepro because it ll be easier <melmoth> note that one of the trick also is, i want to have a mirror with all the ppa stuff needed. reason is, i dont want anything to do with injecting gpg key with ppa styff <melmoth> cause it ll be just fail too <melmoth> grumble, still trying to access security and archive repo <roaksoax> melmoth: i mean makr cloud iniy use the internal repo <roaksoax> melmoth: http://askubuntu.com/questions/144393/how-to-let-maas-cloud-init-client-select-internal-mirror <melmoth> oh <smoser> melmoth, you can probably accomplish this. but you might have to chagne the commissioning_user_data and the enlistment_userdata. <smoser> i *think* if you change the preseed that that will get refelected through and 'stick' after installation <smoser> doing this via proxy "should work" without any tampering from trunk now. <smoser> but i'm not sure about full mirror <melmoth> that seems to be for the old version of maas, is it not ? (the above ask ubuntu link) <smoser> (personally, i'd recommend proxy) <roaksoax> melmoth: (yeah but thats somethibg similar) <melmoth> i did try to change the preseed file in the generic file. <melmoth> without success <smoser> the link above is something that i expect is now fixed. <smoser> (as maas tells cloud-init to 'apt_prserve_sources_list: true' itself) <melmoth> even if you are behind a proxy, things fail (that s my personnal problem for 3 monthes now), as soon as you need a ppa. <melmoth> but anyway, in this particular case, the pre requisite is "completly isolated network" <roaksoax> melmoth: yeah ppa's behind proxy indeed fail but there are work arounds <smoser> melmoth, you can proxy ppa for sure. squid-deb-proxy needs tweaking to make it do that. but you can do it. and it is not 100% offline, but its pretty good. <melmoth> any link that explain the workaround for hte ppa behind a proxy ? <melmoth> my ppa & proxy issue is https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1089389 <ubot5> Launchpad bug 1089389 in software-properties (Ubuntu) "juju bootstrap fail behind a proxy when a gpg key must be imported" [Critical,Confirmed] <melmoth> the "injecting gpg key manually" workaround is not good enough, it must be done withing the zookeeper node during bootstraping. <roaksoax> melmoth: yeah you need to add late commands to add the ppa and stuf <melmoth> like, changing th preseed file so it inject the gpg key manually ? hmmmm <melmoth> gosh i hate preseed :) <roaksoax> yeah <racedo> didn't we hit something similar when on site last time? <roaksoax> so that way it is done automatically <roaksoax> racedo: we did <melmoth> racedo, we currently have such a problem <melmoth> that s the whole reason why i m trying to build stuff in an isolated env. <racedo> and we injected the gpg keys in the charms? <roaksoax> racedo: i think just in late_command <racedo> ok * roaksoax looks for it <racedo> and melmoth for the ppa issue, would this be doable: mount -o loop disk.img, chroot to it and apt-add-repo? <roaksoax> racedo: no need <melmoth> i dont know <racedo> ups not really <melmoth> you know, i have no real understanding of what maas is actually doing. <roaksoax> you can modify enlist_userdata and commissioning-user-data <melmoth> so, if it try to add the repo anyway, even if it still there, may be it ll fail. may be not...who knows. <melmoth> roaksoax, and those are preseed snippet, right ? <roaksoax> melmoth: its really not maas itself <roaksoax> melmoth: it is external factor that we have to workaround in maas <melmoth> i dont undertsand what are enlist_userdata and enlist file. Looks mor elike bash scrpit than pressed <melmoth> so i m not sure if i need to add d-i stuff in it or echo > stuff <<EOF <roaksoax> melmoth: enlist_userdata is cloud <roaksoax> melmoth: enlist_userdata is cloud init's user data <roaksoax> for enlistment process <roaksoax> /etc/maas/commissioning-user-data is cloud init's user adata for commissioning process <melmoth> ok, as we say in france "c est comme le port salut, c est marquÃ© dessus" (it s written on the name :) ) <melmoth> now, i need to read doc about cloud init user data then. <melmoth> ahh, yaml... <melmoth> i need to read doc about yaml :) <roaksoax> racedo: great!! The Paste you are looking for does not currently exist. <roaksoax> Return to the Pastebin <roaksoax> racedo: all the stuff was in pastebins and now they are gone <racedo> oh ok <roaksoax> all the tweaks we did in colorado <racedo> i think we just include it in one charm <racedo> not sure how <roaksoax> racedo: yes, we included it for the cloud archive <roaksoax> racedo: did you or negronjl backed that up? <racedo> i didn't negronjl was the charms person on site that week <racedo> let me quickly ping agy <roaksoax> racedo: already did :) <racedo> oh ok :) <AskUbuntu> Could not internally obtain zookeeper handle | http://askubuntu.com/q/267409 <roaksoax> racedo: ping <roaksoax> err <roaksoax> sorry <roaksoax> rvba: ping <roaksoax> rvba: ping <rvba> roaksoax: hi <smoser> melmoth, :-( forgot about that bug. <melmoth> ah ah, enlisting seems to go a bit further. <melmoth> gosh, please tell me i dont have to mirror the sources too <melmoth> yeahhh, got one node declared \o/ <roaksoax> melmoth: so in your proxy environment, could you try things like this: sudo apt-key adv --recv-key --keyserver keyserver.ubuntu.com --keyserver-options http-proxy=http://502.619.2016:8000/ 93EE8CC5 <roaksoax> melmoth: or sudo apt-key adv --recv-key --keyserver keyserver.ubuntu.com 93EE8CC5 <roaksoax> in the first you'd need to specify the proxy you are using <melmoth> it ll work, it s actually written in th ebug if i remember correctly <roaksoax> melmoth: ok so something like that is what you need to add <roaksoax> to a late_command <roaksoax> in the preseed <roaksoax> and manually add the ppa repository <melmoth> ahh, yes. that could be an option, in a late command, that, or, as i was just reading cloud init stuff apt_sources: in the cloud init config file <melmoth> i will try it out next time i have nothing else to do :) <melmoth> hopefully next week <melmoth> depends how this isolated stuff turns out <melmoth> yes... ready state \o/ <melmoth> hmmm, it does not like my mirror :-( <melmoth> when i bootstrap, the zookeeper node download the Release file from my mirror <melmoth> there is a Codename: precise in it. <melmoth> i can wget it all rightr from the console of the zookeeper node, but still, it complain the mirror does not support the precise release. <roaksoax> melmoth: are you telling the preseed to use the mirror? <melmoth> yes, it does it all right, i see it downloading the release file from my mirror <roaksoax> melmoth: did you remove the proxy option from the preseed? better ocmment it <melmoth> hmm. not sure <roaksoax> melmoth: you'll probably need to do that <roaksoax> melmoth: so installations don't access the proxy in maas, and use the mirror directly <melmoth> not sure what to do, there is a /usr/share/maas/preseeds/generic file that define a proxy marco or something <melmoth> hmm, actually, i already edited this file, to put my mirror ip in it. <melmoth> i have a strange feeling i have no idea what i m doing. <melmoth> let s remove the "d-i mirror/http/proxy string http://{{server_host}}:8000/ <melmoth> " line <melmoth> just in case. <roaksoax> melmoth: yeah <melmoth> now, it seems to complain because my mirror has no /ubuntu/dists/precise/Release.gpg <roaksoax> melmoth: maybe you need to correct the path for the mirror <roaksoax> in the preseed <roaksoax> and that would be it <melmoth> well, i do not have any file ending with .gpg in my repo. <roaksoax> then that's probably whhy <melmoth> i think i start to understand the "VerifyRelease: blindtrust" option i used in reprepro <AskUbuntu> How to PXE boot a virtual machine from another virtual machine which are in same virtual network? | http://askubuntu.com/q/267435 <mross> Hi, I'm trying to automate my MAAS cluster with the maas cli. I've written a script to do this but I can seem to get the maas nodes to allocate to a user after they've been released. <mross> My basic algorithm is: <mross> maas-cli login mezeoadmin <mross> maas-cli mezeoadmin node release <system_id> <mross> â¦ Do this for all the nodes. <mross> maas-cli mezeoadmin nodes accept-all <roaksoax> mross: accept-all is only done when they are first enlisted in MAAS <mross> ssh -o "StrictHostKeyChecking no" ubuntu@node "sudo reboot" <mross> Then I'm going to the web interface and pressing 'start node' for each node that has been released. <mross> This seems to work, but I would like to automate it completely off the cli. <mross> What am i missing. <mross> ?? <mross> @roaksoax OK. So I can remove that. <roaksoax> mross: 1. why don't you use juju ? :) <mross> Because I haven't written a juju plugin for our service yet. <mross> I'm just getting familiar with maas right now. <roaksoax> mross: 2. when you 'Start node' on the web UI, they get turned on to install ubuntu on them <roaksoax> mross: you could use juju deploy ubuntu <roaksoax> mross: and that will simply install ubuntu on the nodes <roaksoax> and on topof that you could deploy your service <mross> So juju deploy ubuntu to nodes that are in a ready state will install ubuntu on nodes that haven't been allocated? <mross> Or do I have to allocate them to a user then deploy? <roaksoax> mross: nodes are allocated when a user wants to install ubuntu on them, in the case of juju, it will take nodes that are in 'Ready' state to deploy ubuntu on them and install a service <mross> OK thanks. Is this a decent workflow for what I'm trying to do. <roaksoax> mross: so in order words, you do not allocated a machine to a user, so the user can install ubuntu on it <roaksoax> however, the user can use a machine from the Ready pool, once it selects that machine to be used, the machines becaomes allocated to the user <mross> Correct <roaksoax> mross: personally, I'd recommend you to use juju <roaksoax> makes things so much simpler <roaksoax> if you need to automate the dpeloyment of your service, you'll need to write a charm that installs and configure your service <roaksoax> a juju charm <mross> Any good examples you can send me for my particular use case? <roaksoax> mross: so for example, you could use: juju deploy wordpress, juju deploy mysql, juju add-relation wordpress mysql <roaksoax> that deploys wordpress and mysql in different nodes, and adds a relation between them saying that wordpress will use the mysql node as its backend db <mross> OK. I think I know where you're headed here. <mross> And where I need to go. Roak thank you very much! <roaksoax> mross: more infomrmation in maas.ubuntu.com and juju.ubuntu.com :) <bigjools> roaksoax: re. bug 1154805 <ubot5> bug 1154805 in MAAS "Juju/MAAS does not provide feedback when there are no available nodes are found" [High,New] https://launchpad.net/bugs/1154805 <bigjools> I am pretty sure this is just a juju bug <bigjools> see the recent thread on juju-dev where I suggested that the user experience is poor <roaksoax> bigjools: yeah i was thinking that, but just to make sure I included maas <bigjools> roaksoax: it's a provider bug in juju <bigjools> or perhaps a core bug <bigjools> I dunno <roaksoax> bigjools: yeah that's why I wasn't sure either <bigjools> but maas returns a 403 IIRC <roaksoax> bigjools: maas does show an error <roaksoax> bigjools: but juju doesn't <bigjools> roaksoax: in the log? <roaksoax> bigjools: yeah maas shows NodesNotAvailable: No matching node is available. <roaksoax> bigjools: but for juju ois just as like nothing happened <roaksoax> bigjools: i'm more concerned on the contraints bug though <roaksoax> bigjools: ppl have been complaining about constraints in customer deployments <bigjools> ah no it returns httplib.CONFLICT <bigjools> roaksoax: it's definitely a juju bug <bigjools> it has no notion of not being able to get a node, it just waits for one forever (I think it retries) <roaksoax> yeah maybe, TBH i didn't look much into it since i needed to get that stuff working <roaksoax> feel free to retarget accordingly <bigjools> done :) <roaksoax> bigjools: i think this is more important though: https://bugs.launchpad.net/maas/+bug/1154803 <ubot5> Launchpad bug 1154803 in MAAS "Default constraints prevents the allocation of nodes if they are equal to the HW." [High,New] <bigjools> looking <bigjools> roaksoax: well that sucks :/ <roaksoax> bigjools: yeah i was thinking might be related to node having 1, and the constraints 1.0 amybe? <bigjools> roaksoax: I don't know who will fix that <roaksoax> bigjools: mgz? :) <bigjools> it won't be us I'm afraid, we're super busy on another deadline <bigjools> roaksoax: should be easy to fix if you write a test to re-create it :) <roaksoax> bigjools: sure, I'll try to do it this week, im packed getting other stuff to work * bigjools hears you <bigjools> roaksoax: FWIW, all of my team are with me here in Brisbane next week <bigjools> in case you expected them to be around to ping :) <roaksoax> bigjools: hehe ok :) #maas 2013-03-14 <shang> bigjools: ping <bigjools> shang: hi <shang> bigjools: hey, had a quick question about the status on the maas backporting to 12.04 <bigjools> that's up to roaksoax <shang> aha! <roaksoax> shang: i uploaded to -proposed, it is up to the SRU team <roaksoax> to review <shang> roaksoax: ah, great! thanks for the info <roaksoax> np * roaksoax sleeps <roaksoax> night <bigjools> nn roaksoax <AskUbuntu> Limitations of Ubuntu Cloud Infrastructure with respect to OS? | http://askubuntu.com/q/267700 <roaksoax> robotfuel: howdy!! <roaksoax> err <roaksoax> sorry <roaksoax>
{'channel': 'maas', 'month': 3}