#maas 2012-05-29 <czajkowski> that was fast :) <czajkowski> ubuntulog2: welcome and logging <Soekris> I can't run juju bootstrap. I get the error http://paste.ubuntu.com/1012908/ http://paste.ubuntu.com/1011995/ the error message and the juju config <allenap> Soekris: I think the maas-oauth line in your juju config looks wrong. Try removing the leading ${ and the trailing } <allenap> Soekris: Do you know how they got there? Was that line presented to you that way? Even if it was a mistake I'd like to know how it happened so that we can adjust the UI. <Soekris> allenap: i've followd the https://wiki.ubuntu.com/ServerTeam/MAAS/Juju <Soekris> allenap: thanx 2012-05-29 13:55:30,124 INFO 'bootstrap' command finished successfully Thas nicer :D <allenap> Soekris: Ah, that's interesting. The token ${maas-api-key} is meant to be replaced in its entirety. I'll have a think about how that could be better represented. <allenap> Soekris: I've updated the docs so hopefully you'll be the last to get stuck on this. Sorry about that, and thanks for letting me know. <Soekris> allenap: if i can get it smooth to work i will try to make a youtube movie. som people can pause and run. <allenap> Soekris: That could be very helpful. <allenap> Hello rvba! <rvba> Hello allenap! <Daviey> ooo, hello chaps! <Soekris> next problem http://paste.ubuntu.com/1013374/ i have done ssh-kegen -t dsa copied de id.rsa.pub in to the dashboard <Soekris> hello i'm staring from skratch. When i boot with a unknow server. With image must i chose ? maas-precise-arch-commision ? t <Daviey> Soekris: no <Daviey> Soekris: just let it timeout <Soekris> haha patience something i doen't know :D <Daviey> Soekris: 20 seconds? :) <Daviey> You can just go down to maas-enlist if you are really impatient <Soekris> haha thanx. <Soekris> Yes i got 1 blue node :D <Soekris> juju bootstrap did work. but juju status didn't work. I got server refused mesages http://paste.ubuntu.com/1013772/\ #maas 2012-05-30 <allenap> Morning all. <rvba> Morning allenap. <rvba> allenap: do you have time for a quick pre-imp call? <allenap> rvba: OTP, but after that sure. <rvba> allenap: cool. <allenap> rbasak: Wow, long email :) I look forward to reading it. <Soekris> juju bootstrap did work. but juju status didn't work. I got server refused mesages http://paste.ubuntu.com/1013772/ <Soekris> I don't know where to look futher <allenap> Soekris: How long after bootstrap did you try this? Bootstrap can take a long time (it has to install Ubuntu for one thing). <Soekris> allenap: is there a command to see of is finished ? <allenap> Soekris: juju status ;) <allenap> Soekris: It's been a while since I tried this, so I can remember exactly what to expect :-/ <Soekris> i have installed a new virtual server so everty time each step goes smoother :D <allenap> Soekris: Have you tried ssh'ing to rhonda96@example.net? <Soekris> yes that wil work <Soekris> in the verbose it looked that er was missing a kind of service <Soekris> I will try it from skrath with more pacient :D #maas 2012-05-31 <jtv1> bigjools: still not seeing any decent way to test my virsh power script, or to make it quite trivial. <jtv> One thing I could do is allow the caller to override the virsh executable it uses, so that a test can inject âechoâ instead. But that leaves baggage in normal execution paths. <jtv> I could make it a power parameter, I guess. <bigjools> jtv: sorry was caught up with things. I don't think you can test the scripts. <bigjools> the approach I took with the wol stuff was to test the templating and to test that the script returns with a 0 code. <jtv> Thanks â I've got some things I can do now. <bigjools> jtv: since the scripts are intended to be customised, I think unit testing them is not useful <bigjools> they should be QAed instead <jtv> I'd like to know that at least it makes some kind of syntactic sense to the shell. <jtv> Which I can, actually, test to some extent. <bigjools> jtv: exit code! <jtv> Alas, no, not that easy. <bigjools> not for all, no <bigjools> and we really don't want to start up VMs <bigjools> so like I said, I'd really leave it to QA <bigjools> this is why I made this level of separation <jtv> Well there's one thing I can do in tests that makes it not start up VMs and yet exercises most of the script. <jtv> One simple thing, that is. Complicated things can do more, I'm sure. :) <bigjools> :) <jtv> And there. Just found a bug thanks to my test! <jtv> bigjools: I was just saying that we could stay on and chat about the anonymous-metadata requirement <bigjools> jtv: ah ok I have a call with gavin first, will call you right after <jtv> OK <rvba> allenap: I see you've got a call now, so just ping me when you will be available to talk about this "migration" problem. <allenap> rvba: Cool, ta. <bigjools> jtv: ok, wanna call? <jtv> bigjools: otp <jtv> w/someone else * bigjools is hurt <bigjools> you said you'd only ever otp me <jtv> uh-oh <bigjools> jtv: I need to head off - maybe you want to talk to Daviey about the requirements for this? <jtv> Yes, I think I will. <Daviey> o/ <Daviey> jtv: <jtv> Daviey: Hi. It's about the anonymous metadata accessâ¦ I wanted to be clear about the purpose. Is it _just_ for debugging? Because several of the things that were said sort of suggested that it might be needed for commissioning. <jtv> Which I hope is not the case, really. <jtv> But if it's for debugging, we might be able to require run-of-the-mill UI admin authentication. <Daviey> jtv: so.. <Daviey> jtv: We have some hardware that can't boot as maas can expect to.. <jtv> I see. This is indeed entirely different from the use-case we had before. <Daviey> Therefore, for 'demo / testing' if we can avoid having to push data to the node, we can get a working setup <Daviey> jtv: yeah, seems to fit the same model tho? <jtv> Wait â I'd like to comment on those two lines separately. <Daviey> ok <jtv> About âseems to fit the same model,â I think admin auth makes more sense _for debugging_ than opening up the whole thing â whereas for difficult hardware it may not be very user-friendly. <Daviey> jtv: well, injecting a known user/pass is MUCH easier than a generated on demand oauth key <jtv> The other thing is: what do you mean by âavoid having to push data to the nodeâ exactly, so that I understand correctly? Do you mean that we don't want to push the credentials for the metadata service to the node through its boot params? <Daviey> So.. using admin auth does help. <jtv> We could probably have a separate key for this. <Daviey> jtv: we don't have access to boot params at deploy time. <jtv> But it's a security hazard if it's open in production. <Daviey> Crappy hw <jtv> But then how does the node even know where to get its metadata? We push that in the same way. <Daviey> jtv: yeah, this is for debug / demo (closed network) workflow <Daviey> jtv: Yeah, knowing where the metadata service is = known constant <Daviey> It's really injecting runtime generated auth is the problem <Daviey> (although, could be really smart and use avahi for auto detecting where the metadata service is... but really.. overkill.) <jtv> I don't suppose we could have a single, shared key and differentiate by MAC address? Bit of a hack, and only good for a limited class of networks. <Daviey> hmm <jtv> Then again, if you're running in the demo configâ¦ <Daviey> the mac address isn't normally exposed in the http request. <jtv> Argh! How does the node even know how to identify itself? It doesn't receive its system_id either! <Daviey> So.. the MAAS server would need to arp <jtv> Maybe the node could do it. <Daviey> X-FORWARDED_FOR: $(hostname) .. seems sane? <jtv> How does it know its hostname at that stage? <Daviey> X-Forwarded-For: rather. <Daviey> jtv: well.. when using the model of dhcp allocated hostname.. MAAS is informed of the hostname, and the node knows it <jtv> DHCP-allocated hostname? Isn't that the node telling the DHCP server what hostname it wants? If so, it'd have to know first â quod non. <Daviey> jtv: hmm.. there are two models.. MAAS controls dhcp, and an existing dhcp <Daviey> this fits existing dhcp quite well. <jtv> Ouch ouch ouch you want to make it rely on a given DHCP setup as well? That's just making things worse & worse. <Daviey> it's not all that bad IMO <Daviey> remember, this is a non-prodcution setup <Daviey> jtv: to contrast, this is how openstack does it.. https://github.com/openstack/nova/blob/master/nova/api/ec2/__init__.py#L253 <jtv> So you're talking about setting up a DHCP server with prepared leases with hostnames, for demo purposes? It feels a lot like designing a whole feature specifically for just one demo. <Daviey> jtv: no, this also fits the debug model quite well. <jtv> Only if you have the hostnames. <Daviey> i can do an out-of-band install of ubuntu server, apt-get install cloud-init, providing i installed with the hostname MAAS expects, we are GOLD <Daviey> right? <Daviey> jtv: the other solution is to do a reverse dns lookup based on the ip address, and post that.. That caters for MAAS controlling the dhcp <Daviey> client side ^ <jtv> But why not have the client look up the mac address it uses for the metadata access? <Daviey> also valid <jtv> In fact it could just loop over its interfaces until it found a hit. <Daviey> jtv: technically, i'd say mac address lookup is more insecure :) <Daviey> but security isn't an issue here <Daviey> jtv: yeah, mac address seems reasonable :) * jtv refreshes his memory on that topic <Daviey> arp -na <jtv> No I mean, ISTR MAC addresses were significant somehow to the EC2 metadata service. Let me see if it matters to rhonda96@example.net. <jtv> (As you say, this is identification, not authentication â and yes, we'll still need to worry about how to fit this in with actual authentication) <Daviey> jtv: nah, mac address isn't interesting to the meta data service afaik. <jtv> You're right. We ditched that. <jtv> Still, it's one of the few things the node does know when it starts commissioning. <jtv> Daviey: how would we do enlistment on this hardware? <jtv> (So I get a picture of what process we'd be working towards) <Daviey> jtv: using the cmd line tool, maas-enlist <jtv> OK so MAAS knows about its MAC addresses anyway. <jtv> Will it need to have a custom(ized) metadata client? <jtv> I guess it will â hardcoded base metadata service URL <jtv> And it can embed its MAC address in the URL as well. <Daviey> jtv: sounds good to me. <Daviey> i think it will be a wrapper around cloud-init TBH <Daviey> running cloud-init post-boot isn't as straight forward as calling a command, sadly <jtv> Daviey: I think we'd have to make it a whole separate http tree. Maybe a hidden /metadata/<version>/node/<mac> <Daviey> jtv: hmm, mac can't just be passed as a parameter through urls.py to the same function which defaults to None, and cecks django settings if METADATA_MATCH_MAC = True ? <Daviey> ahh.. unauth'd url <Daviey> frack <jtv> Yeah. Want separate URL anyway, I think. <Daviey> would be nice to be able to reuse the same code.. <Daviey> but i have every confidence you'd do the right thing :) <jtv> Well you have to get the URL from a different source anyway, right? <jtv> So that might as well include a mac address. <Daviey> yuppers <jtv> And I take it the enlistment program sends all of the node's MAC addresses, in which case we don't even need to worry about which one we use. <Daviey> right! <Daviey> maas-enlist defaults to pushing all mac's <jtv> So then whatever code picks up the metadata URL from your static setup will append one of its mac addresses as a path component, and presto. <Daviey> \o/ <Daviey> sounds like a plan <jtv> (And it'll skip the oauth bit, but that much was obvious) <jtv> I can add a setting for just the dev/demo configs. <Daviey> jtv: you rock my world. <jtv> Well let's wait until it works. :) <jtv> Daviey: I'll work it out in more detail tomorrow. Until then I remain, your loyal servant &c. :) <Daviey> jtv: thanks.. nn sir <jtv> nn! <cheez0r> hey folks, this isn't a dev question, but I'm struggling with MaaS/dnsmasq right now and I was hoping you might be able to point me in the right direction. <cheez0r> I'm trying to stand up a new MaaS, I add 11 nodes to the MaaS, but when I run juju bootstrap, I get an ssh error. When I run verbose, it is trying to connect to the hostname of the node, which fails to resolve. <cheez0r> Shouldn't MaaS be adding the hostname to cobbler to add to dnsmasq automatically when the node is finished commissioning? <cheez0r> The strangeness comes in where one of my 11 blades seems to resolve but the other 10 do not. <roaksoax> cheez0r: can you ping the bootstrap's node hostname from where you are running juju? <cheez0r> roaksoax: no. that's part of the problem. It should have automatically been added via cobbler when I commissioned the node. <cheez0r> I'm trying to understand both why the dns hostname adds aren't working and where I could manually add them <roaksoax> cheez0r: are you running an external dns server or are you using maas-dhcp? <cheez0r> using maas-dhcp <cheez0r> because of that I was under the impression that MaaS would add hostname entries for each of the nodes as they were commissioned. <roaksoax> cheez0r: is the machine where you are running juju, using the maas server as DNS server? <cheez0r> yes, should be <cheez0r> right- I'm following the howto at https://wiki.ubuntu.com/ServerTeam/MAAS/Juju#MAAS:_getting_started_with_Juju <roaksoax> cheez0r: make sure that the machine were you are running juju from is using the maas server as DNS server (you probably have to do it manually) <cheez0r> how can I do that- the node is set as 'ready' in MaaS, it's fresh out of commissioning. I'm running juju bootstrap from the MaaS node. <cheez0r> It looks like a resolution from the MaaS node issue, not from the node itself <cheez0r> and none of the nodes resolve from the MaaS node except for one, for whatever reason. <cheez0r> They were all added identically. <roaksoax> cheez0r: I think I've hit the issue before but can't recall how to fix it but just editing /etc/resolve.conf and nameserver W.X.Y.Z <cheez0r> right, except that I don't know the IP addresses MaaS has handed out to my servers to manually configure their hostnames <cheez0r> it's resolving via dnsmasq; I just need to add the hostnames to the dnsmasq configuration <roaksoax> cheez0r: that's added automatically, unless there's a bug in MAAS where it is not updating cobbler correctly <roaksoax> cheez0r: try: sudo cobbler sync <cheez0r> right, which I think is the problem <cheez0r> I've done cobbler sync about a hundred times so far <cheez0r> :p <roaksoax> uhmmm i'll try to reproduce as soon as I can to further troubleshoot this <cheez0r> Do you know where the cobbler or dnsmasq configuration is located for the hostnames? <cheez0r> I can try manually adding them and see if it gets me past this point <roaksoax> cheez0r: /var/lib/cobbler/cobbler_hosts <roaksoax> cheez0r: what does that file show? <roaksoax> cheez0r: does it show hostname/ip combination? <cheez0r> it's empty. <roaksoax> cheez0r: so if you do: sudo cobbler system dumpvars --name node-XYZ(of whatever node) | grep dns <cheez0r> no output returned <cheez0r> system not found <hostname> <roaksoax> cheez0r: so maybe MAAS is not setting the hostname <cheez0r> doesn't matter if I use the hostnames I specified or the default <roaksoax> cheez0r: you could try: sudo cobbler system edit --name node-XYZ --dns-name node-hostname <roaksoax> and then sudo cobbler sync <cheez0r> ok let me try that <roaksoax> and try to ping it by hostname <cheez0r> how can I get a list of nodes cobbler knows about? <roaksoax> cheez0r: sudo cobbler system list <cheez0r> that's really odd, they all seem to have the same mac address <cheez0r> names are node-<stuff>-d485645878c8 <cheez0r> the systems in that list seem to reflect the correct dns_name in the config it outputs <cheez0r> okay Im guessing this is a bug related to my specifying hostnames when adding nodes to MaaS <roaksoax> yes apparently so <cheez0r> let me delete and readd a node with no specified hostname and see what it does <roaksoax> i'd have to setup a physical cluster to troubleshoot <cheez0r> not asking you to do that, but thanks for the thought ;) <roaksoax> cheez0r: i know :) I just need to make sure it works well :) <cheez0r> well, I'm specifying amd64 arch and was specifiying a hostname in the format city_name-dc_name-enclosure#-blade# <cheez0r> might be throwing errors on all of the hyphens or some such, but I dunno <roaksoax> for hostnames? <cheez0r> yes <roaksoax> cheez0r: hostnames don't accept underscores, but does accept hypens <cheez0r> no underscores in the actual names <cheez0r> like paris-champselysee-enclosure1-blade12 <roaksoax> alright, yeah I think it might be realated to not updating cobbler correctly <cheez0r> well I've got one node re-commissioning with the default hostname node-<MAC> so we'll see if that fixes it <roaksoax> cool <cheez0r> no change- the newly recommissioned node with the default hostname is still not resolving. <cheez0r> cobbler_hosts is still empty <cheez0r> the newly recommissioned node still shows up with a funky name with cobbler system list <cheez0r> re-commissioning with architecture set to i386 to see if that fixes anything <cheez0r> no change, still doesn't resolve <Soekris> Hello I've run juju bootstrap. This was good. I have seen apt-get running. But juju status doens work. I get ERROR Invalid SSH key. ssh ubuntu@hostname works <Soekris> http://paste.ubuntu.com/1016722/ <Soekris> Where goes it wrong at my setup ? <cheez0r> Soekris: the hostname resolves? <cheez0r> or are you ssh ubuntu@IP? <Soekris> cheez0r: it resolves <Soekris> ubuntu@s1-cl1-maas works <Soekris> ssh ubuntu@s1-cl1-maas works <cheez0r> hrm interesting <cheez0r> I have the same SSH issue but mine is because cobbler isn't adding the DNS names for some rason <Soekris> I have configued dns and dhcp on a other server <Soekris> It's strange that something like so easy can be so difficult :D <cheez0r> agreed <Soekris> cheez0r: is it for production or for test purposel ? <cheez0r> kind of both <Soekris> you can quick make the entries in the /etc/hosts <cheez0r> yeah I think that'll be my workaround right now <Soekris> Strange one of the two servers are showing in the juju status. But how it's a MAAS mistery #maas 2013-05-01 <mwhudson> bigjools: hey, can you give me a clue how commissioning works <mwhudson> bigjools: is it a custom initrd you serve up? <bigjools> mwhudson: what's hard in the morning <bigjools> oh not that sort of clue <bigjools> mwhudson: yes, the images live in /var/lib/maas/ephemeral IIRC <mwhudson> oh hm, it mounts a rootfs over iscsi? or am i reading this wrong? <bigjools> mwhudson: that's the installer image <bigjools> but yes <mwhudson> oh hang on <mwhudson> i think i meant enlistment <mwhudson> that thing where you boot a node for the first time <bigjools> it uses the same image <mwhudson> ok <bigjools> as commissioning <bigjools> just different options <mwhudson> ok <mwhudson> what i was trying to find out (as well as learning a bit more about MAAS) was how the name of a enlisted node was determined <mwhudson> i guess ultimately it must come (in my case) from the hostname supplied via dhcp <bigjools> mwhudson: it's a uuid <mwhudson> sorry not name <mwhudson> fqdn <bigjools> mwhudson: from memory, if you don't have a domain on the name already it uses the cluster's name as the domain <bigjools> so if you have your cluster called example.com then you'll end up with node.example.com <bigjools> maas creates dns zone called example.com and ensures the node's name is in it <mwhudson> right, but where does the "node" bit come from? <bigjools> the name of the node, either the auto-generated one or whatever you change it to <bigjools> acutally it's not a uuid any more, there was a change to give it a random 5-char name <Dr{Who}> Q. I am trying to limit the physical connections to my metal. I have 2 ports but using 1 is better. I have it setup as a vlan tagged port on my switches and I have configured the onboard nic to pxe using vlan 10. now it connects and downloads the startup image but at that point it fails because eth0 is not on a vlan. Can i tell maas to make sure to setup vlan on 10 during the next step? <Dr{Who}> also for some reason it keesp trying to iscsi up to the trusted side of my maas server not the other side. I have tried to figure out how to make that perm I have been editing the pxe config files by hand and it keeps reverting back or finding new places where that wrong ip is used. <Dr{Who}> also running dpkg-reconfigure maas does ask for the correct interface ip but it does not do anything. When I rebuild using 'maas-import-ephemerals' it uses the other ip. <bigjools> which version of maas are you using? <Dr{Who}> it is ubuntu 12.04 LTS and what ever came with it. <Dr{Who}> im still a bit confused as to do I use LTS or do I run the latest stuff that just came out the other day in 13.what ever it was. <Dr{Who}> i understood the point of LTS is well it will be "supported" long term : ) <bigjools> Dr{Who}: the release in 13.04 is much superior <bigjools> 12.04 is basically broken, there's a pending SRU which brings it up to the same code as quantal <bigjools> you can use the stable PPA which has a more recent release, until the SRU is done <Dr{Who}> sounds like if I want to wait then 12.04 will be patched or I can use unstable I saw that someplace how to use that. Or I can just use 13.04. I vote 13.04 WAY too many problems and I am not doing production just testing anyway. Will 12.04 have all the latest stuff someday? because in production I like LTS <Dr{Who}> ok. Well anway I will try 13 out tomorrow. but I do hope all these goodies come to LTS soon. <bigjools> Dr{Who}: 12.04 will get an improved version in the near future <bigjools> we're just promoting some components to main from universe to be able to complete it <bigjools> but the version in the stable PPA is identical <bigjools> but 13.04 would be ideal, yes <mgz> allenap: how are manual uploads to maas file storage meant to work? <racedo> ping roaksoax <roaksoax> racedo: here <racedo> hey roaksoax <racedo> quick one, how does maas know during enlisting about the IPMI IP <racedo> i don't know where to look at in an installation where the IPMI settings are not being picked up <roaksoax> racedo: during enlistment maas runs a script that tries to detect ipmi user/password, IP <roaksoax> racedo: in queries the BMC for it <racedo> ok <roaksoax> racedo: so if the BMC might be returning 507-653-2812 for ip address <racedo> can i do it manually to debug why it doesn't work <roaksoax> racedo: /usr/share/maas/preseeds/enlist_preseed <roaksoax> racedo: yeah you can <roaksoax> racedo: first do this: https://lists.launchpad.net/maas-devel/msg00808.html <roaksoax> so you can debug the epemeral image <roaksoax> racedo: so during the enlistment process you can simply ssh into the image and prevent it from rebooting <roaksoax> racedo: and then you could run the script <roaksoax> racedo: unless you are thinking of something else? <racedo> ok, i was thinking about trying it from the maas node where the hardware is the same if possible <roaksoax> racedo: unless you want to install ubuntu on the machine and run the script independently <roaksoax> racedo: and you can too <racedo> yeah <roaksoax> racedo: so /usr/share/maas/preseeds/enlist_userdata <roaksoax> racedo: go to: add_bin "maas-ipmitool-autodetect" <<"END_MAAS_IPMITOOL_AUTODETECT" <roaksoax> and copy/paste that section in a script and run it with sudo <racedo> ok <roaksoax> racedo: what hardware is this? <racedo> this are the blade servers in the boston lab <roaksoax> racedo: what's their IP addresses? <roaksoax> racedo: hold on <roaksoax> racedo: don't given them to me <racedo> :) <AskUbuntu> Juju and MAAS: ERROR No matching node is available | http://askubuntu.com/q/289226 <roaksoax> dho/win 9 <TheChistoso> hello... <TheChistoso> i can't get maas to move a node from deployed to ready <TheChistoso> the machine looks like it's ready to go... <TheChistoso> but maas is showing the nodes as deployed only <bigjools> what are you doing to try and move it? <TheChistoso> the node is up and it said at the end that cloud init had completed <TheChistoso> i'm not doing anything <bigjools> so you enlisted it and then clicked "accept and commission" ? <TheChistoso> i thought that after it was done commissioning it would update the state itself <TheChistoso> yes -- and it completed all of that <TheChistoso> (on that note, though -- it wasn't fully automated -- I had to intervene to specify the file system partition layout) <bigjools> in commissioning it uses the same mechanism at the end of enlisting to signal to maas that it's done <bigjools> so this is odd <bigjools> does it work if you don't specify your custom partition? <TheChistoso> i didn't specify a custom partition <TheChistoso> i told it to use the guided partitioning scheme w/ lvm <TheChistoso> now the default is set to use precise -- i assume that's still okay <bigjools> what happens on the node's console when it commissions? <TheChistoso> everything appears to be fine during that phase <Marlinc> Shoudn't MAAS install Ubuntu on one of my hosts? <TheChistoso> Marlinc: yes <Marlinc> Because it did two reboots to add it to MAAS and one to set it up <Marlinc> But it didn't overwrite the previous install <Marlinc> Or is there something I dont get right <bigjools> maas only installs Ubuntu when something requests to use a node <TheChistoso> i've had a lot of problems w/ maas if you mess any of your configuration up. the only way i can get it to work cleanly is if i blow the entire thing away and start from scratch. it's a real pain... <bigjools> which version of maas? <bigjools> the one currently in precise is broken <TheChistoso> whichever the latest one shipping w/ raring is <bigjools> that one is fine <bigjools> I am surprised you need to start from scratch, that was only necessary in precise <Marlinc> So how would I make MAAS install Ubuntu? Just to see if it works <Marlinc> Or what would I need to do to see if it works <bigjools> Marlinc: bootstrap with juju is easiest <TheChistoso> there are lots of little gotchas and the documentation isn't coherent -- i've had to piece things together. e.g. generating an ssh key using RSA and a 2048-bit cipher and adding that to your account before commissioning... <bigjools> you don't need to add an ssh key before commissioning <TheChistoso> well there is documentation out that that says you do... <bigjools> before deploying, yes, before commissioning no <Marlinc> You need to add your SSH key in the web interface <Marlinc> Thats the only place right? <bigjools> Marlinc: you don't need to do that if you use juju <TheChistoso> minor feature request -- option for maas to generate a jujuu environments.yaml <Marlinc> That would be very nice lol <bigjools> TheChistoso: that will be available in the next release of juju <bigjools> it's not a maas problem <TheChistoso> not saying it is :D <bigjools> if you find problems, please file bugs <bigjools> otherwise nothing will get looked at <bigjools> and there's not always people around on here <TheChistoso> i understand (c: i write software for a living myself <bigjools> we're quite receptive to fixing problems <TheChistoso> so where should i start looking for answers? <bigjools> but the nature of this beast makes it hard to diagnose stuff <bigjools> answers to what? <TheChistoso> why my node isn't transitioning to ready <bigjools> I asked earlier what you see on the console <TheChistoso> i see a login prompt #maas 2013-05-02 <bigjools> if you alt-f8 (I think) you should see more output <bigjools> it takes a while <bigjools> it downloads some stuff from the repos <TheChistoso> it's been about 30 min. <bigjools> ok <bigjools> it should be around 5-10 <TheChistoso> alt-f8 didn't give me anything -- but then again, i'm using it over the bmc and the keystroke may or may not be getting across <bigjools> ok <bigjools> one sec <TheChistoso> i don't suppose you know the default username/password? <bigjools> there isn't one <TheChistoso> (let me guess -- password is generated?) <bigjools> but there's a backdoor image that you can use to commission with that has a user/password set up * bigjools looks for it <TheChistoso> well i can't ssh w/ the key i provided in my preferences <bigjools> no, you can't log in at all with commissioning <TheChistoso> it's past commissioning <bigjools> it's not - otherwise it would be ready <TheChistoso> it's commissioned -- it's deployed <bigjools> ... <TheChistoso> that is, the OS is installed, cloud-init reported as having completed <bigjools> ok let's get the terminology straight <TheChistoso> lol okay (c: * mwhudson was going to say :) <bigjools> you were previously at the "ready" state? <TheChistoso> i was at whatever the name for the state is when you've brought up the machine, maas recognizes it, and then i hit "accept & commission" (or "start & commission") <bigjools> that is, it enlisted, powered off, you clicked "accept and commission" and it powered up, did stuff and powered off again <TheChistoso> that ran -- the OS installed, it restarted, there's a login prompt, w/ some output indicating that cloud init completed <bigjools> wait <mwhudson> how did you tell the os installed? <mwhudson> it probably didn't <TheChistoso> i watched the os install <bigjools> on the second boot, did it power down? <bigjools> it is not installing anything when it commissions <bigjools> nothing usable for you anyway <TheChistoso> honestly, there were a few reboots and i'm not sure when it powered down and when it didn't <TheChistoso> yes, i know <bigjools> what state does it say that node has in maas now? <TheChistoso> it starts up and you throw something on there (enough to communicate w/ maas and provide it w/ information so the admin can review and decide to actually proceed) <bigjools> what state does it say that node has in maas now? <TheChistoso> and then when you accept and commission, it reboots, and at that point the OS is actually installed <TheChistoso> status shows "allocated to sysadmin" <Marlinc> But wait if I get it right. If I want MAAS to have another node I just boot it up ones from PXE and MAAS then handles it from then on? <bigjools> ok so your story of events is incorrect somewhere <TheChistoso> i'm telling you exactly what i saw <TheChistoso> (as i recall it, granted :D ) <bigjools> TheChistoso: if it says "allocated" you clicked something else <bigjools> can you "ssh ubuntu@<your node>" <bigjools> because you've commissioned it, it shut down, and now you clicked "start" <TheChistoso> yes -- i did forget that step <bigjools> right, so you have an installed node <mwhudson> so is it possible that the node is in fact ready for use? <bigjools> so ssh to the ubuntu user <Marlinc> My node can't be booted up from the network (at least it doesn't seem to work) or another way. What could I do about that? Could the node still work with MAAS <TheChistoso> i can't ssh into the machine <bigjools> Marlinc: without control of PXE/DHCP you cannot use maas <Marlinc> PXE and DHCP work <bigjools> TheChistoso: how are you trying to ssh in? <Marlinc> It cant start from poweroff state <Marlinc> Using WOL <Marlinc> Thats what I mean <bigjools> Marlinc: ok so just power up manually after maas requests a startup <Marlinc> Okay great <bigjools> (in that order, specifically) <bigjools> if you try to boot a node that maas doesn't think should be starting, it'll do weird stuff <TheChistoso> from the maas controller, logged in as "sysadmin" (w/ the same account name configured in maas as a super user and w/ the SSH key registered there that corresponds w/ my unix account), i type: ssh ubuntu@<ip of node> <TheChistoso> i've also tried it on port 2181... <bigjools> did you add your public ssh key to the sysadmin account in maas *before* starting up the node? <bigjools> fwiw, juju makes this stuff much easier <TheChistoso> of course i can't be sure that's the ip of the machine b/c i can't configure it to use a static ip <bigjools> no, you can't because MAAS manages the DNS <TheChistoso> i'd love to use juju w/ this -- but i can't get nodes working in order to use juju w/ it (c: <bigjools> you're looking at the problem incorrectly <bigjools> you have got nodes working, you just think you don't <bigjools> it's a cloud-like resource, not an installer <bigjools> once nodes are at "ready" you just set the API key in environments.yaml etc and then you can use juju <bigjools> are you using maas to manage DNS? <Marlinc> Should a MAAS node have PXE boot as primary boot location? <TheChistoso> yes maas is managing dns <bigjools> Marlinc: you mean in the BIOS? yes <Marlinc> Okay <TheChistoso> i understand how it's supposed to work <bigjools> TheChistoso: ok so just resolve through there and ignore the IP - the DNS entry will be the same as the node's name in the MAAS UI <bigjools> the zone file will be named after the cluster's name <TheChistoso> well nslookup doesn't find it - rhonda96@example.net shows that you're not setting up the local machine to use itself for DNS name resolution <bigjools> TheChistoso: no, you have to do that <bigjools> maas cannot set up resolv.conf for machines it doesn't boot, it's set via DHCP <TheChistoso> it can set the machine it's running on's network interfaces <TheChistoso> i'm doing these commands from the same machine that maas is running on <bigjools> that machine is not managed by maas <TheChistoso> at any rate -- i added 507-653-2812 to /etc/network/interfaces, restarted networking, and now i can ssh into the machine <TheChistoso> you would think that maas itself would want to use its own DNS services (c: <bigjools> I just explained why it's how it is <TheChistoso> i understand what you said -- but i simply disagree that maas shouldn't use its own DNS service to resolve names locally <bigjools> I think you don't understand <bigjools> but it;s working so goofd <bigjools> good <TheChistoso> well please explain -- if i truly don't understand something, then a little education would be good (c: <bigjools> I explained above <bigjools> "maas cannot set up resolv.conf for machines it doesn't boot, it's set via DHCP" <TheChistoso> yes i understand all of that <mwhudson> i think i would have been pretty upset if maas had started mutilating the networking of the host i installed it on <bigjools> so you understand why you need to set resolv.conf yourself? <TheChistoso> but when you install maas, you could configure the local machine to resolve against itself <mwhudson> (even if i was using maas-dns, which i'm not) <TheChistoso> at the very least, it would make sense to document this <bigjools> maas should not mess with networking on non-managed machines <bigjools> I agree with the doc <bigjools> we should change it <bigjools> since you had trouble working this out <TheChistoso> it's certainly not immediately obvoius that the reason why something can't transition to ready is that nameservers aren't configured properly <TheChistoso> (assuming this is the ultimate culprit) <bigjools> you are using the state names incorrectly <bigjools> ready is when something is available in the pool <bigjools> not something that's deployed <bigjools> and nameservers configuration has nothing to do with state transitions <mwhudson> TheChistoso: there is no state transition in maas when the node is installed and ready for use <TheChistoso> the maas ui shows the node's deployed -- so what's the next state called that i want to get it to? <Marlinc> I think MAAS isn't a toy but really it feels like it. Especially because I've never worked with PXE and never with machines that boot automatically when needed. Really this is just awesome <mwhudson> it's just "allocated" <Marlinc> I just connected another machine tot the network. Set PXE as default boot mode and enabled WOL <Marlinc> And it works <bigjools> it transitions from "ready" to "allocated" when someone requests a node <TheChistoso> mwhudson: the first screen (i have just 1 node) says "1 node deployed" <bigjools> Marlinc: great! <bigjools> TheChistoso: look at the detail node list <TheChistoso> the entire pie chart is blue <TheChistoso> yes -- that says "allocated" <TheChistoso> so you need to get your verbage consistent <bigjools> the pie chart is geared to managing 1000s of nodes at once <mwhudson> eh, maybe the node list and the pie chart could use the same words... <bigjools> yeah, that's inconsistent * bigjools files a bug <Marlinc> Wow WOL is just magic <bigjools> Marlinc: WoL is a hack, you can't power down using it <Marlinc> Well okay thats true <bigjools> it suffices for basic testing <TheChistoso> okay, so what would my next steps be? <TheChistoso> i've defined .juju/environments.yaml and i ran juju bootstrap and it tells me that no matching nodes are available <Marlinc> Mmm the second machine doesn't do much but I'm thinking it is a hardware problem because it is a old computer <bigjools> TheChistoso: I presume you only have one node commissioned and it's already in use? <TheChistoso> by "in use" do you mean allocated to a user? then yes (although i'm fuzzy on how a node is allocated to a user -- it is tied to the user that accepts, commissions, and starts the node?) <bigjools> TheChistoso: no, it's the user who "starts" it <TheChistoso> (*is it) <TheChistoso> okay -- that makes sense <bigjools> only admins can accept for commissioning <TheChistoso> (minor feature request, although perhaps it already exists -- auto-accept, commission, and start, assigning it to a default user) <bigjools> that won't be accepted as a feature <bigjools> it's not designed to work like that <bigjools> you need to think cloud-like <Marlinc> It looks like the DNS is not working the way its supposed to work: 2013-05-02 02:39:13,994 ERROR Invalid host for SSH forwarding: ssh: Could not resolve hostname fqehy.master: No such file or director <Marlinc> It is managed at least it should be <Marlinc> My laptop is using the MAAS server's DNS server <bigjools> where's that error from? <Marlinc> Juju <TheChistoso> since these are meant to serve physical machines, makes sense to me that if you've racked, cabled, and powered it up, then you want to commission it. <Marlinc> When I try to run juju status <bigjools> TheChistoso: the commissioning process is more complicated than that in a DC, often people will want to do burn-ins, update firmware etc. <bigjools> Marlinc: can you manually look up that host? <Marlinc> Nope <Marlinc> Or well <Marlinc> It doesn't work <TheChistoso> well my suggestion is for those people who don't want to babysit the process :D <bigjools> can you check to see if it's present in the zone file <Marlinc> Let me take a look <bigjools> TheChistoso: we'll add a mass-accept at some point <TheChistoso> that would be better :D <Marlinc> Where's it stored? <bigjools> Marlinc: under /var/ib/maas IIRC <TheChistoso> okay, so do you think that i'm at the point where i should be able to use juju? <bigjools> TheChistoso: yep <bigjools> you need to release that node <TheChistoso> how do i do that? <bigjools> sadly there's no way in the UI of doing it (don't ask...), you need to use maas-cli <TheChistoso> ah... <Marlinc> Mm there's nothing related to DNS <bigjools> maas-cli does API calls <TheChistoso> can you point this out in the documentation? b/c i did not see it. gotta have my eyes checked... <bigjools> TheChistoso: I think we have gaps in the docs <TheChistoso> this seems like a major one <bigjools> which bit? <TheChistoso> that to actually use a node w/ juju you have to run a command <bigjools> you don't <TheChistoso> ? <TheChistoso> well i'm confused (c: <bigjools> juju will use available nodes on maas <bigjools> but you don't have any available <bigjools> because you started one manually already <TheChistoso> you'll have to slow down there b/c i'm thoroughly confused <bigjools> basically - just don't use "Start" in the UI <TheChistoso> i was just going to ask that <bigjools> it's completely unnecessary <bigjools> juju manages everything <bigjools> even the SSH key <TheChistoso> is that in the docs as well? <bigjools> it's in juju docs I expect <TheChistoso> it's natural, i think, to hit a "start" button when you see it (c: <bigjools> but there's a section on juju in the maas docs IIRC <TheChistoso> i don't recall reading anything that says "don't press that button!" <bigjools> I think the problem here is that you've not really understood how it works from a higher level. Maybe we need a doc section for that. <TheChistoso> minor feature request: add a label or help text (or something) next to the button in maas when it says 'start" to prevent this from happening <Marlinc> Mmm.. None of the zone files contain any reference to the node bigjools <TheChistoso> i understand how juju is supposed to work (at a high level), but these are small details. c'mon now -- "don't hit start if you want to use it w/ juju"? that seems pretty confusing to anybody starting out. <bigjools> Marlinc: ok that means that something went wrong with parsing the dhcp leases <bigjools> TheChistoso: I think it's only confusing if you don't understand resource allocation concepts in maas <bigjools> Marlinc: please check the celery.log <Marlinc> Sure <bigjools> TheChistoso: which boils down to a higher-level doc that we need to explain these concepts <TheChistoso> given the bad documentation and the fact that i don't work on the product, i don't think it's hard to see why not <bigjools> this support stuff is largely done on an "as best" basis for non-paying customers <TheChistoso> i assumed juju worked w/ already running systems, simply orchestrating services atop already-installed OSes <Marlinc> Okay the're one error in there but it isn't very clear http://pastebin.ubuntu.com/5624546/ <bigjools> I think you need to read the juju docs <TheChistoso> your paying customers don't read docs then, eh? (c: <bigjools> TheChistoso: no, they pay for consultants <TheChistoso> have you read your docs recently? <bigjools> You're welcome for the help <bigjools> Marlinc: aha, that's the problem <TheChistoso> i do thank you for your help -- i'm not trying to be ungrateful :/ <bigjools> Marlinc: well, an indication of the problem <TheChistoso> but it does seem likely that a newbie (such as myself) has read the docs more recently than somebody working on the product <Marlinc> True <bigjools> TheChistoso: that's very likely <bigjools> the product evolves and things get missed, but the vast majority of people don't have problems <TheChistoso> so that experienmced somebody is trying to warn you that it's confusing. <TheChistoso> how do you know that? my previous attempt had me throwing my hands in the air and moving on to other solutions <TheChistoso> iow, you don't know how many people get frustrated w/ the product and leave w/o saying a word <bigjools> I know from speaking to many people they are happy with it. There will always be people with problems though - this is a complicated product solving a complicated problem. <TheChistoso> i know bind, dnsmasq, dhcp, cobbler, pxe, tftp, etc. and i found things confusing <Marlinc> There is a bug report with the same error but it is pretty old and I don't know if its still relevant <bigjools> Marlinc: are there log entries talking about leases? <TheChistoso> perhaps those folks have consultants who know the product and have been using it for a while? (c: <Marlinc> Well in my syslog <bigjools> Marlinc: specifically in the celery.log <Marlinc> [2013-05-02 02:57:33,775: INFO/Beat] Scheduler: Sending due task provisioningserver.tasks.upload_dhcp_leases <bigjools> TheChistoso: like I said, it's best effort, like all OSS projects. <bigjools> Marlinc: ok cool, one sec <Marlinc> [2013-05-02 02:57:33,780: INFO/MainProcess] Got task from broker: provisioningserver.tasks.upload_dhcp_leases[****] <Marlinc> These things? <Marlinc> Okay nice <TheChistoso> is maas not backed by canonical? <bigjools> it is <TheChistoso> so you have real dollars being thrown at it w/ staff dedicating at least some partial amount of their time to it. that's a great thing! now as an open source user of your product, i'm just trying to warn you that your documentation is lacking and you could be turning away users without knowing it. <bigjools> Marlinc: can you see your node's MAC/IP in the leases file? <Marlinc> It is <bigjools> TheChistoso: I'm sorry it wasn't up to scratch for you - people will try to make it better <TheChistoso> okay, so what are my options at this point? can i "un-start" (or "un-use") it so i can use it from juju? <bigjools> Marlinc: can you look at the celery-region log for errors <bigjools> TheChistoso: look at maas-cli <bigjools> it has a release command <Marlinc> There are bigjools http://pastebin.ubuntu.com/5624558/ <bigjools> Marlinc: did you install maas-dns? <bigjools> and maas-dhcp <Marlinc> Yes I did it is running because my laptop is using it right now <Marlinc> And it is using the dhcp too <bigjools> weird <Marlinc> I did need to edit the /etc/network/interfaces file in order to make the DHCP work <bigjools> is maas in /etc/sudoers? <bigjools> is "rndc" available? <Marlinc> rndc is available <Marlinc> But maas is not in the sudoers file <bigjools> mmmm <bigjools> which version of maas are you using? <Marlinc> I've used the Ubuntu 13.04 install <bigjools> ok <bigjools> maas should be in sudoers anyway, which expains the other "omshell" failure <bigjools> not sure why it's not there <Marlinc> Well I don't know either <Marlinc> Just add it or? <bigjools> packaging should have added it <bigjools> one sec <Marlinc> k <TheChistoso> okay awesome -- pie chart is now green showing "1 node queued" and the node list shows its status as "ready" <TheChistoso> so at this point i can "juju bootstrap"? <Marlinc> Yes <TheChistoso> awesome! <Marlinc> environments.yaml is setup? <Marlinc> Or you could get the same problem as me :P the hostname's of nodes are not resolving <TheChistoso> i ran juju bootstrap and the node is now rebooting <bigjools> Marlinc: do you have a /etc/sudoers.d/99-maas-sudoers ? <TheChistoso> Marlinc: can you provide more information wrt your environment? network (including network mask), are these machines on the same L2 network, etc. <bigjools> his dns is not resolving because the node is not in the zone file <Marlinc> Ah that is available <bigjools> there are a few causes for that <Marlinc> I don't think it is my network setup <bigjools> Marlinc: have you got zone files in /etc/bind/maas/ <TheChistoso> well bigjools has been working on the issue longer than i've paid attention so probably best for me to follow his line of thinking <Marlinc> named.conf.maas named.conf.rndc.maas rndc.conf.maas zone.1.168.192.in-addr.arpa zone.2.168.192.in-addr.arpa zone.master <bigjools> is your node in zone.master? <bigjools> there should be a CNAME if so <Marlinc> Nope <TheChistoso> hmmm -- well it prompted for the partitioning scheme again <bigjools> TheChistoso: I'd ask in #ubuntu-server about that, I've never had that <Marlinc> It is not bigjools :p <bigjools> Marlinc: one secx <Marlinc> okay :) <bigjools> Marlinc: http://pastebin.ubuntu.com/5624592/ <bigjools> try that and see what output you get <Marlinc> Okay <Marlinc> The output is good <Marlinc> ip->mac mappings <bigjools> you see your node? <Marlinc> Yep <bigjools> ok so something went wrong where it writes out the zone file <bigjools> let's try: sudo service apache2 restart <Marlinc> Mmm okay <Marlinc> Restarted <bigjools> it will reload the wsgi for maas which causes a job to rewrite the zone file <bigjools> so check the zone file(s) again <bigjools> and check celery-region.log to see if anything went wrong since you restarted <Marlinc> No new output in that log <Marlinc> And it is still in none of the bind maas files <bigjools> hmmm <bigjools> can you try renaming the node <bigjools> ands watch the log <Marlinc> Sure <Marlinc> It did a 'upload_dhcp_release' <Marlinc> It did a 'upload_dhcp_leases' <Marlinc> * <bigjools> yeah that runs periodically <Marlinc> Other then that nothing <bigjools> arey you looking in celery-region ? <Marlinc> Well tailed both of them <Marlinc> [2013-05-02 02:05:34,807: INFO/MainProcess] Task provisioningserver.tasks.rndc_command[6499426a-eb2c-4b30-bfbc-0314632cef2f] retry: None <Marlinc> None <Marlinc> Is the last <bigjools> I'm not sure what's going on, there are no errors but the file is not getting written <Marlinc> I'll reboot the thing <Marlinc> Maybe something is not working <Marlinc> Now <Marlinc> [2013-05-02 03:40:27,551: INFO/MainProcess] Task provisioningserver.tasks.rndc_command[6499426a-eb2c-4b30-bfbc-0314632cef2f] retry: None <Marlinc> None <Marlinc> Runs without issues <Marlinc> But there's still no entry <Marlinc> bigjools, ^ <bigjools> yeah# <bigjools> trying to think <bigjools> someone else will be around soon that may have more ideas <Marlinc> Okay <Marlinc> What is soon? Because it is well time to sleep <Marlinc> For some hours already though <Marlinc> Usage: time [-apvV] [-f format] [-o file] [--append] [--verbose] <Marlinc> [--portability] [--format=format] [--output=file] [--version] <Marlinc> [--quiet] [--help] command [arg...] <Marlinc> do mei 2 03:50:07 CEST 2013 <bigjools> ! <bigjools> there's a guy near your TZ and one about to start, so just ping later <Marlinc> Okay <Marlinc> I'll see
{'channel': 'maas', 'month': 5}