#ubuntu-cloud 2011-01-03 <kim0> raymdt: Morning <superxgl> hi, is there any one who have successfully configured hadoop on Eucalyptus cloud? <superxgl> Does any one know how to do ? <kim0> superxgl: hey, I once configured a hadoop cluster. I don't really think they're related <kim0> i.e. after you get your cloud instances running, you can run hadoop on top like any other application right <superxgl> hmm,but.. <superxgl> the problem is <superxgl> how to make hadoop interact with walrus, instead of copying data from local file system to hdfs? <superxgl> and how to start several instances such that each instances runs like a slave node of hadoop? <superxgl> how to ensure that hadoop access my private cloud s3(walrus in eucalyptus) instead of the amazon's s3? <superxgl> i want to setup a hadoop cluster using the vm instances in the cloud <zoopster> superxgl: forgive my ignorance as I've never set it up, but as I understand it you would use it the same as you would in EC2 <zoopster> superxgl: the same syntax/usage model applies to hadoop in UEC/eucalyptus as it would in EC2 <superxgl> zoopster: thank you. but how cloud i make hadoop interact with walrus? <zoopster> superxgl: do you mean how do you change the configuration to point to walrus vs hdfs? <zoopster> superxgl: it's the same as changing it to s3, no? <superxgl> zoopster: hmm..let me think <zoopster> superxgl: so for a basic hadoop implementation you change the details in the hadoop-ec2-env.sh not sure about cloudera's or yahoo's implementation <zoopster> superxgl: if you are replacing hdfs with s3 then you need to update the hadoop-site.xml file with the appropriate changes for s3? <zoopster> superxgl: looks like whirr is a better option these days...same concepts apply <superxgl> zoopster: thank u very much.i will have a try. <superxgl> hmm..what is whirr? sorry, i dont know it. i will google <zoopster> superxgl: have fun... http://incubator.apache.org/whirr/ <zoopster> superxgl: keep in mind that most syntax will be the same in ec2 as in eucalyptus - the naming and paths may be different, but the concepts are similar if not identical <superxgl> zoopster: thanks:) u help me a lot today. <zoopster> superxgl: glad to help...let us know how it works out! <kim0> whirr looks cool thanks @ zoopster <superxgl> zoopster: thanks again. i will have a try <kim0> superxgl: Once you get it running like a champ, and if you're interested about sharing your setup with the community, please let me know :) <superxgl> kim0: okay. i will:) <kim0> superxgl: rock on :) <superxgl> thanks :) <CharlieSu> Anyone setup OVPN on Ec2 before? I'm trying on a Ubuntu instance and anytime I add a bridge interface and bring it up my instance goes dark. No more network connectivity. Any ideas? This is the config I'm adding. https://gist.github.com/e9ed7188be1d1ee04f16 <erichammond> CharlieSu: I wrote this a long time ago: http://alestic.com/2009/05/openvpn-ec2 #ubuntu-cloud 2011-01-04 <Guest17443> hi everyone <Guest17443> ubuntu 11.04 amazon ami are not responding at all - i have tried both 32 and 64bit <Guest17443> any idea how can i make them working <erichammond> Guest17443: What AMI id and region? <erichammond> Guest17443: What do you mean by "not responding" (specific command you are trying and specific results) <Guest17443> zone: us-east-1c - ami id: ami-78f10711 <Guest17443> unable to ping or ssh to it <erichammond> Guest17443: Are you able to ping or ssh to instances of other AMIs using the same account? <erichammond> Guest17443: That looks like an old AMI for Natty. Have you tried one of the more recent daily AMIs? <Guest17443> should i try ami-6ef00607 or ami-cef107a7 <erichammond> Guest17443: I've not used Natty. Just tossing out ideas. <erichammond> Guest17443: Ok, just ran an instance of ami-78f10711 and am able to ssh in just fine. <erichammond> Looks like it's not a problem with the AMI. <Guest17443> hmm <erichammond> Guest17443: If this is a new AWS account, you'll need to configure the security group. <Guest17443> i double checked it before :) <saju_m> In EC2, How get image category objects (Amazon images, Public images, etc--) and instance type objects (m1.small', 'm1.large', 'm1.xlarge', etc--)? Have any methods for that? <Guest17443> please have a look on console out for ami-6ef00607 - http://pastebin.com/6MvaVrB9 <CharlieSu> erichammond: thanks for the OVPN link.. worked like a charm <smoser> my guess is that Guest17443 was running on a t1.micro. <smoser> (bug 686692) <uvirtbot> Launchpad bug 686692 in linux "natty kernel does not boot on ec2 t1.micro" [High,Confirmed] https://launchpad.net/bugs/686692 #ubuntu-cloud 2011-01-05 <theholder> HI <theholder> with Ubuuntu cloud do all machines need to be identical in hardware specifications <flaccid> no <theholder> just wondered <theholder> is the cloud computer seen as one big cpu or multipule> <theholder> ? <flaccid> by what? <theholder> the system <theholder> ie htop <theholder> or other cpu information gathering software <flaccid> what system? <theholder> the cloud server <theholder> running on the cloud <flaccid> do you mean an instance? <theholder> yes <flaccid> its a virtual machine. it sees the virtual machine it is allocated, not the cloud controller <theholder> hmmm <theholder> so its seen as a single machine with its own ram cpu etc <flaccid> its a virtual machine <flaccid> yes, each instance gets its own 'slice' <theholder> am i getting it confused with paralle processing <theholder> then <theholder> =/ <flaccid> i guess so <theholder> lol my bad <walt_> I am trying to get a clear answer on what it the preferred way to build 'gold copies' most of that would be adding packages, which can be done via cloud-init but I have some manual s/w to install and config also <walt_> my prior question is specific to 10.10 /UEC2 * kim0 waves hi to everyone <kim0> Any new faces around .. anyone wanting to chat, shoot questions and the like .. Hit me <daker> kim0, not sure it's the good time <kim0> daker: what do you mean <daker> 18h GTM is not the good time <daker> GMT <kim0> well it should be .. <kim0> it's 10am-1pm in US timezone <kim0> and 6pm-9pm Eu <daker> just a thought ã <kim0> Yeah, it's not it .. it's week on week off <zoopster> kim0: enjoyed your blog post on cloud-init...would really like to see other use cases/examples of usage! <kim0> zoopster: great! any suggestions ? <kim0> I'd love examples of more things to demo/talk-about :) <kim0> zoopster: btw, I just pushed a screencast building a LAMP stack in 60 seconds http://www.facebook.com/l.php?u=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHinDARd72nQ&h=ab7cf <kim0> it's basically cloud-init as well <zoopster> kim0: how about using it to setup a basic dev/test env...not sure what to start with, but take a base ubuntu image and putting all the parts to pull a newly built app? Dunno...anyone with ideas? <kim0> zoopster: actually, my next screencast is gonna be multi-tier LAMP setup (web, app, DB) <kim0> using cloud-init inlcudes and all <zoopster> perfect <elasticdog> kim0: nice posts on cloud-init...seems like it would be ideal to bootstrap a configuration management system (chef/puppet/etc) from a pristine machine image <kim0> elasticdog: Indeed! <kim0> elasticdog: that would be a good idea <kim0> I know mathiaz had created some great puppet bootstrapping guides .. however I'll try to improve any missing bits around that <kim0> elasticdog: nice site btw <elasticdog> thanks :-) <kim0> hehe <elasticdog> I need to actually write stuff on a regular basis <kim0> Yeah <kim0> elasticdog: why did you choose to teach yourself factor <elasticdog> a couple years ago I kept hearing about it from various places, and like learning new things <elasticdog> I had never really messed with stack-based languages before, so it was somewhat mind-expanding <kim0> I see <kim0> elasticdog: do you use ubuntu server or cloud somehow .. if you can tell, I'm interested to hear <elasticdog> kim0: yep, I've been working on setting up a test UEC environment on a few machines at work <kim0> elasticdog: how is that going <elasticdog> it's going well so far, but I'm admittedly pretty new to most things cloud <kim0> elasticdog: If you're looking for things to write about, contributing your UEC setup experiences would be awesome :D <kim0> elasticdog: give it a tought and let me know if you're interested. I'd love to have contributors such as yourself start sharing their info/findings <elasticdog> kim0: that's a good idea...my issues usually aren't that I don't have things to write about, but that I'm a slow/perfectionist writer, but I'll let you know if I post anything <kim0> cool hehe :) #ubuntu-cloud 2011-01-06 <hubidubi> hi <hubidubi> I'm new to cloud computing and ac2 and I need to plan an ubuntu server running in ac2 <hubidubi> is there anybody who would answer some general questions? <hubidubi> I read lots of docs although I still have lots of dark spots * orangeac1e recommends learning by doing <hubidubi> orangeacme: you're right, I'm planning to do it, but still have some basic question <hubidubi> this time of ubuntu server is similar to a simple vmware virtual server? <hubidubi> I've read somewhere that it can't persist data so I have to save data to s3 <hubidubi> is it true? <orangeacme> the local instance storage disappears when you shut the machine down <orangeacme> you can use the elastic block store which is like a persistent disk <kim0> hubidubi: welcome <orangeacme> but ultimately you probably want to keep backups in s3 <hubidubi> ok, but I have access to the file system of my virtual machine, isn't it? <hubidubi> so I can write data to my file system <orangeacme> local instance storage and ebs look just like a local device <hubidubi> ok, but if I use my virtual machine should I know anything about amazon infrastructure? <hubidubi> I mean, I won't see outside from my virtual box <hubidubi> let me more straight <hubidubi> We have a java based server application with sql database backend that we would like to migrate to an ec2 based ubuntu virtual machine <hubidubi> do we have to make any changes on infrastructure/architecture comparing to a standalone server? <hubidubi> can I use any SQL database on EC2 host as I would use it on a standalone server? <orangeacme> i think your questions can generally be answered by "yes, it looks just like another server" <orangeacme> but you might want to consider backups and bringing up new servers if that one goes down <hubidubi> an ec2 server image is as persistent as a vmware or any other virtual server image? <hubidubi> I mean it won't lose data after restart eg. <hubidubi> yes, normal backups are the same as on normal servers, absolutely necessary <orangeacme> http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/ # says on instance storage: "If an instance reboots (intentionally or unintentionally), the data on the instance store will survive. However, if an underlying drive fails, or if you stop or terminate the instance, the data in the instance store will be lost." <orangeacme> but see also amis backed by amazon ebs <hubidubi> thanks for your help <hubidubi> I go and read all of these docs <kim0> orangeacme: Thanks for all the answers :) #ubuntu-cloud 2011-01-07 <ttx> smoser, kim0: nice --> http://www.talkincloud.com/ubuntus-cloud-init-promotes-customization-in-the-cloud/ <smoser> dilbert: http://dilbert.com/strips/comic/2011-01-07/ <smoser> cloud <eric_hill> Since an Amazon AWS secret key is required to sign requests to S3, does that mean I *have* to put it on my server in EC2? <eric_hill> i.e. Is there another way to have a web server act on behalf of my AWS key without actually putting it on the server? #ubuntu-cloud 2011-01-08 <pmatulis> is choosing UEC tasksel option at the end of the server install the same as choosing the option at the beginning (inserting cd)? <waltc> test message <waltc> maverick (and natty) images from uec-... won't enter running state under UEC 2.x 10.10 <waltc> state is Extant <waltc> There was one forum hit that said the solution, found from frustration, was reformat, reinstall CC server and add new user <waltc> There is a better solution...right???????? <waltc> the real question is would somebody please explain the 'better' solution? <khetzal> hello, i've a problem with ubuntu cloud trial on amazon ec2 (buti d'ont know if it's really a problem) <khetzal> the message say "you could ssh ... using ssh ubuntu@" without any hostname :/ <waltc> khetzal, I suspect that is, worst case, wrong, best case it meant @ the returned ip address of the running instance <waltc> oh...were you referring to my forum response? @ip address...it was shorthand <khetzal> hum, timer start at 110min and is now 82min, is there any "initilisation time" before knowing the ip ? <khetzal> waltc, what post ? <waltc> someone had asked that question on the Eucalyptus site and I responded back with try ubuntu@ <waltc> meaning ssh ubuntu@<ip addr> <waltc> so you are running a Ubuntu ec2 image on Amazon? <waltc> I know when running ubuntu images on UEC you sign in as ubuntu, not root <waltc> I've run ubuntu @ amazon but it was maybe a year ago <khetzal> but i've no ip adress <waltc> does it show state as running? <khetzal> i see the state only on https://10.cloud.ubuntu.com/maverick/instance_info/ <khetzal> it look like to be running but no real information about that <waltc> this is running at amazon ec2? <waltc> wait <waltc> where is that image running? <khetzal> "Your instance is now running. Your remaining time is approximately:" i assume it is running ;) <waltc> I would guess the same <khetzal> "You now have an Ubuntu Server running in Amazon EC2!" i've no more information than that :/ <khetzal> it's the ubuntu trial, i'm only logged into the ubuntu website, not directly on amazon <waltc> ah...ok...are you on the amazon ec2 webpage? it should give you an ip or host name that looks like an ip with some characters prefixing it <khetzal> it's not a big issue, is just that i can't use the trial from ubuntu <waltc> something like ec2-amazon-(864)930-0890 or ec2amazon or amazon-ec2- <khetzal> i've only the ssh key fingerprint (is it possible that i-013dc56d at the end of the fingerprint is the hostname ?) <waltc> I am not familiar with the trial and without the address to shell into, it doesn't matter whether it is root@ or ubuntu@...unless that page you are looking at gives you a level of indirection to connect as in a button that says login here <khetzal> i don't see any indication, and that is the problem <khetzal> the place where the ip should be is totaly blank <waltc> sorry, I don't think I can help...if its worth anything ubuntu runs fine under amazon OR UEC or Eucalyptus <waltc> maybe whoever put up the trial screwed up the page <waltc> can you try a Lucid image? <khetzal> i'm looking everywhere, i can't "restart" the trial or stop it <khetzal> and i can't change anything <khetzal> it's certainly a bug <khetzal> i've leave my feedback about "what i did" during this trial ;) if they can correct that for other users, it will be a good point ! <khetzal> thanks for your help ! #ubuntu-cloud 2011-01-11 <maruq> hi guys <maruq> how can I use something like cloud-init to set hostname at node launch (EC2)? <anacrolix> hiya, i'm having some trouble connecting to a cloud trial <anacrolix> Permission denied (publickey). <anacrolix> i've determined that i need to upload an ssh key to my launchpad.net account, there is one there but i don't think that i have the corresponding private key anymore <TeTeT> anacrolix: what about generating a new key where you still have the private key and u/l the public key to LP? <anacrolix> i've done this, now there are 2 public keys on launchpad <anacrolix> i still get the same error <anacrolix> perhaps keys are only copied over when the instance is created, which was some 20-30 mins ago? <anacrolix> hm it was 40 mins ago, and i uploaded the new key about 15 mins ago <flaccid> launchpad has a public cloud? <TeTeT> anacrolix: yes, keys are copied when the instance is started from what I know. <anacrolix> okay, i've put all the information i just gave in the feedback, what's the chance i'll get another trial oppurtunity now i have this worked out? <mhall119> does UEC support the S3 API? <smoser> mhall119, yes <mhall119> cool <mhall119> I just found a Django storage backend for S3 <mhall119> now I just need an excuse to use it <flaccid> django rules #ubuntu-cloud 2011-01-12 <daxt> Hello <daxt> i get this error "Looking up studiopaceatl.com <daxt> Cannot look up hostname studiopaceatl.com: Name or service not known" in ubuntu EC2 instance <daxt> but ftp server works and i am connected to using gftp <daxt> can somebody direct me to the right path ? <daxt> anyone home ? <daxt_> HHHHHHHHHHEEEEEEEEEEEEEEEEEEEELLLLLLLLLLLLLLLLLLLLLOOOOOOOOOOOOOOOOOOOOOOOOOWWWWWWWWWWWWWWWWWWWWWWWWWWW <erichammond> daxt_: From what you've described so far, this does not sound like an issue that would be specific to EC2. <daxt_> hi erichammond , then what it can be ? <erichammond> Note that it sometimes takes hours for EC2 and UEC questions to be answered on this channel. Non-cloud questions will probably get better help elsewhere. <daxt_> so mine is non-cloud ? :D <daxt_> can u point me to a proper place which might help me to resolve this <erichammond> daxt_: It sounds like a DNS lookup issue for a general host name. DNS for this on EC2 works pretty much the same as anywhere else. <daxt_> actually my ftp gui client connects and works properly <daxt_> only the ftp cli at ec2 instance does not manage to lookup the host , is the problem <daxt_> shall i modify the resolv.conf and add externam dns than what amazon gives ? will it help me ? <erichammond> daxt_: Sounds like you'll need to investigate how the name is being resolved for your local client and how your local environment differs from the environment on your server. I don't have enough information about any of that to be able to help you sufficiently <daxt_> erichammond , when i connect to the ftp hoststudiopaceatl.com from my PC , it connects , but when i try to connect from the EC2 instance using ssh , it does not <daxt_> so it has to be a dns issue on EC2 according to my feeling <erichammond> daxt_: I'm not telling you to go away, just pointing out that it probably isn't directly related to the topic of the channel, and given the nature of this channel, you might find better help over on #ubuntu or other places. The fact that it is not working on an EC2 instance is unlikely to be related to EC2 as a service. <daxt_> erichammond , let me show u a screenshot , so you can get a clear understanding :) <daxt_> http://img820.imageshack.us/img820/8213/screenshot18f.png <daxt_> this issue is EC2 related <daxt_> EricHammond , could this be an issue with security groups ? <daxt_> aka amazon firewall ? <waltc> I realize this is semi vague here (detail in forum) but is it reasonable to get warnings and or errors during a startup of UEC 2 (10.10)? <waltc> the first one is <waltc> The file cloud-error.log contains the following: <waltc> 10:54:16 ERROR [SystemUtil:Thread-23] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop0 error: loop: can't get info on device /dev/loop0: No such device or address <waltc> the second one is: 10:54:17 ERROR [ISCSIManager:Thread-23] com.eucalyptus.util.EucalyptusCloudException: tgtadm: invalid request <waltc> Since I don't have iscsi implemented that may explain this but why there error? <waltc> the third one is:10:54:20 ERROR [NioServerHandler:New I/O server worker #1-1] Internal Error. <waltc> com.eucalyptus.ws.ServiceNotReadyException: System has not yet completed booting. <waltc> at com.eucalyptus.ws.server.NioServerHandler.messageR eceived(NioServerHandler.java:107) <waltc> at org.jboss.netty.handler.stream.ChunkedWriteHandler .handleUpstream(ChunkedWriteHandler.java:114) <waltc> at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:385) <waltc> at org.jboss.netty.handler.codec.replay.ReplayingDeco der.unfoldAndfireMessageReceived(ReplayingDecoder. java:459) <waltc> at org.jboss.netty.handler.codec.replay.ReplayingDeco der.callDecode(ReplayingDecoder.java:443) <waltc> at org.jboss.netty.handler.codec.replay.ReplayingDeco der.messageReceived(ReplayingDecoder.java:381) <waltc> at com.eucalyptus.ws.handlers.http.NioSslHandler.hand leUpstream(NioSslHandler.java:31) <waltc> at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:342) <waltc> at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:329) <waltc> at org.jboss.netty.channel.socket.nio.NioWorker.read( NioWorker.java:330) <waltc> at org.jboss.netty.channel.socket.nio.NioWorker.proce ssSelectedKeys(NioWorker.java:282) <waltc> at org.jboss.netty.channel.socket.nio.NioWorker.run(N ioWorker.java:203) <waltc> at org.jboss.netty.util.internal.IoWorkerRunnable.run (IoWorkerRunnable.java:53) <waltc> at org.jboss.netty.handler.execution.MemoryAwareThrea dPoolExecutor$MemoryAwareRunnable.run(MemoryAwareT hreadPoolExecutor.java:502) <waltc> at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1110) <waltc> at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:603) <waltc> at java.lang.Thread.run(Thread.java:636) <kim0> waltc: the losetup and tgtadm would worry me <kim0> the rest not really .. Java apps are chatty :) <kim0> Are you actually able to start isntances and connect to ebs volumes <waltc> as I had said earlier...I blew away the logs (they do get massive) and did a fresh restart <waltc> no....its a function of when the author choses to issue the log.info(...). I actually don't mind info (I use them all the time) but they can be overused thus lowering the 'signal to noise' ratio <waltc> what I am trying to ascertain is for the real issues I have, are their roots during startup <waltc> you know...separate cause from effect <kim0> yeah .. <waltc> I don't have iscsi so the admtgt I don't think should even be an issue. Well, the other thing too, are these benign, like the iscsi one <waltc> the loop device....that worries me too <waltc> at first, I tried to do things outside the 'script' like not have the 'everything but node controller' on my desktop because it saved buying a dedicated machine and it seemed like it 'should' work <waltc> largely it did except for disappearing public ipaddrs <waltc> with 10.10 I now have 3 machines, a dedicated Ubuntu server for 'everything but' and a dedicated ubuntu server for the nc <waltc> I did a free install...out of the box, manually config'd eth1 <waltc> works GREAT for store images of ubuntu. does not work for anything else <waltc> frankly, what I am trying to do kim0 is get buy-in at work to put up UEC there for my dev team, but I don't want egg on my face as soon as someone wants to start a centos image and can't <kim0> waltc: have you tried publishing a centos image <kim0> that should work AFAIK <waltc> I really want to get this to be predictable and reliable...and, as I said to you earlier, become really well versed in this...local 'expert' and all <waltc> I did try an image published by Eucalyptus <waltc> one commonality is the lack of a ramdisk image...but that should not necessarily cause a failure <waltc> the image starts, just never transitions to running <waltc> since it is not running I can't get to a log <waltc> with maverick and naddy images do I need to pull them apart and publish the individual parts separately? They run great under kvm natively <kim0> waltc: can you try with this image http://eucalyptussoftware.com/downloads/eucalyptus-images/euca-centos-5.3-x8 <kim0> 6_64.tar.gz <kim0> broken link :) <waltc> just publish tarball? <kim0> yeah mostly <kim0> waltc: Also check out this message http://craighoward@example.net/msg00367.html <kim0> where smoser mentions an updated patch to the uec-publish-tarball utility <kim0> which may help <waltc> I will check it out...What I did for centos is pull it apart, remove the xen subdir moved the kvm subdir up a level and re-tared it <waltc> didn't go to run state <waltc> in other words, in images that work, there is no kvm subdir, the kernel manifest and ramdisk image are all in the single subdir <waltc> while centos is on my list...I don't want to shotgun the issues...so I am content, for now, resolving why maverick and naddy uec images from uec-ubuntu don't 'just work' <waltc> there was that one forum problem someone entered where he said he got frustrated and wiped the 'everything but nc' server and started from scratch to reinstall (plus added a new user) and now his works. <waltc> I don't find 'reformat and reinstall' a satisfying solution <waltc> there is code in walrus land that differentiates administrator from not administrator but that is just for permissions <waltc> is the focus on having these images work on ec2 and 'the hope is' they also will work on uec? <kim0> waltc: hey, are you "wcorey" from forums <waltc> kim0, do you have a maverick based uec cloud where all images store or otherwise just work?....yes I am <kim0> waltc: nice meeting you man :) <waltc> you too! <kim0> I don't currently have UEC running with me <waltc> currently is this second or post 10.10? <kim0> but yeah image store images, should definitely just work <kim0> this second <waltc> my question is not 'should' they work it is 'do' they work <kim0> when I played with 10.10, they did boot <waltc> the 'store' images do run perfectly...the maverick and naddy I get off the ec2/uec site start just fine but never go to running <kim0> hmm <kim0> well, with no particular error message <kim0> it's gonna be a bit hard <kim0> did you try nc logs ? <waltc> did you ever look at forum, t=1588826? <waltc> yes I did <waltc> to re-state what I believe you said is you have (somewhere) a UEC cloud, based on 10.10 and you downloaded the image from the ec2/uec site of maverick and naddy and published them uec-pub-tarball and they run just fine. Is that accurate? <kim0> yeah, just not natty, it wasn't there yet <waltc> now I feel ill...that means the problem is here <kim0> waltc: it might have needed some manual step though, can't quite remember <waltc> oh...do you run them as you or admin? <kim0> waltc: man, I understand you want to help UEC users "debug" such problems <kim0> waltc: as opposed to the current format/reinstall appraoch <kim0> is that correct <waltc> btw, I enjoy your 'blogs' like publishing what scott did with the running the cloud-init under kvm directly <kim0> waltc: What about we (you and I) start like some project to freshly install UEC, documenting the exact setup <kim0> waltc: with every command ever needed ...etc <kim0> if we stumble across an issue, we can hopefully get some help from the devs over here <kim0> the advantage being .. <kim0> instead of being an individual support case <kim0> it would become .. like an open UEC manual for the whole world <waltc> in that case it wasn't so much trying to help him, he already got a solution. What I was trying to do is either from him or someone like yourself, explain which part of what he did actually fixed the problem. I logged in as admin, and just downloaded the credentials there so maybe I should not use admin just add myself as joe user and run images that way <kim0> Everything you wanted to know about UEC but were afraid to ask .. comes to mind :) <kim0> waltc: the thing is .. he doesn't know, nor do I .. and if you ask the devs .. so many things could go wrong <kim0> there is no simple answer <kim0> which is why I am suggesting we start documenting a "controlled" envrionment <kim0> I would personally love to work on this with you <waltc> kim0....yes, yes, and yes <kim0> just let me know if you can commit to doing it <waltc> I've been in software, as a developer for decades <waltc> but part of that is support <waltc> what, as a person paid to do this stuff, really annoys me is people who open a problem, find some solution and just simply announce, "OK, thanks, it's fixed now" without ever saying what they did <kim0> hehe yeah .. some tend to do that :) <waltc> the next worst thing is people that say, I did this, this, this, this, and this....and now it works <waltc> that, as he described it, is the "hail mary" solution... <waltc> ALOT tend to do that <waltc> I like crisp and clean answers..THIS (and only this) is the cause and the solution is do this (and only this) <waltc> link cause and effect and remedy together <waltc> Oh...I have become an expert at re-installing, from scratch, the ubuntu server (as cluster) and the ubuntu server as node <kim0> waltc: I am in total agreement <kim0> about the "this and only this" statement <kim0> thus I am suggesting we start working on a controlled env <kim0> documenting everythgin aloong the way * kim0 is on crappy Internet for now .. slow answer <waltc> the other thing I hate is people who open an issue and it gets kidnapped by others who say I have the same problem except... After a couple of those you don't even know what the problem environment even is any more <waltc> I realize to most, if not all, people who work on the forums and maybe even launchpad, theirs is a 'labor of love' in other words they are there because they are interested in being there, not they have to be. So when a problem comes in and it is, or becomes by others additions, too confusing they move on as it is now too confiusing <waltc> let's start here...something I wanted to verify with you....10.4 is LTS, 10.4 UEC is 1.6, 10.4 won't run Eucalptus 2 or UEC 2.0 as it needs the newer kernel. What you get with UEC 2 is the cloud init stuff which means you don't have to specialty create new images for everything but do stem cell image that gets specialized at boot time, not create time...correct? <waltc> or, asked differently, why would one want to use UEC 2.x? <kim0> waltc: 10.04 is LTS and runs 1.6 yes .. can it run 2.0, maybe <kim0> cloud-init is not directly related to EUC version AFAIK .. it's an image thing <waltc> I seem to recall a forum question and answer saying you could not install UEC 2 as Eucalyptus 2.x required 2.6.35 <TeTeT> waltc: there's linux-image-generic-pae-lts-backport-maverick for lucid in the repo <TeTeT> I guess an backport of euca 2 for lts would be an interesting thing to do <waltc> but cloud init IS a Ubuntu thing...until very recently you couldn't use it on non ubuntu images. Amazon recently released a centos image that does support cloud init...right? <kim0> TeTeT: any idea if euca2 does need a newer kernel <TeTeT> kim0: actually not <kim0> waltc: ^^ thought so <kim0> waltc: yeah amazon released their own "Linux distro" basically <waltc> weird huh? <kim0> waltc: and they did choose to include cloud-init <waltc> that pretty much locks Amazon into being a distro as unless RH includes cloud-init it won't be part of stock centos <waltc> correct? <kim0> Amazon's Linux, IS a separate distro <kim0> and yes .. centos will only get it when RH acccepts it (if ever) <kim0> unless someone packages it for centos directly <waltc> you mean all Amazon created AMI's or just that centos with cloud-init? <\sh> moins <waltc> why wouldn't they (RH)...just like why wouldn't Ubuntu/Canonical include Spice? <\sh> hmmm...when I want to upgrade an ec2 lucid instance to maverick, and I did a dist-upgrade from lucid to maverick...what aki id do I need to set to boot the correct kernel (which should be linux-image-2.6.35-24-virtual) <kim0> waltc: It's not like Ubuntu is opposed to spice .. It's just barely mature right now .. I'm sure Ubuntu will get it (personal opinion) <kim0> waltc: it's a matter of opinion .. if RH wants it, it can get it .. it's all open source <kim0> \sh: did you try "ec2-describe-images" <kim0> \sh: it should tell you the associated aki/ari <\sh> kim0: yes, I followed smosers howto http://ubuntu-smoser.blogspot.com/2010/04/upgrading-ebs-instance.html <\sh> but I don't find the kernel craighoward@example.net... <waltc> We recently went through an exercise at work to put much of our infrastructure in EC2. building specialized images are a pain in the butt, slow, fragile, etc..I MUCH prefer the stem-cell provision on boot approach <\sh> during dist-upgrade it installed the 2.6.35-24-virtual flavor <kim0> waltc: Yep! :) <waltc> between delta-cloud and RHEV RH has staked it's 'claim' on the cloud and virtualization space....it stands to reason they (and SUSE) would grab cloud-init <waltc> I would think the only incentive not to is to 'one-off' someone else's distro <kim0> \sh: smoser 's blog post seems to nail it <\sh> kim0: he was upgrading to lucid...and I'm doing dist-upgrade from lucid to maverick, so I adjusted his shell/ec2-api-tools magic to the kernel version I got during dist-upgrade <\sh> kim0: but this version I can't find the version in the ec2-describe-images --all list <\sh> kim0: the only kernel which matches on the list is IMAGE aki-2316fd4a 099720109477/ububtu-test-kernel-2.6.35-12-virtual-x86_64-20100730-test-10 <kim0> \sh: The 64 bit aki seems to be aki-427d952b, 32bit aki-407d9529 <smoser> shang, what kernel do you have installed <smoser> and what release (maverick?) are you running ? <kim0> smoser: he's upgrading lucid to maverick <\sh> smoser: I installed the t1.micro ebs image of lucid...and dist-upgraded to maverick now <\sh> smoser: the running instance kernel is linux-image-2.6.32-311-ec2 and the dist-upgraded kernel now is linux-image-2.6.35-24-virtual <smoser> ah. then, yeah, you want the pv-grub kernel. make sure that you install grub-legacy-ec2 <\sh> (hence no linux-image...-ec2 package for maverick) <\sh> the grub-legacy-ec2 is already installed <\sh> smoser: and what is the pv-grub kernel? <smoser> yeah, then you have a /boot//grub/menu.lst that lists <smoser> linux-image-2.6.35-24-virtual <smoser> kim0 listed the pv-grub kernel above <\sh> smoser: ah ok... <kim0> \sh: this actually chainloads the kernel inside the image .. which is cool <kim0> smoser: do you actually memorize the akis ;) <\sh> so I just update via modify-instance the kernel id to the aki mentioned. <smoser> no, i look them up. but i do have cached describe-images of all regions <smoser> yes, \sh <kim0> smoser: um, once on pv-grub .. upgrading maverick->natty .. should just involve dist-upgrade, like a bare metal box right ?! <kirkland> kim0: when does that video go up? <smoser> thats the goal, kim0 yes <smoser> kirkland is looking for his 15 minutes of youtube fame <kim0> kirkland: :) Jono is holding it hostage .. should definitely go up today ... * kim0 will bug Jono as soon as I see him <\sh> hmmm <\sh> I think I destroyed the instance, grmpf...so now I directly start a maverick instance <kim0> :) <kim0> \sh: I hope you didn't loose any data on this :) <smoser> \sh unless you terminatede the instance, you didn't loose its root volume <smoser> (as it is ebs) <\sh> kim0: no...it's just this t1.micro free amazon offer <kim0> \sh: stopping is not terminating <smoser> kim0, you or i could walk the lucid -> maverick upgrade path in a blog post <smoser> that might actually be useful <\sh> kim0: terminated :) <kim0> smoser: Sounds good :) * kim0 likes <kim0> I just wish people would stop using ebs images .. it's a cloud c'mon :) * kim0 notes that blog entry <kim0> smoser: I suppose Amazon could update their PVgrub kernel .. how can we always find out the latet ID ? <smoser> kim0, no way. * kim0 #&$^%$^$ @amazon <smoser> other than look at the kernel associated with our latest image of that release/arch <kim0> Yeah ok <crazed> how do you enable root logins in an ec2 ami? <crazed> i changed /etc/cloud/cloud.cfg disable_root: false <crazed> but still seems to put in the "please login as ubuntu" <waltc> kim0: I actually have fully documented the installation process and post installation I did this last time around. It appears managed-novlan requires a second eth on the CC (in a default UEC install) yet it doesn't configure it. In fact, if it doesn't exist, it doesn't care. <waltc> that's what I was missing in the 9.04->10.04 time frame. BTW, my old issue of disappearing public ip addresses went away perhaps because of 10.10 or perhaps because of eth1 <kim0> kirkland: Your vid is out and loud woohoo http://www.youtube.com/ubuntudevelopers#p/a/u/0/CWNjzwci5e8 #ubuntu-cloud 2011-01-13 <kirkland> kim0: thanks, i reposted it at http://blog.dustinkirkland.com/2011/01/working-toward-ubuntu-1104-server.html <kim0> kirkland: awesome, I posted as well .. you can never have too many blogs ;) #ubuntu-cloud 2011-01-14 <kjdash> I noticed some chatter on the mailing list about problems rebundling the new lucid (10.04) images <kjdash> can anyone clarify if this only affects images using the pv grub kernels? <kim0> smoser: would know <smoser> kjdash, i'm sending a mail right now with info <kjdash> great <kjdash> i'm in the process of rebuilding my images, and wanted to take the path of least resistance <smoser> kjdash, are you using 10.04 or 10.10 ? <smoser> mail sent to ec2ubuntu and ubuntu-cloud (https://groups.google.com/group/ec2ubuntu/browse_frm/thread/5d392b2325269484) <smoser> kjdash, if you're using lucid, i suggest moving to using the pv-grub kernels. <kjdash> 10.04 <kjdash> that was my intent <kjdash> so your fix <kjdash> ERROR: undefined method `startswith' for "ext3":String <kjdash> seems like there is some supporting lib missing in there <kjdash> i fixed that line, only to find more errors <smoser> oh crap. shoot. i do have that fixed in natty. but let me cheeck. <smoser> thank you for tresting <kjdash> i'm guessing you just backported your fixes for natty? <smoser> shoot. <kjdash> i actually tried that myself yesterday <smoser> i screwed it up.. so many versions of stuff. <kjdash> grabbed the .deb and installed it manually <kim0> smoser: when you get a chance to check out that BP let me kow ... thanks man <smoser> kjdash, i'm working on the ec2-ami-bundle :-( <smoser> kjdash, are you there ? <smoser> if so, could you try putting http://pastebin.com/iiEsdMkz in /usr/lib/ec2-ami-tools/lib/ec2/platform/linux/image.rb <\sh> guys, does the eucalyptus cloud thingy, which we ship as UEC, support XEN somehow? <kim0> \sh: AFAIK it's KVM out of the box <kim0> Ubuntu server doesn't support xen dom0 to begin with right <kjdash> well it is running now <kjdash> i'll report back in a bit after it bundles <\sh> kim0: any reasons why we can't start to support kernels with dom0 support? <kim0> \sh: Yeah, coz upstream Linux (kernel) hate Xen patches, it's hyper intrusive to the kernel codebase <kim0> \sh: with the latest .37 kernel partial xen0 support has ben upstreamed (for the first time in years) .. it's still not enough though <kim0> \sh: Basically the future is KVM all the way .. Every Linux vendor is supporting KVM only <\sh> kim0: oh well, kvm...I tried it so many times, and all the way down I'm stucked with easy things like virtual switching....I hope this will be supported in one of the next ubuntu releases...so I can throw away some esx hosts <kim0> \sh: what's virtual switching ? <kim0> networking ? <\sh> kim0: yes...this vde thing <smoser> kjdash, https://launchpad.net/~smoser/+archive/bundle-fix-sru-test is now updated. <\sh> kim0: http://vde.sourceforge.net/ <kim0> \sh: well that's not related to kvm vs xen <kim0> \sh: you might be interested in http://openvswitch.org/ too <\sh> kim0: any preferences for ubuntu server? <kim0> \sh: btw, UEC does its own vlans to separate customers .. etc <kjdash> smoser: made it all the way through the bundle that time, gonna test the resulting image real quick <\sh> kim0: how? I'm looking for something similar like esx vswitches...and kvm on ubuntu didn't have support for vde (while kvm-72 already introduced it) <kim0> \sh: I'm just saying with UEC, every user's VMs get their own vlans, no need to do anything to manually set the mup * kim0 lunching <kjdash> smoser: yays, i can report success <smoser> kjdash, still around ? <kjdash> yep <smoser> kjdash, did it work for you? were you able to test the ~ppa2 ones ? <kjdash> [13:17] <kjdash> smoser: yays, i can report success <smoser> hm.. why iddn't that get to me. oh well. thanks. <kjdash> np, that was from the pastie, just updated from the ppa <kjdash> i have a few more images to bundle, so i'll report back any errors #ubuntu-cloud 2011-01-15 <waltc> is kim0 on? <waltc> has it always been the case that when logging into an instance as ubuntu you are automatically root, as in sudo -i? <waltc> I didn't think it was prior to 10.10 but I noticed it when launching natty and maverick images under kvm natively #ubuntu-cloud 2011-01-16 <cdbs> I have seen that the numbers of AMIs are different, on http://uec-images.ubuntu.com/releases/10.10/release/ and when we click a link on the AMI numbers, the resultant page has a different number. Is this normal? Do both links point at the same image? <tenshy> Hi ! <tenshy> I've a few questions about user-data <tenshy> I want to start an instance and pass a script to it with a parameter depending of the region <erichammond> tenshy: That makes sense. What's your question? <tenshy> I've found that I can set a user-data-file when launching an instance with bash <tenshy> but I would like to do the same using aws php-sdk <tenshy> because crÃ©ating AMIs is too long <tenshy> (and I'm a poor student puting a lot of bugs in his source code) <tenshy> If I put an option who sets the userData, it seems that nothing happens <erichammond> tenshy: So you got what you wanted to do working with shell commands, but not with PHP? <tenshy> I must assume I've tried in shell commands <erichammond> tenshy: Are the shell commands doing what you want? <tenshy> I'm trying it <erichammond> ok <tenshy> I've some matters with the credentials #ubuntu-cloud 2011-01-17 <areay> hi all.. i'm using UEC -- is it possible to have more than one VM per cpu core? i understand this is the default but is there a way to override it? <kim0> areay: Hi .. are you saying if your NC has 2 cores .. you can only launch two VMs ? <TeTeT> areay, kim0 : there's a MAX_CORE or MAX_CORES setting for the nc that overrides the default <areay> kim0 -- i haven't actually gotten that far yet lol.. but the ubuntu server guide says that "by default, Eucalyptus will only run 1 VM per CPU core on a Node." <areay> TeTeT, how do i configure that setting? <TeTeT> areay: I have the following entry: <TeTeT> /etc/eucalyptus/eucalyptus.conf:MAX_CORES="32" <TeTeT> on the node controller. So this node can host up to 32 vm's, which is a bit high, but it's only for training and testing <areay> thanks bro :) sorry i'm new at this -- that should have been the first place i looked <areay> TeTeT, how do i reload the settings after i edit the file? i better ask or i'll end up rebooting like a noob lol <TeTeT> areay: sudo restart eucalyptus-nc or sudo restart eucalyptus-nc CLEAN=1 <areay> tyvm! <TeTeT> areay: be aware that you loose any already running instances on that node <areay> np i don't have any yet :) also my node is saying kvm disabled by bios -- is that as big of a problem as it sounds? <kim0> I think so .. visit bios and try enabling any virtulization thing <areay> kk cool i'm on it :) thanks u guys are stars <areay> ok, i've downloaded and installed an image... once i instantiate it, what happens if it, the NC, or the CC reboots? do i need to save snapshots? i'm kinda lost :/ <areay> ok ignore above post -- my instance is running... but i cannot ping or connect via ssh. i instantiated using hybridfox, and when i choose "connect" it's asking me for a private key file? where do i find this key? <areay> ok i'm getting further -- i get output from ssh now but i need that key :/ <areay> <-- stupid. i didn't create a keypair before i instantiated... disregard above question ;) <TeTeT> areay: good luck with your discovery of uec :) <areay> haha thanks tetet :) <superxgl> echo -at $logdir <superxgl> oh,sorry <kim0> :) <areay> Hi all... I have a couple questions about the basics of UEC/cloud computing in general -- firstly, i understand that instances are considered disposable - if i want specialized "appliances" (mysql, vpn, etc) do i need to create custom images in order to be able to swap them out on the fly with minimal configuration? <areay> or do i create a "snapshot"? <areay> also, i understand that sensitive data should be kept on a "volume" -- are these volumes held on the nc or on the clc/cc/sc/walrus box? <areay> if anyone can answer even one of these questions i'd be extremely appreciative :) <kim0> areay: yes instances are disposable, yes important data should be on a volume .. which is stored on SC <kim0> areay: for appliances .. I would boot a generic image then customize it using something like cloud-init and/or puppet <areay> kim0, so customize it as a running instance? or chroot into the image? --also once the image is done what's the best way to save it? <kim0> areay: It's not like that .. the "modify instance, then make an image out of it" <kim0> is a valid approach .. but not the solution I'd recommend .. instead <kim0> I'd boot generinc image, it picks up customization instructions from cloud-init and/or puppet, it becomes "mysql-appliance", then mounts ebs volume which your data, now it's your-appliance <kim0> no snapshotting needed <kim0> everything can be rebuilt in seconds <areay> kim0, sounds good -- which is better, cloud-init or puppet? <kim0> areay: ehum .. consider them complementary :) <kim0> cloud-init is for initial bootstrapping .. you can stop there .. or you can continue with puppet <kim0> read more about both .. and see what you want to use <areay> sorry for the noobish questions -- i'm VERY new to all this and the documentation available is craighoward@example.net.. i was using multi-roled servers before.. is there like a guide or anything? a simple guide to setup a lamp appliance or something would be ideal so i can see all this in practice... <kim0> areay: check out my cloud-init stuff at http://foss-boss.blogspot.com/search/label/cloud-init <areay> awesome :) tyvm kim0 <kim0> areay: read bottom to top :) has wordpress example (LAMP stack) and all <areay> kim0, nice blog btw -- i take it that's yours :) <kim0> yep :) * kim0 afk for now #ubuntu-cloud 2011-01-18 <superxgl> hmm...UEC is easy to install, but it only supports kvm, it's a pity that my cpu do not support vt-x <superxgl> so i have to use XEN+Centos+Eucalyptus <kim0> Hi folks, Walt Corey, an Ubuntu community member has put together a document summarizing "surprises" faced when installing UEC. The target is to come out with a set of recommendations/best-practices for UEC <kim0> Can you guys please take a look, leave comments ..etc Here it is https://docs.google.com/document/d/1xm5mRFXT-0cla2JG5Bm7b4oQVlxPR2KM0Ak8DzHVxak/edit?hl=en# <ubuntucloud007> ahoy hoy, I've just installed my first enterprise cloud, I'm following https://help.ubuntu.com/community/UEC/CDInstall#STEP%207:%20Run%20an%20Image but when I try to start an instance, it stays in pending for a long time then switches to terminated <ubuntucloud007> I'm kind of at a loss for how to debug it <kim0> theTrav: congrats :) <theTrav> is there a log file somewhere for what the cloud was doing when attempting to start that instance? <theTrav> thanks kim0 <kim0> theTrav: what about /var/log/eucalyptus/* on the NC <theTrav> looks promising, thanks kim0 <kim0> theTrav: also try creating a new user, not the "admin" one .. for some reason some people are reporting problems with that <theTrav> oh, you said NC not CC <theTrav> how do I determine which node it attempted to start the image on? <kim0> theTrav: when installing you must have chosen a node to be your node-controller ? <theTrav> kim0: sorry, got called away for a bit... I thought node controllers were for individual nodes and the cluster controller controlled groups of nodes <theTrav> or am I getting my terminology mixed up again? <kim0> theTrav: that sounds good <kim0> NC is what runs KVM and runs your VM <theTrav> yeah, so my cloud has two nodes and a cluster controller <theTrav> so if I'm starting an image <theTrav> I assume the VM is running on one of those nodes <theTrav> ? #ubuntu-cloud 2011-01-19 <tsarles> Im interested in ubuntu cloud, but it seems almost as if there isn't basic information availible about how it works. I imagine UEC takes the place of say VMWare? But what about storage clustering, does it do that sort of stuff too, or do you need a SAN? is there anywhere i can read up on the basics? <tsarles> is there a v-motion sort of feature for high availability? <theTrav> tsarles: check out walrus <theTrav> I'm super new to cloud as well, but it seems like walrus takes care of that side <theTrav> ok, so firing up elastic fox to try to connect to my cloud, I'm unable to add the cluster controller <theTrav> looking in cloud-error.log I've got NoAcceptingPipelineException <theTrav> do I need to set up / start some sort of service on the cloud side to handle the WS interface? <theTrav> hmm, on another angle that's possibly related, I am trying to run these things on a network that already has a DHCP server that I have no access to <theTrav> from what I can tell (euca-describe-instances) it's trying to do something with (864)930-0890 however nothing on the network has that IP address <theTrav> I notice the CC has (864)930-0890 on its interface <theTrav> so I'm guessing that's something to do with the --addressing private that the instructions told me to use <areay> hi all... i'm having trouble creating a volume with euca-create-volume... it returns "creating", but after i run euca-describe-volumes see "Failed" the first time, then "Deleted" <areay> I get this each time in /var/log/eucalyptus/cloud-error.log: 03:53:54 [BlockStorage:pool-10-thread-2] ERROR com.eucalyptus.util.EucalyptusCloudException: Could not export AoE device /dev/vg-EnZbTA../lv-hkAQIA.. iface: eth0 pid: 3181 returnValue: <areay> nm, i noticed the error said eth0; i've configured the network using eth1.. i changed it in the web interface under configuration and it works perfectly now :) <theTrav> so if I'm running my cloud behind a proxy, is there any way for the "store" tab on the web interface to work? <theTrav> like for downloading images etc? <theTrav> I should be able to ssh into a VM instance running within the cloud from outside the cloud shouldn't I <theTrav> it seems I can ping it from the cluster controller but not from another computer on the same network <flaccid> theTrav: configure security groups correctly for the instance
{'channel': 'ubuntu-cloud', 'month': 1}