#ubuntu-uds-core-1 2014-03-10 <agus> ls #ubuntu-uds-core-1 2014-03-11 <sofeafaef> Hi all <sofeafaef> Hi to all <ogasawara> https://plus.google.com/hangouts/_/hoaevent/AP36tYdwzZ1kXc6oi121rGs9L9ex5ZAZ7_MJRoCt3gGWBiIM--djug?authuser=0&hl=en <ogasawara> smagoun, jibel, xnox ^^ * ogasawara looks for others <mdeslaur> \o <ogasawara> mdeslaur: https://plus.google.com/hangouts/_/hoaevent/AP36tYdwzZ1kXc6oi121rGs9L9ex5ZAZ7_MJRoCt3gGWBiIM--djug?authuser=0&hl=en <ogasawara> mdeslaur: hiya <ara> hello! <xnox> o/ <roadmr> hi <ara> shouldn't https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule be updated to show that date? Aug 7th 12.04.5? <smagoun> ara: yes <smagoun> :) <rbasak> What Adam says makes sense. But the flip side is that users would ideally always be able to install from the latest image, and never get a message immediately. <xnox> as long as precise+trusty-hwe upgrade to trusty <bdmurray> what about landscape? <jamespage-uds> rbasak: dhenrich on the landscape team <jamespage-uds> ogasawara: ^^ <ogasawara> jamespage-uds: awesome, thanks <rbasak> apw: that one also needs to be able to answer people's question "will this be happening to me?" in advance of an actual notification. <smagoun> ogasawara: I need to drop for another commitment - thank you for facilitating this <ogasawara> smagoun: thanks <ogasawara> jodh: https://plus.google.com/hangouts/_/hoaevent/AP36tYeuCFlc0lIjfDjhzuNFhXzVCok55HB8kUFPhezu3jicPS7A4Q?authuser=0&hl=en <ogasawara> jodh: no hurry, you've got 10min <ogasawara> jodh: let me know if you need help pinging participants <jodh> ogasawara: thanks * pitti waves hello * slangasek waves <slangasek> pitti: https://plus.google.com/hangouts/_/hoaevent/AP36tYeuCFlc0lIjfDjhzuNFhXzVCok55HB8kUFPhezu3jicPS7A4Q?authuser=0&hl=en <ogra_> what about services that systemd needs, how do we transition things like rsyslog ? <cjwatson> $ dpkg -L rsyslog | grep 'systemd.*service' <cjwatson> /lib/systemd/system/rsyslog.service <cjwatson> ogra_: ^- <ogra_> oh ? <ogra_> i thought that wasnt possible anymore <ogra_> cool then <cjwatson> yep, you can absolutely have both upstart and systemd configs in the same package <gQuigs> anyone played with this? https://launchpad.net/~ondrej/+archive/systemd * gQuigs couldn't get it to work <ogra_> cjwatson, no, i mean using rsyslog on systemd ... <ogra_> i thought systemd needs journald <ogra_> as an essential bit <cjwatson> the systemd journal writes through to rsyslog <tedg> It seems like using Upstart/Systemd together means that we have more flexibility on the session side. <cjwatson> that's the default config in Debian already <tedg> If the session systemd can't run without a system systemd <ogra_> sweet <sergiusens> property watcher for android is system <cjwatson> you get both information in the journal, and traditional logs <cjwatson> the journal isn't nearly so evil a thing with that as it's been painted <ogra_> well, i cant use less on the logs :) <ogra_> reluctancy to change and all :) <cjwatson> you can absolutely use less on the rsyslog-generated logs bridged through from the systemd journal <ogra_> right, but i cant use it with plain journald logs <cjwatson> I would encourage setting up a quick Debian VM, configure it to boot with systemd, and play with that <ogra_> i'll surely do that (not right now indeed) <cjwatson> ogra_: you can use journalctl/systemctl or whatever, or you can use less on the trad logs; while I can see that you want to look at the raw journal files if you're debugging systemd, I don't understand why that would be justified by a reluctance to change <ogra_> sergiusens, well, before we even think about how to make the upstart bridges work with systemd we need to see if the android kernels can even run with systemd without to much patchwork :) <cjwatson> if you're reluctant to change, you read the traditional logs and you ignore the journal <ogra_> cjwatson, i know many people that run silly screen scraping shellscripts on logfiles etc <sergiusens> ogra_, true <ogra_> cjwatson, such setups kind of assume you have plain text logs <cjwatson> ogra_: those will still work <cjwatson> /var/log/syslog is *still there* and *still contains useful content* <ogra_> right, in our setup <ogra_> (or debians) <sergiusens> ogra_, we will need to rewrite our overrides ourselves as well <ogra_> "raw" systemd wouldnt though <ogra_> sergiusens, yeah :( <cjwatson> ogra_: why would I care about something that isn't our setup? :) <cjwatson> or why should we care <ogra_> heh <ogra_> well, i dont know the debian setup yet ... * sergiusens stops commenting for ogra to single thread :-) <cjwatson> anyway, my point is that the logs thing is a non-issue * ogra_ was always very happy with upstart ... i never felt the need to try other inits <ogra_> right, i didnt know that debian had a solution already <gQuigs> that seems painful to debug .... <gQuigs> ... https://bugs.launchpad.net/upstart/+bug/160150 is this relevant? nit: cannot be run as a PID other than 1 <udsbotu> Launchpad bug 160150 in upstart "init: cannot be run as a PID other than 1" [Wishlist,Triaged] <ogra_> hmm, that used to work <ogra_> (you were able to boot with init=/bin/sh and run /sbin/init manually for debugging) <ogra_> sergiusens, the android property watcher just writes to a socket, isnt it ? <ogra_> i think systemd should be capable to read from that <rsalveti> probably <apw> slangasek, can we not just move the system jobs to like /etc/init/system/* and only run those if pid==1 and not pid==2 <slangasek> apw: could be done, but that's a lot of glue code to add to upstart that doesn't currently exist <apw> slangasek, by which i mean when we add a new systemd unit to the job move the upstart job to the systemd directory, so we know it is only needed when not using systemd <apw> "a lot" is it really <slangasek> ah, you mean moving them out of the standard upstart path so that they're not seen when booted as a helper, ok - seems the discussion on the hangout has caught up a bit :) <gQuigs> which version of systemd are we aiming for? <apw> slangasek, right if when you convert a job to system as well, you move those into a second place, upstart uses both dirs, upstart helper does not use the "converted" directory on its path <xnox> apw: don't use subdirectory in /etc/init as upstart reads all subdirectories and it's all valid jobs. <slangasek> right <xnox> gQuigs: that bug is not relevant, upstart can run as both session or system init as not pid1. Most of our unit test rely on that. We have some safety checks in place, which will need to be tweaked, but nothing major. <xnox> slangasek: apw: at the moment, we disable running system upstart with multiple config dirs, but that can be updated/re-enabled. <ogra_> do you guys plan to cover the android related bits too ? * ogra_ doesnt see any recent notes about it, we need to make sure the helpers we use inside the android container still work or are portable ... and indeed there is that android kernel question ... can systemd work on 3.0 kernels etc <rsalveti> at least on >= 3.4 <sergiusens> rsalveti, that's min req I guess, but will it work? <rsalveti> not without patching the kernel I believe <xnox> pitti: but our 204 is with or without stable-branch patches? <rsalveti> someone needs to get an action to check that for the next cycle <pitti> xnox: not a lot of backported patches, mostly just for hwdb <pitti> xnox: nothing stops us from pulling some in, of course <xnox> pitti: ok. * ogra_ guesses the guys in the session dont read IRC <ogra_> :P <xnox> cjwatson: but touch is next session =) <ogra_> xnox, well, touch has also system jobs :P * tedg has jobs that use the apparmor stanza :-) <ogra_> next session is about session <pitti> heh <tedg> the session session <xnox> pitti: slangasek: yeah, how are we going to support android-bridge on systemd? <slangasek> xnox: in the first iteration, is there any reason not to run that under upstart compatibility? <ogra_> xnox, ++ <ogasawara> https://plus.google.com/hangouts/_/hoaevent/AP36tYda5hjUl0qHK6-TMT6DkfUsU8iSeC_i8DATvp-9NyIO21R-WQ?authuser=0&hl=en <ogra_> slangasek, well, long term we need to switch <xnox> slangasek: sounds good to me. I just thought phone could switch to systemd without any compatibility layer, as it doesn't need gregory37@example.net. <ogra_> and we start to rely more and more on system properties from android <cjwatson> xnox: good to be accurate anyway :) <ogra_> xnox, well, only if the bridge helpers work on systemd <slangasek> xnox: well, it needs one for the android bridge, initially :) <ogra_> and for session stuff we rely 100% on upstart today <xnox> slangasek: cjwatson: e.g. do systemd unit files support arbitrary new stanzas and/or generic events thingies? <ogra_> (way more than desktop for example) <cjwatson> apw: I think we should avoid doing this in a way that requires us to do extra per-job work just to make upstart jobs work when they haven't been converted to systemd yet <cjwatson> that way round means that we probably won't meet any sensible compatibility goals <cjwatson> and in many cases if we're going to touch a package anyway it's just as easy to write a systemd unit for it <cjwatson> e.g. I wrote systemd units for click which I'm pretty sure will work even though I've never run them <slangasek> ogra_: do you want to join the hangout? <ogra_> slangasek, i'm fine with IRC if someone pays attention to it :) * ogra_ would need to find a place where he can speak <ogra_> (not easy atm) <slangasek> ogra_: well, /I'm/ not fine with IRC, the hangouts are used for a reason :) * tedg likes a silent ogra_ ;-) <ogra_> :P * ogra_ reloactes <slangasek> ogra_: (best case, there's significant lag between us talking and you seeing it on the feed) <tedg> So an initial topic, it seems like we can't do any session migration until systemd is PID 1, right? <xnox> tedg: correct, but once it is, we need a plan for conversion. <tedg> But that's for trust + 1, right? <tedg> We can't even think about it until then. <ogra_> slangasek, gimme the HO url <cjwatson> Some time between trusty+1 and trusty+4 <cjwatson> We can absolutely think about it <cjwatson> And should <slangasek> ogra_: https://plus.google.com/hangouts/_/hoaevent/AP36tYda5hjUl0qHK6-TMT6DkfUsU8iSeC_i8DATvp-9NyIO21R-WQ?authuser=0&hl=en <tedg> I guess I think the session is going to be much easier to migrate than the system. <tedg> We're not as complexly invested there, we simply haven't used it as long. <slangasek> tedg: I don't agree that it's going to be *much* easier <tedg> I'd love to avoid getting to tied into Upstart, but I do want to use "modern init features" more and more. <cjwatson> There are things like associations with :sys: events <tedg> slangasek, Well, I'm scared of servers, so it might be a perspective thing :-) <slangasek> the uses of upstart in the session are largely orthogonal to how we're using it in the system init <xnox> tedg: modern features would be using dbus-activation =) <xnox> tedg: as that's managed under systemd ;-) <tedg> That makes dbus-activation not suck. But it still doesn't align with purpose of session length jobs :-) <tedg> +1, I think flag day. <xnox> jdstrand: are you about? <jdstrand> I am here <xnox> jdstrand: join the hangout. <xnox> jdstrand: https://plus.google.com/hangouts/_/hoaevent/AP36tYda5hjUl0qHK6-TMT6DkfUsU8iSeC_i8DATvp-9NyIO21R-WQ?authuser=0&hl=en <tedg> ogra_, Unity shouldn't see much change, we've already abstracted it with libupstart-app-launch <tedg> We just need to port UAL. <xnox> tedg: and what would that involve? <tedg> xnox, I mean I think it's more just "doing it" I'm guessing it's a week or two of work. Just need to schedule it. <ogra_> tedg, well, doesnt ubity-mir do some lifecycle stuff that also might want to talk to upstart ? <tedg> xnox, I don't think it's worth doing until we think we can have a systemd as PID 1 though. <tedg> ogra_, AFAIK it does all of that through libual. <xnox> tedg: well, we'll be able to start testing it as soon as we have ppa up with systemd as pid1 <ogra_> tedg, ok, so if you think we can easily transition that one to systemd we should be good <tedg> xnox, Sure, but my manager doesn't take "I want to play with a PPA" as a good reason to schedule my time :-) <xnox> tedg: which is in a matter of weeks, early 14.10. not something "in the very long future" <xnox> tedg: ask slangasek to convince your manager ;-) * ogra_ would really prefer if we could stay on upstart with the phone until at least 15.04 <ogra_> (for sessions) <cjwatson> Could you elaborate on reasons? Is this just risk-aversion? <ogra_> the system is far from being readily designed <tedg> I'd rather design it more on systemd. Get it early, get it right. <jodh> ogra_: FYI, I've started on a wiki page (it covers overrides btw): https://wiki.ubuntu.com/SystemdForUpstartUsers. More input welcome! :) <serue> jodh: oh, nice, thanks for that <slangasek> separately, the extensive use of overrides on the phone is really rather problematic <pitti> jodh: hah, nice cheat sheet! <sergiusens> tedg, so the u in libual will s/upstart/ubuntu/? <tedg> sergiusens, Heh, that'd be a great idea! <jodh> pitti, serue: :) I was hoping that that page would form part of the release notes once we've switched (or made that an option atleast) * tedg wants to name it bob <sergiusens> ogra_, that's a problem from the prior session ;-) <pitti> stgraber, ogra_: child reaper support is linux >= 3.4 <ogra_> sergiusens, yeah it was not dealt with in the former session, so i thought i should bring it up <tedg> I don't think UAL is a "pain point," but it's a work item that we need to schedule. <tedg> It's a significant thing that needs to be ported. <tedg> I'd like to avoid supporting both at the same time, but if we have to, that's probably possible. <cjwatson> I think we have to have some degree of dual running, both at the system and session levels <cjwatson> I suppose worst case we could fork a systemd-app-launch if you really don't want both in UAL ... <ogra_> hopoefully only during development <cjwatson> (it'd want to be renamed anyway, right?) <tedg> Eventually I think it should be. <tedg> I like sergiusens' idea of "Ubuntu App Launch" though :-) <ogra_> why does it have the session manager in its name ? <tedg> We wouldn't have to support two APIs in Unity for sure. <ogra_> it is used for launching click apps ... * ogra_ would call it click-app-launch :) <tedg> Eh, I don't know. It was originally supposed to be a couple of Upstart jobs. It got feature creep :-) <sergiusens> tedg, you forget the surfaceflinger/mir transition (still on going) <sergiusens> :-) <xnox> tedg: right it would be one or the other. you will not need to launch one app with systemd and another with upstart within the same session. <tedg> sergiusens, I'm trying to forget ;-) <cjwatson> ogra_: it's used for launching apps, not exclusively click apps <tedg> It's also used for managing them. Finding PIDs, etc. <ogra_> well, currently it only launches click apps ... due to circumstance :) <xnox> ogra_: it launches click and legacy. <sergiusens> ogra_, no, it launches legacy apps too <tedg> Webbrowser isn't a click app :-) <cjwatson> ogra_: perhaps, but consider the long tail of apps with unity8 on the desktop <ogra_> oh <ogra_> i stand corrected :) <ogra_> cjwatson, well, they will hopefully all become click too ;) <cjwatson> "long tail" <ogra_> :) * sergiusens doesn't wnt to convert any more apps; too hard with the tight train schedule <cjwatson> as the click author it is not my goal to stop people launching apps when they aren't in click format <ogra_> sergiusens, just wait for libreoffice ;) <cjwatson> I do not expect many apps to *ever* be converted, particularly people's local apps that they haven't rebuilt since 1995 <cjwatson> but this is for another session I suppose :) * ogra_ does ... but thats because i can read asacs mind :) <cjwatson> asac is wrong :P <ogra_> heh <xnox> ogra_: one _never_ recompiles go-apps! <xnox> =)))) <ogra_> xnox, who cares about go :) i want firefox and libreoffice click packages :) <xnox> ogra_: nobody else wants those at all =) <ogra_> convergence ! <xnox> ogra_: webbrowser-app https://drive.google.com <ogra_> pfft <ogra_> :) <gQuigs> how can community contributers best help? <slangasek> gQuigs: by taking workitems from the blueprint? :) <gQuigs> err.. I mean from the perspective of changing jobs over.. when it's time to start, etc <ogra_> gQuigs, and if you dont want to do that, we should have something to test at some point and someone will blog about how to use it and we'll need people to test the hell out of it <gQuigs> ogra_: right <ogra_> for changing jobs over we will first need the basics in place ... <ogra_> once thats there you should be able to go wild on it :) <gQuigs> ogra_: cool <cjwatson> right, probably simplest to wait until we have a PPA you can work with and then start converting jobs where dependencies allow <rsalveti> the android one is also system level <ogra_> does it pick up anything there yet ? <rsalveti> no, not picking at system level, I mean, it's part of system, not session <ogra_> right <ogra_> stgraber just translated your brazilian for me :P <sergiusens> ogra_, that last comment came from the future <sergiusens> :-) <rsalveti> yeah, this delay is quite interesting <ogra_> heh <tedg> Thanks! * ChanServ changed the topic of #ubuntu-uds-core-1 to: Currently no events are active in this room - http://summit.ubuntu.com/uds-1403/core-1/ - http://irclogs.ubuntu.com/2014/03/11/%23ubuntu-uds-core-1.html #ubuntu-uds-core-1 2014-03-12 * ChanServ changed the topic of #ubuntu-uds-core-1 to: Currently no events are active in this room - http://summit.ubuntu.com/uds-1403/core-1/ - http://irclogs.ubuntu.com/2014/03/12/%23ubuntu-uds-core-1.html <asac> ho <ogra_> hohoho <asac> :) <asac> ogasawara: you run this track, right? <ogasawara> asac: yep <asac> ogasawara: when will we boot the room? <ogasawara> asac: I usually start it up about 15min before <asac> in 20 minutes early enough? <asac> cool <ogasawara> asac: then I ping the url here <asac> good <ogasawara> asac, didrocks: https://plus.google.com/hangouts/_/hoaevent/AP36tYejx4a57LGNk7fbW9HeUUpDvo-i-SHKLuRhKzSha-qxFpMxUA?authuser=0&hl=en <didrocks> thanks ogasawara <asac> ok lets see if this thing works here * ChanServ changed the topic of #ubuntu-uds-core-1 to: Track: Core | Landing process rules for ubuntu Touch | Url: http://summit.ubuntu.com/uds-1403/meeting/22174/core-1403-landing-process-touch/ <asac> bfiller: https://plus.google.com/hangouts/_/hoaevent/AP36tYejx4a57LGNk7fbW9HeUUpDvo-i-SHKLuRhKzSha-qxFpMxUA?authuser=0&hl=en <olli_> hi folks <asac> hi olli_ <karni> Someone could drop the presentation URL into the session pad, once it's over. <asac> karni: will get it posted afterwards <karni> elopio: you might want to mute yourself <karni> asac: thanks! <elopio> sorry. I'm muted now <asac> dont want to interrupt didier <dobey> anyone who is not talking should mute themselves <dobey> in every session :) <karni> elopio: np <ogra_> hmm, cant get video here <tedg> ogra_, didier is signing Hollywood music, so it's restricted. <ogra_> heh, well, it doesnt load on yourtube (nor in the embedded thing) <sergiusens> ogra_, I restarted firefox for it to work... * ogra_ only gets a spinner . <ogra_> sergiusens, igh ... certainly wont do that <ogra_> (too many open things here, cant restart it) <rickspencer3> didrocks, for the "fix ready" line ... <ogra_> (other videos work fine ) <sergiusens> ogra_, pkill firefox and restart <rickspencer3> does that include reversions? <sergiusens> ogra_, will restore all your tabs :-) <ogra_> sergiusens, i will lose several oppen endit forms <ogra_> *edit <sergiusens> ogra_, you are so complicated :-P <ogra_> and i can play any other video on youtube, just this one doesnt load at all <rickspencer3> ogra_, maybe try joining the hangout? <xnox> .. <ogra_> rickspencer3, cant from this device ... <xnox> autopilot did for sure catch failures. <ogra_> heh <dobey> ogra_: try shift-reload to reload without cache? <ogra_> it also produced a lot <ogra_> dobey, all tried <dobey> ogra_: i've found i have to do that for all the uds videos when the session actually starts <sergiusens> QUESTION: with most apps moving to autopilot and most test being against apps; is there a plan for the ci train to accommodate that? <sergiusens> or is the focus going to move to running autopilot on system components (osk, unity8, uitoolkit) <sergiusens> yes we can <didrocks> asac: rick is talking! <sergiusens> lol <dobey> i can hear rick! <geddy> we can hear him <balloons> lol <sergiusens> we can hear rickspencer3 <sergiusens> asac, restart your hangout <lool> asac: you need to stop talking <bregma> laaaaag! <lool> asac: you're watching the youtube stream or something <asac> sergiusens: well, leann also cant <sergiusens> yeah; the delay doesn't help :-) <rickspencer3> know we know why asac and ogasawara are so productive <ogasawara> heh <asac> :P <lool> are they leaving 2 minutes in the future? <lool> *living <asac> oh i remember i installed the hangout block feature <asac> for rick <kenvandine> haha :) <xnox> rickspencer3: clearly some people clicked "ignore this person" button =)))))))))) </kidding> <rickspencer3> whatever works! <tedg> It seems like a lot of the delay is in rebuilding images. But it seems like building multiple images is just a matter of CPU time. <tedg> Can we build images for "instant revert" proactively? <xnox> tedg: that's getting fixed, by moving cd building into launchpad and have a full pull of all builders which will be able to build image per each silo. <asac> tedg: image building is only 1/4th of the time <xnox> s/pull/pool/ <asac> the testing is what takes a long time (4-6h) <ogra_> image builds take ~1h <asac> we a) work on allowing parallel testing which will allows us to pipeline <tedg> That actual building. But let's say we built all the image combinations. So we were just choosing which one. <tedg> i.e. there's already an image without a certain silo landed. <asac> b) later do swarm testing so it will take just 1h or even less <asac> swarm waits for emulator <olli_> kgunn, wasn't the number of silos somethnig you mentioned in an earlier conversation <dobey> x86 emulator will make testing a lot faster <asac> tedg: we have a bottleneck in our official image build infra <tedg> asac, By swarm you mean AP with gemgem? <asac> ogra_ knows what that bottleneck is <ogra_> do we ? <asac> tedg: well, taking 10 phones and running one AP on each instead of sequence <asac> as an example <kgunn> olli_: yes, more specifically - dedicated silo, available all the time to a team/proj <asac> ogra_: we cant produce multiple images in parallel, right? <asac> people told me we wait for buildd cloud <ogra_> asac, ah, right, they get queued ... <asac> ogra_: what part of infra cant do parallel? <kgunn> olli_: but this is what didrocks calls "airline" vs train ....so some x amount of infrastructure or change to it needed to accomplish that <ogra_> but that part of the build only takes 20-25min <tedg> Can we "swarm test" manual testing by using the QA test tracker? <ogra_> so we could raise the frequency a bit <tedg> Basically make one entry per-image like we do with daily CDs? <asac> tedg: we could look at swarming manual testing, but problem is that with manual testing the testers need to be calibrated <asac> so you cannot just test once in a month one test case <asac> usually you will report confusing things etc. <asac> our problem is really getting the AP results asap <tedg> Sure, but couldn't the calibrated testers just test the cases that are raised by the swarm? <rickspencer3> I wonder if there is a way to make reversion faster and more reliable, but without slamming upstream trunk <asac> rickspencer3: we revert without slamming upstream trunk <rickspencer3> like could we keep copies of the last binaries around and put those into the image or something <asac> rickspencer3: reverts are done in archive <rickspencer3> ? <rickspencer3> right, I think that's a good thing that we want to keep <rickspencer3> but maybe we could make rollbacks easier and keep that goodness <tedg> I guess for me I'd say we want rollbacks quicker so that they can be decisive and make it so that other things can land. <rickspencer3> in other words, can we make it so that there is nothing that is "hard to rollback" <xnox> rickspencer3: old binaries are available from librarian, but there is no guarantee they are installable when everything else moved on. Using old version binaries works for e.g. image-based upgrades. But doesn't help other silos/packages (they still see higher/broken version) or apt-get based machines. <rickspencer3> xnox, but could we not calculate all the binaries that changes in the rollback? <asac> i think we could do it by grapping old binaries, but it becomes pretty tricky for everything not a leave package <asac> grepping * ogra_ thinks we just need to become better with testing so we dont have to roll back *after* something entered <asac> err :) you know what i mean <xnox> rickspencer3: from legal point of view, we build from the archive, as we publish matching source-code for those versions. <xnox> rickspencer3: calculation could be done, but it would need verification that it indeed doesn't regress. <rickspencer3> that doesn't seem like a blocker, I mean, the source code is available in the commit history <rickspencer3> so it should be easy to get the source that is published <xnox> rickspencer3: what i have seen with reverts, is that revert happened without consulting the uploader (myself), which is a no-win revert as it re-introduces fixed things. <rickspencer3> to be clear, my implementation is maybe not good, but I think it's worth thinking about how to do rollbacks smarter <pmcgowan> Question: are all regressions treated equally or is there some analysis of minor regression vs major feature landed and needs to be released <tedg> rickspencer3, +1 <cjwatson> I think we *are* doing rollbacks quite smartly, and people are asking for them to be done more quickly but less smartly ... <cjwatson> that's the strong impression I get <rickspencer3> cjwatson, interesting point <tedg> For me I'd like rollback, etc, to be more automatic. Not a "someone decides how important" it's just a part of the machine. <xnox> rickspencer3: the way we do reverts, of a single package, not a full revert to previous image number, it still needs to go though full testing. As it's a new state. It has been identified to resolve /a regression/, but it's rarely verified fully to not introduce any /new regression/ as those reverts don't go through full test-plan. <cjwatson> the pushback always seems to be in cases when we're in the middle of doing proper root-cause analysis <rickspencer3> more quickly but also still smart sounds good <tedg> So then things move on quickly. <rickspencer3> cjwatson, I'm curious why that impacts the developers, I would think didr0cks could do a rollback to keep the image green while the developers keep going with their root cause anlaysis <rickspencer3> what am I missing? <xnox> didrocks: QUESTION: why reverts don't go through silo and a full test-plan execution? <cjwatson> rickspencer3: I've suggested before that this could trivially be done by having a separate PPA that we copy old versions into <rickspencer3> interesting <cjwatson> and having that pinned through the roof in the image build process <rickspencer3> in the meantime, profile before you optimise <cjwatson> so far my suggestion has been largely ignored though *shrug* * ogra_ sighs ... no dice for me with the video, even on another machine with different DSL connection :/ <rickspencer3> the 5 hour automated test lag is clearly where we should focus <pmcgowan> cjwatson, then the image does not match trunk or archive, is that a concern? <cjwatson> pmcgowan: I think that's fine as long as we keep the list of active rollbacks small <cjwatson> which should be good engineering practice anyway <olli_> QUESTIONS: one thing that Google CI supposedly does well is to relate code changes to relevant test cases automatically and then run only that subset (automated). Have we any plans on targeting testcases more precisely? <xnox> rickspencer3: re:automated lag - it takes less than 5 hours on armhf emulator, yet adding emulator to ci has been stalled it seems. <rickspencer3> xnox, I thought you were going to add x86 emulator so that we could quickly scale out automated testing <tedg> didrocks, asac, can we come up with a metric for "risky" so that people can understand it? <xnox> rickspencer3: armhf emulator is available and it scales well on top of openstack. x86 emulator is in progress in bring up. <tedg> i.e. not assigned by people, but by an algorithm <xnox> rickspencer3: i made intial proposal to add amr emulato to jenkins / ci, and someone from #ci was working on it, but no updates for a few weeks now. <xnox> s/amr emulato/arm emulator/ <sergiusens> rickspencer3, when you want to talk mute asac and unmute when done; should provide better sync and collision avoidance :-) <ogra_> :) <xnox> didrocks: QUESTION: there was full-train stop, which did delay python2-removal for example. <rickspencer3> just like irl <ogra_> duct tape ? <asac> ah rick is talking <asac> hehe <didrocks> asac: yeah :) <didrocks> asac: will tell you here <didrocks> (I hope someone can relay the question in the hangout, I've my slides focused) <asac> for me the definition of risky landing could be: if your testplan is affected by current regressions, you are at risk of landing more regressions <asac> that you dont see <tedg> I don't think it does. I think we need a way to ensure that "risky" is transparent. If you can't measure it, it really can't be part of the determination. <asac> so its not so much where in the stack you are, but its kind of correlated <rickspencer3> </soapbox> <asac> tedg: is that a comment on my proposal of risk above? <cjwatson> measure> yeah, absolutely; I get the sense that the "risky" category is kind of prejudice / based on something that happened months ago <asac> feel thats pretty measurable. you have a risky landing if your component testplan is currently not 100% green <cjwatson> sometimes, anyway <tedg> rickspencer3, Isn't that really for images that get released to users. For instance, we're not going to do a daily release to all users. I think we need a different level for daily and to-user releases. <ogra_> often enough, yeah <asac> i think with this it will usually correlate on where you are on the stack <asac> but it wouldnt be just "you are low, so you are risky" <tedg> asac, I'm not sure what you mean by "component testplan" <ogra_> tedg, the testplan for the component you maintain <asac> tedg: every component like mir has a testplan. thats manual checks + a set of autopilots that folks run in the silo <tedg> Yes, but how is that "green or not" ? <ogra_> tedg, either your change passes or not <balloons> I think even in normal states, it's difficult because nothing can merge to trunk <tedg> We have no place that it's getting recorded. It's "green" in someone's head. <asac> tedg: if you can run all the manual checks and autopilots tha tyou need for your landing in the current image and it is 100% succeess <asac> its green <balloons> trunk should not be the same as the archive <asac> tedg: no its recorded <asac> the manual part not. but the APs <ogra_> well, the manual part is kind of as well <asac> right. <ogra_> by you setting the silo stuff <tedg> asac, So if all the APs for a component is green, you consider it non-risky? <asac> but we haven't figured how to ensure people that have testplans know whether their manual testplan is currently affected <ogra_> if you say "tests passed" we have to assume your manual tests worked <asac> tedg: all APs that you need to run for a component landing ... which is more than just the AP for that component <balloons> under a normal process, you would have a release branch, a development branch, and feature branches. Features land into development, and occasionally you would push those then to a release branch <asac> tedg: e.g. if you land mir we also run a bunch of app APs, unity8 AP etc. <asac> tedg: so the APs in the component testplan being green == non-risky in the sense of phonecon1 <tedg> asac, How is that list generate? <tedg> generated? <ogra_> tedg, you as the developer define it <asac> tedg: we start with a good initial guess out of experience ... <asac> tedg: and if we identify that something slipped through, we grow that list <asac> tedg: we try to not start with the complete list of reverse dependencies to stay efficient <olli_> kgunn, isn't that how Mir is working atm? <ogra_> you have to provide an initial testplan when getting the silo <tedg> asac, Where is it recorded? <asac> tedg: supposed to be recorded in the testplan <ogra_> tedg, on the landing sheets <xnox> .. <cjwatson> tedg: https://wiki.ubuntu.com/Process/Merges/TestPlan/<foo> * asac is brave and reconnects <sergiusens> a year and a bit ago we had the concept of a release commit which only then would push to our archive (which was a PPA at that time) <xnox> e.g. with ubuntu-ui-toolkit we tried to land 17 branches in one go, which ofcourse, failed to merge and 4 branches did conflict between themself. <ogra_> tedg, when you request a silo, you have to link your testplan ... when you did the testing, you have to note in the silo spreadsheet that it passed (or not) <sergiusens> so people could merge to trunk and _release_ when it's ready <asac> \o/ i can hear rick :) <xnox> and therefore was pain to land, as there was no sensible conflict resolution. 4 branches got kicked out, and eseentially landed one by one, weeks later. <cjwatson> rickspencer3: anything that provides an API to anything else has a clear incentive to get things into the archive so that they can land changes that depend on new APIs, IME <tedg> So, I'm confused. So for every image, QA takes the list of every silo landing and generates a list of APs to run. <ogra_> tedg, no <tedg> Where is the list of AP tests that'll be run on that image? <cjwatson> rickspencer3: (e.g. I had a strong incentive to get libclick landed so that I could start landing the things that depend on that) <ogra_> tedg, for your very own changes you use the silo ... <ogra_> tedg, you do your testing, if your testing passes you can land it in the archive <sergiusens> cjwatson, yeah, that's when stopping the line is the most burden <xnox> .. <ogra_> tedg, images are built out of sync here <cjwatson> right, it's terrible for anything with multi-level stacking <tedg> Or 100's of MR's queued... <ogra_> tedg, and the images run *all* AP tests automatically before we consider them releasable <kgunn> olli_: yes - mir works this way today on lp:mir/devel <kgunn> and lp:mir <ogra_> tedg, so if at that point a regression is found in the component you landed, we roll it back (or as you to) <ogra_> *ask <tedg> I'm confused. So project X is risky if any run of that it declares in its test plan has failed. <dobey> all projects are risky <ogra_> right <tedg> So how do I determine if my project is risky? <ogra_> the issue when your testplan passes but the image one doesnt is most likely that your testplan misses something <dobey> tedg: if it has code, it's risky :) <cjwatson> tedg: I think there's some talking past each other here; some people seem to be talking about risky projects, others about risky landings <olli_> tedg's project are always risky <rickspencer3> tedg, I think that the landing team tells you if it's risky <cjwatson> those are obviously quite different things <rickspencer3> and what olli says ;) * ogra_ would like to get away from that "risky/non risky" stuff <cjwatson> rickspencer3: but that should be based on science <cjwatson> and vaguely predictable <ogra_> the issue we have is that the first level testing isnt good enough very often <rickspencer3> cjwatson, indeed, I was subtly pointing out a potential problem <dobey> to mee, it just seems like CI needs to be more async <ogra_> if that is 100% reliable, you wont have to roll back ever <tedg> I think it should be predictable, but I'd settle for transparent at this point :-) <cjwatson> rickspencer3: *nod* <rickspencer3> I do wonder if there is some kind of rdepends that we can run that predicts such things <rickspencer3> like, a certain weight of things that depend on you equals risk level or something <ogra_> not really, since we are working image based <dobey> more async and more automation <ogra_> one of the probs is that you test on the last image that was built... the stuff that you land lands in the next image that gets built ... <ogra_> which means you have other changes that go in alongside .... <ogra_> which can change the whole situation amnd stability <tedg> Sure, but that doesn't effect how risky the project itself is. <xnox> .. <xnox> ideally we'd have integration branch for landing, ahead of the landing. <dobey> tedg: ideally, every package/project would be treated as equally risky <ogra_> tedg, no, but you have to asses the risk always in combination with all the other changes <sergiusens> I agree with Saviq and want to add that merging a component of yours that depends on changes on many other projects (like mir or unity8) you sort of block everyone <sergiusens> it's silo-cked <ogra_> heh <dobey> and you eliminate the thinking about whether something is risky or not, because everything is <kgunn> lol <cjwatson> the problem with this is that it puts massive friction on improving our core <kgunn> "silo-cked" word of the day <cjwatson> now, it also puts massive friction on breaking our core <rsalveti> sergiusens: it'd still be an issue if we had devel/release branches <cjwatson> so I get the trade-off, but still <rsalveti> would not block trunk, but the issue would still be around <sergiusens> rsalveti, oh, I'm not stating solutions today; just wanted ane xcuse to say silo-cked :-) <cjwatson> cf. the gigantic project of Qt 5.2 <dobey> sergiusens: and then the silo-ns will nuke us all <ogra_> ++ <rickspencer3> maybe risky = time since landing + weight of rdepends + history of regressions caused <sergiusens> it was rsalveti :-P <rickspencer3> ? <rickspencer3> oops <rickspencer3> sorry sergiusens and rsalveti :) <sergiusens> np <asac> yeah, but rsalveti said "i dont want my team to do that" :) <rsalveti> another guy from south america :P <sergiusens> we ar the same team and have the same opinion <cjwatson> rickspencer3: also I think some judgement of the actual changes should be involved <rickspencer3> size of patch? <sergiusens> it's been this way for us since we started this project <cjwatson> the stuff that's already in the image, well, we've accepted the risk <kgunn> 2 other "small" things i would ask for today - self service reconfigure & not locking silo's on project use (only lock landing) <rickspencer3> maybe risky = time since landing + weight of rdepends + history of regressions caused + size of patch? <kgunn> asac: ^^ <sergiusens> but there are some corners where we might consider flag days perhaps <rsalveti> asac: because we want to share the responsibility, and we can't afford having someone coordinate trunk/release merges <sergiusens> as much as the android bump was tested; it wasn't enough; and qt 5.2 will have a similar problem most likely <rickspencer3> sergiusens, yup, exactly <rickspencer3> and that's not necessarily representing failure conditions <cjwatson> sergiusens: risk-aversion is a continuum - it doesn't IMO make sense to crank the dial all the way to one side <rickspencer3> some things are just tough <ogra_> sergiusens, thats what i'm saying all the time above ... we need to get our pre-testing better <cjwatson> the problem is that any time something goes wrong people ask "how could we have avoided this" <cjwatson> which is healthy in one sense <rickspencer3> cjwatson, well, we have the promotion as the ultimate safe gaurd <cjwatson> but it means it's very hard to say "sometimes, it just wouldn't have been worth the degree of friction required to catch this" <slangasek> managers enforcing releases once a week> that means the managers have to have eyes on all the individual components that their team has touched, right? <rickspencer3> so, it seems to me that the "risk" really comes down to a lag between promoted images <ogra_> rickspencer3, but we most of the time only discover the issues at that point <ogra_> which is what causes us to roll back <tedg> We could have sync points where everyone makes sure to have a release. <dobey> i would land MPs in trunk as fast as possible, and automate daily releases and testing <slangasek> also, what if one team is ready to do their weekly release, and it needs a change to a package from another team, and the other team has staged changes that are not yet releasable? <ogra_> we need to get better (a lot) on the first test run and with the first test plan in the line <xnox> rickspencer3: i'm ok for integration branches, but it shouldn't be an open-ended branches, but e.g. "next-landing" branch which would be limited to what is desired to land, not something open ended. Once it's full, it must land before merging more stuff into it. <rickspencer3> ogra_, right, I guess I'm saying that just because we do a landing and we find issues, it's not a freak out situation <cjwatson> tedg: we could have them every six months :) <rickspencer3> ogra_, I think we all agree with that <tedg> cjwatson, I was going to propose some interim ones, we could call them alpha/beta ;-) <rsalveti> once we get the x86 emulator we'll at least the get the test results sooner <cjwatson> tedg: heresy <rickspencer3> as I say, it seems that we are good at keeping the impact to us as developers <rickspencer3> and that promotion protects users <rsalveti> and another issue is that we can't yet give a custom image to be tested like the ones we have in the dashboard <ogra_> rsalveti, but the x86 emulator will test x86 binaries <rsalveti> most of the issues are arch-indep <ogra_> we will only see over time how reliable that even is <cjwatson> ogra_: it's still catch a lot of things <cjwatson> *it'll <ogra_> sure <cjwatson> the click problems over the last couple of days would absolutely have been caught by that <cjwatson> for instance <ogra_> yep <mardy> bfiller: +1 <rsalveti> like, if I want to land hybris or android, I'd like to be able to create a custom image and give it to be automatically tested <ogra_> but i assume there will still be many false positives <cjwatson> it'll get better over time <ogasawara> didrocks, asac: gotta start wrapping up... <ogasawara> didrocks, asac: as I'll need to shut down this hangout in a few mins <ogra_> rsalveti, use rootstock until cjwatson's PPA builds start working then * ChanServ changed the topic of #ubuntu-uds-core-1 to: Track: Core | Networking Health-Check | Url: http://summit.ubuntu.com/uds-1403/meeting/22215/core-1403-networking-healthcheck/ <tedg> I think it's important that we keep branches to releases for maintenance. <didrocks> ogasawara: ok, got it, just have 30s to say <ogasawara> cyphermox: will get your hangout up as soon as this one is done <ogasawara> didrocks: ack <rsalveti> ogra_: my issue is not creating the image, but using that to be automatically tested :-) <cyphermox> ogasawara: ah, ok <tedg> We are supporting those releases and need to base off of those. * stgraber waves <tedg> But we use symbolic naming in LP for that. <rsalveti> so later on we can have landing silo ppa -> image -> dashboard (tested with the x86 emulator) <ogra_> rsalveti, well, doanac is working on a "home use utah" for that <tedg> lp:foo goes to lp:~team/foo/trunk.14.04 <rsalveti> ogra_: right, but that's a feature I asked before we started trusty :-) <ogra_> heh <dobey> tedg: that's a bit annoying though <sergiusens> ogra_, I think that utah is mostly done; at least my tools plan is to ask him to test against that :-P <cjwatson> you can often create the maintenance branches for older series lazily <cjwatson> lots of projects never need them <ogra_> sergiusens, well, you dont really want to run systemsettle so testing locally will only take 1h instead of 3-4 <tedg> Sure, I find that's easy for us, but people who are more drive-by have a more difficult time finding those points. <dobey> merging to trunk and releasing to archive/image being separate, but closely knit, seems like the best solution to me <xnox> rickspencer3++ <pmcgowan> rickspencer3, +1 <sergiusens> ogra_, oh, these take 4h so I might be in full blown mode ;-) <ogra_> sergiusens, yeah, its awful <xnox> cjwatson: lp:project/stable <rickspencer3> does anyone notice the artwork behind didrocks <xnox> (not need to encode the codename) <rickspencer3> those are by his wife, julie, she's an amazing artist <dobey> rickspencer3: i have yeah <Saviq> :) <ogra_> rickspencer3, she has an afro haricut ? <rsalveti> haha, indeed <ogra_> oh ... "by his" <ogra_> :) <rickspencer3> yeah, the "by" is an important word there ;) <pmcgowan> next sessions are starting fwiw * didrocks move the camera to make more ad :p <dobey> the session ended 5 minutes ago guys :) <ogra_> haha, you need to put prices on <ogra_> should we schedule a followup session ? <sergiusens> bfiller, asac for automerge; the auto tests were ran <ogasawara> cyphermox: sorry for the delay, I'm proding these guys to wrap up <cyphermox> ogasawara: I m following that session no worries <cyphermox> mine will be short and sweet I think <rickspencer3> maybe do it on the phone mailing list or something? <asac> sergiusens: i know :) <asac> rickspencer3: sure. <sergiusens> just reconfirming <asac> thanks guys <didrocks> thanks <asac> was a good session ... and i was even able to hear rickspencer3 at some point :) <ycheng> hi, presentation URL please <ogasawara> https://plus.google.com/hangouts/_/hoaevent/AP36tYeAxqTjWsaQHLR0uYFiN-6eEqJ4Hl0KAOI1HFX3gsnQNtpEug?authuser=0&hl=en <ogasawara> cyphermox et al ^^ <xnox> cyphermox: down with bluez-gstreamer! can i go and drop that from ubuntu-desktop?! =) <asac> jfunk-canonical: bfiller: kgunn: can you guys give a bullet list of what features "trunk" comes by with beyond review bot and automerge? <cyphermox> xnox please don't <xnox> cyphermox: =( <awe__> cyphermox, do you know whether or not upstream is working on P2P support? * xnox is on like a minute delay on the stream =) <xnox> cyphermox: thanks for your reply ;-) * ogra_ cant watch this stream either :( <cyphermox> ogra_: why is that? <rsalveti> ogra_: just join instead <ogra_> no idea, the live streams dont work for me <ogra_> i canm watch any other video on yourtube and i can join the hangouts, but i cant watch the stream <asac> didrocks: you have the presentation URL? <didrocks> asac: sure, https://docs.google.com/a/canonical.com/presentation/d/1G4eWxQ8OnwPMXFJFefYdRTKuKZd_Da9Zh2W_X5XRJCc/edit#slide=id.p <asac> karni: ^^ <karni> thank you! <xnox> .. <xnox> who is making clicking noises ?! =))))) <stgraber> I think it's cyphermox, everyone else is mutedd (but me, but I'm not the source) <cyphermox> no, not me <ogra_> must be tony then <ogra_> no <stgraber> nope, everyone else but cyphermox are muted now :) <ogra_> definitely cyphermox <ogra_> he needs a can of oil ;) <xnox> stgraber: too late, photronix went live with articles saying we are updating bluez to 5 and pulseaudio to latest for 14.04 lts =)))) <xnox> </joking> <awe__> xnox! ;D <awe__> http://www.freedesktop.org/wiki/Software/PulseAudio/Notes/5.0/ <rsalveti> it'll be fun <ogra_> totally <rsalveti> do we need bluez5 for our first device? :-) <cyphermox> rsalveti: no we don't. it was only awe pestering me about bluez5 before, for HSP/HFP ;) <rsalveti> right <ogasawara> rsalveti: https://plus.google.com/hangouts/_/hoaevent/AP36tYfZwrD2PD85D95eFMkloDXhOD2DAOUeki4Iln_rrdj-gA8Fng?authuser=0&hl=en <ogasawara> rsalveti: assuming you're attending the Qt5 session next <rsalveti> yup <rsalveti> thanks * ChanServ changed the topic of #ubuntu-uds-core-1 to: Track: Core | Establish forward plan for Qt5 versioning | Url: http://summit.ubuntu.com/uds-1403/meeting/22158/core-1403-qt5-versioning/ * ScottK is here. <cjwatson> Cool. Do you want to / can you join the hangout? <rsalveti> Mirv: pmcgowan: https://plus.google.com/hangouts/_/hoaevent/AP36tYfZwrD2PD85D95eFMkloDXhOD2DAOUeki4Iln_rrdj-gA8Fng?authuser=0&hl=en <didrocks> asac: ^ * sergiusens notices that all the interesting sessions are today * sergiusens at least for him <sergiusens> ... at least for me <didrocks> sergiusens: â¦ at least for you <didrocks> :) <didrocks> kidding, same for me <ScottK> probably. <ScottK> Can someone invite me into the hangout? <asac> ScottK: https://plus.google.com/hangouts/_/hoaevent/AP36tYfZwrD2PD85D95eFMkloDXhOD2DAOUeki4Iln_rrdj-gA8Fng <asac> doesnt work? <asac> ogasawara: can you invite ScottK <asac> ? * lool listens to stream <ogasawara> ScottK: I can invite you. let me know the email you prefer I use. <sergiusens> Mirv, just use the devel list for updates <Mirv> sergiusens: yep, CC:d then later on there, but devel is the correct place indeed <ScottK> I'm on and off the hangout. <ogra_> now you are on <ogra_> :) <cjwatson> I'm watching IRC from time to time, at least <cjwatson> So happy to relay if need be <ScottK> Can anyone hear me when I talk? <ogra_> nope <ogra_> i cant see you either anymore <ogra_> (you are there but camera seems off) <ScottK> I turned my camera off to cut bandwidth. <xnox> <xnox> cjwatson: i'd expect each major qt 5.x release to have something major in it ;-) and it did harm us not moving to 5.1 first. We only started fixing 5.1 issues post 5.1 got released, thus those fixes landed in 5.2 only. <xnox> <xnox> ditto our 5.2 fixes are landing in 5.3 only. <xnox> <xnox> cjwatson: ideally, we'd have touch images with 5.3-trunk available under test. <ScottK> zero regression tolerance is a recipe for failure. <xnox> similarly how we test openstack-trunk on ubuntu-server all the time. <cjwatson> ScottK: I haven't heard any audio from you on the HO so far <xnox> (and on top of 12.04 LTS as well, since we backport openstack to last LTS) <ScottK> K. can we ask people to be quiet for a moment and I'll talk. <cjwatson> (moment) <ScottK> sure <cjwatson> word-in-edgeways problems :) <ScottK> also Qt5 going through the landing process effectively makes it a Canonical only process. <cjwatson> ScottK: you probably want to
{'channel': 'ubuntu-uds-core-1', 'month': 3}