#ubuntu-ci-eng 2014-06-01 <imgbot> === trainguard: IMAGE 58 building (started: 20140601 02:10) === <imgbot> === trainguard: IMAGE 58 DONE (finished: 20140601 03:35) === <imgbot> === changelog: http://people.canonical.com/~ogra/touch-image-stats/58.changes === <ogra_> Mirv, did any of these get acked yet ? <ogra_> (the split greeter packaging changes) <ogra_> (if not ... ACK ... ) <ogra_> popey, looks like unity8 didnt land until 2h ago https://launchpad.net/ubuntu/utopic/+source/unity8/7.87+14.10.20140530.1-0ubuntu1 <ogra_> it is funny that all the tests pass though ... the results for 57 and 58 dont look massively bad or anything <ogra_> i'll trigger an image build to pick up that package <ogra_> image triggered ... <imgbot> === trainguard: IMAGE 59 building (started: 20140601 12:10) === <popey> thanks ogra_ â» <ogra_> sent a followup email too ... <popey> made it home eventually then? <ogra_> 1am instead of 8pm ... and i had the worst train ride in my life (weird connetions on ultra slow trains going in wrong directions etc) but yeah ... <ogra_> many aussies got stuck in dubai for the night ... though i guess dubai isnt the worxt place to spend a night :) <cjwatson> unless you're the "wrong" skin colour :P <cjwatson> yeah, I feared they'd get stuck <cjwatson> had some train difficulties myself; got to King's Cross and it was all OH HAI NO TRAINS FOR THE NEXT TWO HOURS so I had to reroute <ogra_> well, after standing for over an hour in the bright sun with only few opportunities of shadow i guess the average skin colour was bright red <cjwatson> got home at 1:30am <ogra_> mine for sure is :) <cjwatson> heh, yeah, I was asymmetrically sunburnt <cjwatson> right arm noticeably redder than left <ogra_> (susie was pretty shocked seeing my face) <ogra_> especially my frehead looks funny now :) <ogra_> *forehead * ogra_ found the most funny note about all this is that canonical didnt make it "in the news" but "made the news" this time :) <ogra_> seems our social media was the base of all news about this (gustavos pic, your twitter post) ... <cjwatson> Not quite; I started by searching for "Malta airport" on twitter, and found Malta Independent and Malta Today breaking-news articles about it before I said anything there <ogra_> oh <cjwatson> Malta Today were clearly running a search since they replied to me immediately and then edited their article <cjwatson> But yeah they were obviously trying to find anyone who was there <cjwatson> There were a couple of other (non-C) people who tweeted just before me <cjwatson> It's amazing how much attention twitter gets for this kind of thing nowadays though <cjwatson> I didn't really have much first-hand information since there were no announcements and all we saw was planes with passengers clearly not being allowed off and fire engines going past <ogra_> yup ... especially since there seemed to be a lot "real" news people around doing interviews etc ... i havent seen much from them and google revelaed nothing either <cjwatson> I dodged the BBC asking for me to DM them my phone number for an interview :) <cjwatson> Mostly 'cause I don't much like being interviewed but it was really noisy anyway <ogra_> there was a lot guessing going on around me since thwe two passengers were announced pretty urgently while we stood in line for like 30mins beofre the evacuation started <cjwatson> Yeah, I was already airside so had no idea about that <cjwatson> Hadn't heard that it was two passengers - I only heard one name <ogra_> it became clear that this was somehow related when a guy with a megaphone ran around asking the same named people to contact the security/emergency personnel <cjwatson> Also: loudest alarm I have ever heard <ogra_> ++ <imgbot> === trainguard: IMAGE 59 DONE (finished: 20140601 13:35) === <imgbot> === changelog: http://people.canonical.com/~ogra/touch-image-stats/59.changes === * popey updates to #59 <ogra_> black screen for me on flo :( <ogra_> ugh <ogra_> phablet@ubuntu-phablet:~$ ps ax|grep unity8 <ogra_> 2230 ? Ssl 0:06 unity8 -testability <ogra_> why the heclk is it running with that option <popey> fine here on flo and mako <popey> 2172 ? Ssl 0:12 unity8 <ogra_> ok, then it is the condition of my flo <popey> my flo booted to black with #57 iirc and i --wiped to get back to normal <ogra_> iirc xnox played with it to fix the lsb breakage ... might be that i have weird test stuff installed * ogra_ wipes too <popey> suppose i better mow the lawn â» <ogra_> on a sunday ? <ogra_> (my neighbors would freak out :) ) <ogra_> bah ... the test results look awful ... seems unlocking doesnt work with many of the apps <ogra_> (Saviq actually feared that ... hopefully we can get that fixed tomorrow) <Mirv> ogra_: nope <Mirv> ah, but apparently did and robru has published it already <Mirv> on Friday already actually <Mirv> so I'll do Sunday's good deed and kick a rebuild of unity8 in 018 then <ahayzen> 3 <bzoltan> Mirv: are you around? <bzoltan> ogra_: Mirv: The Silo3 is fixing the gallery test failures ... I have pushed both the lp:gallery-app AP tests and the UITK AP tests to /home/phablet/autopilot and the tests look good <popey> hmm, the filemanager 193 published to the store refuses to download for me on nexus 4 <popey> 2014-06-01 18:53:56,897 - CRITICAL - ../../../../lib/SignOn/connection-manager.cpp 106 setupSocketConnection p2p error: QDBusError("org.freedesktop.DBus.Error.FileNotFound", "Failed to connect to socket /run/user/32011/signond/socket: No such file or directory") 1 <ogra_> bzoltan, i'll carry it into tomorrow mornings landing meeting ... <bzoltan> ogra_: thank you, I will be here if I am needed <ogra_> great ... <ogra_> there are quite some app test failures thanks to the greeter split ... we'll need to dig into them too <ogra_> (i think the unlocking behavior changed, seems the test suite needs some love= <ogra_> ) <popey> bug 1325402 <ubot5> bug 1325402 in ubuntu-system-settings (Ubuntu) "Can't download app updates on #59 on mako" [Undecided,New] https://launchpad.net/bugs/1325402 <popey> if anyone fancies trying to reproduce that, I'd appreciate it <ogra_> popey, works fine <popey> bah! <popey> why is it broken here â¹ <ogra_> do you have an U1 account set up already ? <popey> yes <ogra_> afaik thats now needed <ogra_> then look if you have a signond crash file ... <popey> no crashes today <ogra_> oh ... is the new filemanager unconfined now ? <ogra_> the last version didnt allow me to navigate above home <popey> hmm <popey> can't download anything now * popey removes and re-adds u1 account * ogra_ notes that apps get killed a lot heavier on his flo <ogra_> i cant really just flick through them if i have more than two running ... they are restarting all the time <popey> bah <popey> that fixed it <ogra_> sigh <ogra_> thats really unpleasant <popey> not the first time that's happened <ogra_> two news webapps and G+ open and whatever is in the bg gets killed * popey tests that <ogra_> the two that you can flick between are fine ... as soon as you switch to the tird that was in bg for a while it gets chaotic <ogra_> hmm <ogra_> now i managed to have G+ only showing a white screen <ogra_> and with that all three other open apps woork just fine <popey> I have G+, BBC News, Music and Calendar open <ogra_> seems G+ i the evil bit here ... eating my ram <popey> right, just tried to install filemanager on #50 and it did exactly the same thing! <popey> so maybe it's a 1st of the month thing? <ogra_> well, it installed just fine here <ogra_> looking with htop i see unity8 uses constantly 205M ... plus unity8-greeter eating another 95M <ogra_> each open webapp container eats between 90 and 100M <ogra_> thats insane ! <ogra_> and the load on flo is constantly above 5 <ogra_> well, when i interact with it #ubuntu-ci-eng 2014-06-02 <imgbot> === trainguard: IMAGE 60 DONE (finished: 20140602 03:35) === <imgbot> === changelog: http://people.canonical.com/~ogra/touch-image-stats/60.changes === <Mirv> cihelp jenkins full, can't even launch the clean jobs because also those fail from out of disk space.. :( <popey> bug 1325467 for anyone on #60 who has an alarm any time soon. <ubot5> bug 1325467 in unity8 (Ubuntu) "Alarms appear underneath greeter #60 mako" [Undecided,New] https://launchpad.net/bugs/1325467 <popey> if you can confirm pls <sil2100> Morning! <ogra_> hey hey <sil2100> Now what the heck happened with test results of the recent images! * sil2100 looks at what has changed <ogra_> seems there are timeouts in the tests with split greeter <ogra_> sil2100, the landing wasnt complete, the unity8 bit only landed yesterday <sil2100> hmm <sil2100> ogra_: the split greeter one? How was it not complete? Not everything was present in the silo? <ogra_> not sure why, but unity8 only made it into the archiove yesterday <sil2100> Ah, hm <sil2100> Interesting, as I thought the faux-package bumping was no longer required - maybe some autopkgtest failures? <ogra_> https://launchpad.net/ubuntu/utopic/+source/unity8/7.87+14.10.20140530.1-0ubuntu1 <sil2100> Uh, something broke the CITrain backend I just noticed\ <ogra_> not sure what happened, but yeah, could be autopkgtests taking long <sil2100> Holy sh... ci-train jenkins seems to be 'broken' * ogra_ isnt sure if the "published" date in the proposed pocket entry is actually the date when it entered proposed <sil2100> https://ci-train.ubuntu.com/ <sil2100> cihelp: hi guys! Does anyone know why all the ci-train executors are dead? <sil2100> cihelp: as per https://ci-train.ubuntu.com/ <- <ogra_> anyway ... there are now timeouts in the tests ... Saviq was fearing we could have issues with unlocking with the new greeter ... not sure it is that <ogra_> (i didnt inspect to deeply yesterday, just made sure we get an image with the landed unity8) <Saviq> ogra_, if we affected it, it would be *all* of them, is that the case? <sil2100> Saviq: there are like many many test failing now, around 120 failures <ogra_> Saviq, nope ... but only -app tests <Saviq> ogra_, that shouldn't be the split greeter then, means we're indeed unlocking the screen if we can pass any of them <ogra_> Saviq, thats the changeset after which it started ... seems the ailo only half migrated on friday ... http://people.canonical.com/~ogra/touch-image-stats/59.changes <popey> sil2100: bug 1325467 is interesting <ubot5> bug 1325467 in Unity 8 "Alarms appear underneath greeter #60 mako" [High,Triaged] https://launchpad.net/bugs/1325467 <sil2100> I don't see anything else that could have caused the failures <popey> in that snap decisions appear underneath the greeter <sil2100> In the changes <Saviq> popey, commented on it already <sil2100> popey: oh! <ogra_> sil2100, well not sure if dbus-x11 is a dep of the greeter split <popey> morning Saviq â» <Saviq> ogra_, it is <Saviq> popey, that's indeed fallout of split greeter <ogra_> ok <popey> nice one <ogra_> Saviq, and the xlibs ? * sil2100 is too tired today to feel any pressure regarding the test results <Saviq> ogra_, I believe so, it did bring some weird things with it <ogra_> hmm, probably deps of the dbus-x11 package * Saviq looks <Saviq> ogra_, my apt history.log entries from testing the silo: http://pastebin.ubuntu.com/7571253/ <Saviq> ogra_, so yeah, the x things are pulled in by split greeter <ogra_> yeah, looking at the dependency chain of dbus-x11 shows similar results <sil2100> ogra_: yeah, and in the ubuntu-touch-session changelog I only see the greeter added, so no new explicit deps have been filed <Saviq> ogra_, direct dep for dbus-x11, too http://bazaar.launchpad.net/~unity-team/unity8/trunk/revision/933/debian/control#debian/control <sil2100> I was chatting a lot with Mike on Friday, but I can't remember if he'll be around today or not... <sil2100> Since he might be traveling <Mirv> sil2100: I reported the jenkins problem to cihelp in the morning <Mirv> sil2100: it's out of disk space, and also trying to free up disk space with that one job fails with not enough disk space.. <ogra_> i havent seen him at the bomb evacuation ... so he might not have hit one of the delays <ogra_> (but he might just have been at another part of the airport) <ogra_> so we have chances he traveled fine and is around later today <ogra_> sil2100, my suspicion is more towards the infrastructure though <ogra_> that we are either lacking something for these failed runs or that something is different from how mterry and Saviq tested * ogra_ bets on the latter <popey> t1mp / bzoltan do you have an ETA on when the fix for 1322527 will land in the image? <popey> (morning btw) <ogra_> popey, it is ready ... <ogra_> bzoltan, pinged yesterday (assuming you talk about gallery) <popey> Ready != on my phone â» <Saviq> ogra_, now that I look through the diff, I'm not sure why dbus-x11 is a direct dep, we'll have to wait for mterry to comment :| <ogra_> wheer is the bot ? <Saviq> but then dbus-test-runner is used in a test, and isn't a build-dep... <ogra_> Saviq, well, did you run all AP tests or just a selection ? <popey> ubot5 is here. <ogra_> then it doesnt like you :) <ogra_> bug #123456 <ubot5> bug 123456 in xine-lib (Ubuntu) "podcast crashes amarok" [Undecided,Fix released] https://launchpad.net/bugs/123456 <sil2100> Mirv: morning! Now this is unexpected! Didn't he had the 'free disk space' job set up to be ran periodically? <popey> common <ogra_> hmm <sil2100> Mirv: I suspect it was in plans and never happened, right? <ogra_> bug 1322527 <ubot5> bug 1322527 in Ubuntu Terminal App "Terminal app shows no text in #44 on mako" [Critical,Triaged] https://launchpad.net/bugs/1322527 <Mirv> sil2100: I don't know, the apt-get clean job was at least disabled. that's probable. <ogra_> Saviq, if you ran them all it must be something that is different in the infrastructure ... <sil2100> Let's wait for CI to appear and resolve the problem <Saviq> ogra_, indeed, we didn't run the whole suite with the latest builds, I got pressed for pushing the button... <sil2100> Mirv: for now I disabled auto-refresh of the spreadsheet as it was failing because of the backend being broken <Saviq> ogra_, in any case, http://q-jenkins.ubuntu-ci:8080/job/autopilot-release-gatekeeper/140/ was a run some time before, but nothing much changed that would warrant an all-over-the-place fail <sil2100> (mostly because of jenkins dying in the middle of some configuration save operation) <sil2100> Saviq: do you know of any differences in how tests are ran at the gatekeeper and in smoketesting? <Saviq> sil2100, in theory there shouldn't be <Saviq> sil2100, they all use phablet-test-run by now <sil2100> Since psivaa is not around today, we also basically have no CI to support us <Saviq> sil2100, did you have a chance to try and reproduce the issue locally? <sil2100> Saviq: not yet, but will try in a moment once my image flashes <Saviq> sil2100, doing the same <ogra_> Saviq, it seems to hit largely the same tests/app that fail in both image that have the full landing (59 and 60) <ogra_> sil2100, ^^^ <ogra_> the only one i dont see in the first image is clock app ... all the others are roughly the same <Saviq> hmm "via upstart-app-launch", are we monitoring tedg's rename? it didn't happen yet did it? <Saviq> looks like launch_click_package got screwed somehow <Saviq> the common denominator <Saviq> dialer fails on phonesim, so another issue <Saviq> gallery has a sh*tload of sqlite errors and then an SDK exception ?? <bzoltan> popey: t1mp: ogra_: the fix is in the silo3 <sil2100> Saviq: yeah, so the name change didn't land yet if anything <sil2100> hm, grrr, device problems <Saviq> sil2100, ogra_, btw python2 is gone from the device, but phablet-test-run seems to be using it still <popey> oh jeez <ogra_> Saviq, for some deb tests thats ok ... it should pull it in <ogra_> (if needed) <Saviq> ogra_, well, sure, unity8-autopilot pulls it in... until it won't :P <ogra_> Saviq, gallery is something else ... thats older, ignore it <Saviq> aargh this whole click test setup is just crazy complex and delicate :| <sil2100> Saviq: you mean, the downloading, looking up versions etc? <Saviq> sil2100, yeah <ogra_> Saviq, and thats fully in sergios hands ... who is on vac iirc <ogra_> fun ... <Saviq> huhm <ogra_> and i think rolling back is also no option ... <Saviq> my phablet-test-run is being prevented by apparmor ?! <ogra_> oh ? <popey> balloons mentioned that happened to him too last week iirc <Saviq> although that would have ended up in the smoke logs if that was the case <ogra_> yeah <popey> Saviq: have you run "phablet-config autopilot --dbus-probe enable" ? <Saviq> popey, ah <sil2100> :) <Saviq> this shit is CRAZY <Saviq> may I remind https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1262879 <ubot5> Ubuntu bug 1262879 in Ubuntu CI Services "There should only be one, documented, way to run tests on devices" [Undecided,Confirmed] <popey> heh <Saviq> and yeah, phablet-config just hangs... <popey> https://wiki.ubuntu.com/Touch/Testing is the "canonical" page * Saviq reboots <popey> it takes a while <popey> patience <Saviq> ah probably rebuilds apparmor policies? <popey> yes <popey> so worse if you have lots of apps installed <sil2100> Saviq: yeah, it can take even a minute sometimes <ogra_> well, we usually only test preinstalled apps <popey> takes more than a minute here <ogra_> for these it should take 20-30sec only <popey> alan@deep-thought:~$ time phablet-config autopilot --dbus-probe enable <popey> real 1m36.286s <ogra_> woah <popey> but takes less time on subsequent runs <popey> alan@deep-thought:~$ time phablet-config autopilot --dbus-probe enable <popey> real 0m4.027s <ogra_> yeah <ogra_> it is the initial profile creation that takes ages <ogra_> security has plans to ship some pre-created ones in the future * ogra_ goes to make meeting coffee <ogra_> well seeing ToyKeeper's mail it seems the issues are really AP centric ... doesnt look like she found many issues in dogfooding <ogra_> (a unity crash though) <Saviq> I wonder if the unlock script is used at all in the end... and if it works... <Saviq> maybe it's unreliable <sil2100> Saviq: you mean like during the app tests? <sil2100> Since there it should unlock the screen once I guess, if it would fail then all tests should fail <Saviq> sil2100, I know :| <ev> Mirv: where are you seeing ENOSPC? https://prodstack-nagios.admin.canonical.com/cgi-bin/nagios3/extinfo.cgi?type=1&host=ue-prodstack-ci-train-jenkins-0 suggests we have plenty, so maybe this is in a tmpfs? <Saviq> so yeah... of course it's fine locally :| <ogra_> well, all tests do fail for the affected apps <ogra_> the 4 that you see passed there are systemsettle, setup and teardown <ogra_> they always work <ogra_> its not "all apps" but if it fails its "all tests" of an app <ogra_> sil2100, teds renaming didnt land yet, did it ? <sil2100> ogra_: no <ogra_> 04:27:02.746 INFO _launcher:116 - Attempting to launch application 'com.ubuntu.calendar_calendar_0.4.296' with URIs '' via upstart-app-launch <ogra_> ok <sil2100> ogra_: after Mike's landing it required a rebuild <sil2100> ogra_: so no one published it afterwards yet - it's not marked as tested even <ogra_> because i see this for all failing tests <ogra_> RuntimeError: Timed out while waiting for application to launch <sil2100> Yeah, I was suspecting this to be the problem as well, but it doesn't seem so <ogra_> ... <ogra_> right, but the apps dont seem to start properly <ogra_> Saviq, another thing i noticed is that your ram usage is quite high with the new greeter <ogra_> its stable ... but high <ogra_> 200M for unity8 ... another 90 or so for the greeter <sil2100> How did it look like before? <Saviq> calendar is OK locally ;| <ogra_> i think i remember far below 200 for unity8 ... <ogra_> 170 or 180 or so <Saviq> ogra_, if anything, it should go down in unity8 <ogra_> and we didnt have a greeter <Saviq> ogra_, can you file a bug please? we'll investigage <ogra_> yeah <Saviq> investigage... <Saviq> nice word <ogra_> :) <ogra_> Ursinha, go away ! you are on vacation !!! <Ursinha> lol <ogra_> :) <Ursinha> I'm not here :) * ogra_ pretends he hasnt seen anything <ogra_> *whistle* <Ursinha> :P <sil2100> ;p <Mirv> ev: https://ci-train.ubuntu.com/computer/%28master%29/executors/5/causeOfDeath <Mirv> ev: and first it was in this job output before the jobs also started failing to even start: https://ci-train.ubuntu.com/job/landing-010-2-publish/8/console <ogra_> clientlogs touch (run.py:2245): WARNING **: Unable to connect to Upstart bus: Cannot autolaunch D-Bus without X11 $DISPLAY <ogra_> hmm <ogra_> 04:27:01.320 DEBUG __init__:187 - Patched home to fake home directory /home/phablet/autopilot/fakeenv/tmpyt7h1kxo <ogra_> error: XDG_RUNTIME_DIR not set in the environment. <ogra_> oha ! <sil2100> Uhh? <sil2100> WTF <ogra_> there seem to be more env issues <Saviq> dialer = OK <ogra_> initctl: Rejected send message, 1 matched rules; type="method_call", sender=":1.22" (uid=32011 pid=2242 comm="/sbin/initctl set-env HOME=/home/phablet/autopilot") interface="com.ubuntu.Upstart0_6" member="SetEnv" error name="(unset)" requested_reply="0" <ogra_> rejected ... <Mirv> ogra_: meeting <ogra_> yeah <Saviq> ogra_, sil2100, not sure what to say... I followed the standard testing process and everything's fine over here :| <ogra_> Saviq, yeah, as i suspected ... most likely the infra <ogra_> 04:27:01.079 WARNING emulators:26 - The ubuntuuitoolkit.emulators module is deprecated. Import the autopilot helpers from the top-level ubuntuuitoolkit module. <Saviq> yeah, that's just a deprecation warning, though, should not affect anything <ogra_> yeah, i was just mentioning it in the meeting :) <ogra_> (abusin IRC for copy/paste) <t1mp> popey: I'm checking the bug and our landing.. * ogra_ wonders if dbus gets started differently with the new greeter <ogra_> not using the upstart job ... <t1mp> bzoltan: what's the status of this landign? https://code.launchpad.net/~bzoltan/ubuntu-ui-toolkit/landing_28-05/+merge/221199 <t1mp> popey: ^ the fix for terminal-app is included there^. We are currently trying to land it <popey> kk <popey> in silo3 bzoltan said earlier? <t1mp> bzoltan: ^ the changelog of that landing is not complete, since we merged staging more stuff is in there <t1mp> popey: yes, that's the one. trying to land it since last week. <t1mp> popey: feel free to try a phablet-config writable-image --ppa ppa:ci-train-ppa-service/landing-003 <brendand> lots of ap test failures today <popey> t1mp: will do! <ogra_> sil2100, oh, are we in traincon until we get proper tests again ? <ogra_> i think we shouldnt land anything til thats resolved <sil2100> ogra_: for now we can't land anything anyway ;) <sil2100> :D <sil2100> (jenkins b0rken) <ogra_> (well, probably the gallery fix, but i wouldnt go further) <ogra_> yep, i know <ogra_> but it might come back during the day <bzoltan> t1mp: what do you miss there? <ogra_> brendand, yeah, some issue with the environment in the infrastructure ... and everyone who could help is either traveling, on vacation or otherwise not around <sil2100> I wonder if Paul will be around today, or he also got stuck somewhere in transit <sil2100> ev: do you know if Paul will be able to pop up today? <t1mp> bzoltan: r1087, 1088 of staging <ev> sil2100: I'm not sure which day he is taking as swap. What do you need him for? <ogra_> ev, 150 failing AP tests :) <ogra_> seems the environment is messed up and we need someone to help investigating ... tests pass fine locally <ogra_> ah, well, i'm lying ... only 120 <t1mp> bzoltan: I think only those two are missing <sil2100> ev: right, we would need someone with smoketesting infra experience/access to help us figure out what's wrong with the environment in smoketesting <sil2100> ev: which is causing many app failures (which are false positives) <ev> the whole team is meant to have access and understanding. We're constantly fighting against the silo effect. Unfortunately though we are very short staffed today. <ogra_> yeah, like almost everyone <ev> Siva is on swap. Vincent is out sick. I suspect a fair amount of the US contingent will be taking swap, but it's hard to tell without the new HR system :) <ogra_> 04:27:01.287 INFO logging:45 - str: Set the value of an initctl environment variable. Arguments ('/home/phablet/autopilot/fakeenv/tmpyt7h1kxo',). Keyword arguments: {}. <ogra_> initctl: Rejected send message, 1 matched rules; type="method_call", sender=":1.22" (uid=32011 pid=2242 comm="/sbin/initctl set-env HOME=/home/phablet/autopilot") interface="com.ubuntu.Upstart0_6" member="SetEnv" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init") <ogra_> error: XDG_RUNTIME_DIR not set in the environment. <ogra_> clientlogs touch (run.py:2236): WARNING **: Unable to connect to Upstart bus: Cannot autolaunch D-Bus without X11 $DISPLAY <ogra_> thats a summary of th re-occuring errors we see ... usually its these three lines <ogra_> it started with the split greeter ... we assume the infra needs something updated for the new setup <bzoltan> t1mp: I will fix it <ogra_> (not sure what though ... and which versions of which tools are installed there) <bzoltan> ogra_: sil2100: Mirv: is there any news about the UITK landing from the Silo3? <ogra_> bzoltan, well, jenkins is dead ... nothing can land <bzoltan> ogra_: what an excuse :) <ogra_> bzoltan, and we have issues with the AP tests for the last two images <sil2100> bzoltan: as per description on the spreadsheet, waiting on it to be resolved :) <ogra_> so even if jenkins was working we would need to stop the line to get the infra back in shape <sil2100> No worries, we'll land as soon as we can! We want to have the infra working and some test doubts solved first <bzoltan> sil2100: ogra_: ohh.. I missed the yellow cell ... and added a new landing line. Should I delete it? <ogra_> no, but you wont get a silo atm <ogra_> adding entries is fine, they just wont be processed <ogra_> oh, wait <ogra_> well, probably better remove it so you get auto-update if you add it the next time <ogra_> sil2100, does the "please dont fill new landings" apply to the pending page too ? <ogra_> (/me thought we just cant process silos) <sil2100> bzoltan: no no, leave it :) We'll just assign it once possible <bzoltan> ogra_: how long this stop will last? Do I have time to rebuild the landing MRs in the silo? <ogra_> sil2100, the text should be more clear ... it sounds like you shouldnt even add to the pending page <sil2100> ogra_: you can fill in new landings, let me change the description as it's well... from the moment I didn't know what happened <ogra_> yeah <Mirv> sil2100: ok we should be more alive now..I'm restaring a few threads <popey> t1mp: that fixes terminal for me, thanks. <ogra_> bzoltan, up to sil2100 what and if he wants to land stuff even after jenkins is back up <ogra_> bzoltan, we cant really get test results atm ... thats the second stopper imho, though sil2100 might still want to move ... he's the authority here :) <sil2100> Mirv: so, what was the problem in the end? <bzoltan> ogra_: sil2100: a full rebuild would need an hour <Mirv> sil2100: old backups cleaning not working correctly <ogra_> bzoltan, well, i think you have time ... it will take a while til we can get someone from the infra team who knows the AP testing infra enough ... <sil2100> bzoltan: you can rebuild, it'll give me some time to try and recover from our earlier problems <sil2100> (just remember that the SS does not auto-update itself for now) <ogra_> hmm, i see a dbus-launch process on my flo ... we never used dbus-launch before <ogra_> seems we are running two dbus-daemon processes for the phablet user <ogra_> one from the dbus upstart job, the other started via ssh-agent by the greeter <ogra_> phablet 1350 0.0 0.0 2700 188 ? Ss Jun01 0:00 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session ubuntu-touch-session <ogra_> phablet 1354 0.0 0.0 3360 604 ? Ss Jun01 0:00 //bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session <ogra_> phablet 1450 0.0 0.0 4308 1540 ? Ss Jun01 0:08 dbus-daemon --fork --session --address=unix:abstract=/tmp/dbus-wwyt1Zs6rI <ogra_> the first two are spawned by the greeter ... the last one is our normal dbus ... <t1mp> popey: great, thanks for testing :) I hope it will land today <ogra_> heh, the boot animation is quite tiny on the flo screen * ogra_ just enabled it * sil2100 sighs * sil2100 has to hack through jenkins to get the backend back to some usable state <Mirv> sil2100: so the file that got corrupted is causing problems? <sil2100> Mirv: yeah, I need to use the temporary jenkins job to at least recover some parts of the config so that I can reassign it once again :) Doing that <Mirv> ogra_: could you say 'ack' on another "drop using the transitional package, instead use the real one that has been there since trusty": https://ci-train.ubuntu.com/job/landing-010-2-publish/lastSuccessfulBuild/artifact/packaging_changes_signon-ui_0.16+14.10.20140530-0ubuntu1.diff .. <ogra_> Mirv, ackedy ack :) <Mirv> ack! :) <ogra_> Mirv, we still need to go over the seed one day ... i meant to do that at the sprint with you before i realized you wouldnt be there the same week <Mirv> ogra_: indeed. so many renames. <ogra_> yeah <Mirv> ogra_: I was going to be there on the same week, but I got moved.. <ogra_> which probably made sense ... <Saviq> ev, hey, do you need guinea pigs for the airline maybe? * sil2100 is grateful for the "cyphermox test" jenkins job <sil2100> :) <Saviq> sil2100, are we any closer to finding the issue in smoke? <ev> Saviq: absolutely. Shall I get you on the list of people we need to engage with once we have the kinks knocked out of staging? <Saviq> ev, please do! <ogra_> Saviq, i see two sessuon dbus daemons ... <Saviq> ogra_, oh <Saviq> ogra_, well, wait <Saviq> ogra_, the greeter has one, the session has one <sil2100> Saviq: a little bit... but I guess it would be best if we had someone wit haccess here <Saviq> ogra_, or are you seeing two on the usser? <ogra_> Saviq, http://paste.ubuntu.com/7572007/ <ogra_> there is also one running as lightdm ... that should be the greeter owned one <ogra_> the ssh-agent one is new <ogra_> i'm just re-flashing 60 to get a clean env ... <ogra_> (another 15min to download over my slow line) <Saviq> ogra_, I don't have ssh-agent here <brendand> does there happen to be a way, currently, that we can duplicate a test run from a jenkins job? <ogra_> Saviq, on 59/60 ? <Saviq> ogra_, 60, yes <ogra_> hmm <ogra_> that was 59 above <Saviq> oh wait <Saviq> FUX <ogra_> ps aux|grep dbus|grep session|grep phablet <Saviq> greyback, remember what I did in the room on Friday afternoon that we said it's gonna become interesting the next time I try to use it? <ogra_> i think thats from lightdm ... i cant find any change in the landing that would make it use dbus-launch <Saviq> ogra_, ignore everything I said since this morning... I had 55 flashed all the time :| <Saviq> don't ask <ogra_> oh my <greyback> Saviq: yep! <Saviq> greyback, guess what :P <greyback> Saviq: noooo <sil2100> OOoh * sil2100 runs some tests on his 59 then <ogra_> i think we could easily go with dbus-launch but would need to drop the upstart job in the session *and* make sure that the dbus socket address ends up in the file in ~/.cache/upstart/dbus-session ... thats what the tests rely on <mhr3_> sil2100, can you check on 009? prod it so it gets into proposed? <ogra_> seb128, do you remember of the top of your head where dbus-launch is called from in a lightdm session startup ? ... we end up with http://paste.ubuntu.com/7572049/ after the new greeter landed on the phone ... and somehow the AP tests get confused not finding the right dbus <sil2100> mhr3_: hi, the hud one? Let me try :) <ogra_> i fail to find where dbus-launch is called <seb128> ogra_, I would say Xsession.d scripts <ogra_> oh, i dont even have the dir on the old image ! <seb128> ? <seb128> do you have it on the new one? <ogra_> ah, no ... i'm to sleepy still ... its there <ogra_> root@ubuntu-phablet:~# grep dbus /etc/X11/Xsession.d/* <ogra_> root@ubuntu-phablet:~# <seb128> dbus-x11 ships /etc/X11/Xsession.d/75dbus_dbus-launch <ogra_> but no dbus stuff <seb128> but I guess that binary is not installed on touch <seb128> not sure how it's started there then <ogra_> seb128, right, that landed together with the new greeter stuff <ogra_> http://people.canonical.com/~ogra/touch-image-stats/59.changes <seb128> ok <seb128> so that explains why you get a new one <ogra_> in the old world we use an upstart job <ogra_> now there are both <seb128> right <ogra_> though i wonder why the apps work fine ... they should get confused too * ogra_ waits for the flash to finish to try some things <seb128> well, do the apps need to use the session bus? <ogra_> AP tests do <ogra_> and they get the address from a fil that is pointing to the secondly started dbus <ogra_> *file <ogra_> i guess the session ises theone dbus-launch started <ogra_> *uses <ogra_> and the secondly started one is just cruft <ogra_> seb128, thanks ! that got me on the right track i think <seb128> yw! <seb128> good luck debugging it <ogra_> heh <ogra_> well, i'm fine with dropping the upstart job ... but we need to create that file somehow so AP can read it to find the bus <sil2100> But hm... <sil2100> I'm running clock-app tests here now on my device and so far they seem to look ok (we'll see once they're finished) <sil2100> Shouldn't it also make them fail on my device? <sil2100> Oh, wait <sil2100> hm, I got 5 failures, it's still less than what I should expect * ogra_ disables the dbus upstart job <ogra_> lets see if the session still comes back <ogra_> hmm <ogra_> doesnt <ogra_> so lets try the other way round <ogra_> disabling the Xsession.d script <ogra_> yeah <ogra_> thats fine <ogra_> fun, i still get two dbus-daemon processes ... but they use the same address <ogra_> sil2100, mount -o remount,rw / && mv /etc/X11/Xsession.d/75dbus_dbus-launch /home/phablet/ && mount -o remount,ro / && reboot <ogra_> sil2100, then run your tests again <sil2100> hah, ok, running <ogra_> not sure what it uses dbus-x11 for ... i think we need mike to tell if it is important <Mirv> sil2100: I'll probably mark line 9 as Landed manually, since it didn't pick that up <sil2100> Ok :) <sil2100> Mirv: thanks! <sil2100> bzoltan: did you see the error in silo 003? <Mirv> also, that means that the first Qt SRU to trusty finally is available to users <- bzoltan <Saviq> sil2100, ogra_, in any case, tests work locally on 60, too <Saviq> and whoa new calendar! <popey> â» <popey> Sans vomit. â» <ogra_> Saviq, right ... but in the infra i think the tests try to connect to the wrong session dbus <Saviq> lol <ogra_> Saviq, i'm pretty sure /etc/X11/Xsession.d/75dbus_dbus-launch confuses them <Mirv> mhr3_: do you have any better sightings than we do about the hud trusty SRU? it's nowhere to be found. it's not rejected either according to the queues. <Saviq> mhm :| <ogra_> i'm just not sure how to solve that without dropping the dbus-x11 dep <Saviq> I did ask mterry whether it's correct that it pulls all that <Saviq> and he said yes, didn't dig more, though <ogra_> my session works just fine removing that file <sil2100> Here it works fine as well, but tests didn't finish yet <ogra_> i also see a session dbus running for lightdm <ogra_> (which used dbus-launch before) <ogra_> phablet@ubuntu-phablet:~$ ps aux|grep dbus|grep -v upstart|grep session <ogra_> lightdm 1193 0.0 0.0 3892 1260 ? Ss 10:29 0:00 //bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session <ogra_> phablet 1395 0.1 0.0 4320 1612 ? Ss 10:29 0:01 dbus-daemon --fork --session --address=unix:abstract=/tmp/dbus-vrhmrTD7DQ <mhr3_> Mirv, how would i know? <Mirv> mhr3_: just in case it has been discussed before by eg. sil2100. I think I saw already last week that the line claimed "in no known space or time" for the hud 14.04+14.04.20140528, and it's surely isn't on its way to trusty at the moment <ogra_> hmm <ogra_> interesting <mhr3_> Mirv, i rebuilt it since, but no idea what happened to it <ogra_> if i put the file back in place i see that the greeter session doesnt use dbus-launch at all <bzoltan> sil2100: I have seen and I still see... But I do not get it <cjwatson> Mirv: I rejected hud from trusty unapproved last week because its changelog delta made no sense and was no good for SRU tracking <cjwatson> I talked to Pete Woods about it <Mirv> cjwatson: ok, I thought this "round 2" was already after that, but maybe not <Mirv> this was published on Thursday <cjwatson> Mirv: hm, I don't remember dates, but I don't see it in unapproved, at least <Mirv> cjwatson: does that changelog sound to you as not making sense or better: https://launchpad.net/~ci-train-ppa-service/+archive/landing-009/+sourcepub/4202156/+listing-archive-extra ? <ogra_> Saviq, sil2100 ... so with dbus-x11 in place it not only starts one by dbus-launch and one via upstart ... it also starts the upstart driven daemons twice ... http://paste.ubuntu.com/7572218/ <cjwatson> Mirv: looks more plausible though I haven't diffed it against the old one <cjwatson> Mirv: LP logs suggest that this package was copied from silo 9 to an "sru-staging" PPA <Saviq> ogra_, ok, so obviously that's not good, we should get rid of the dbus-x11 dep <ogra_> Saviq, right, but i dont know what else this could break <cjwatson> Mirv: https://launchpad.net/~ubuntu-unity/+archive/sru-staging apparently <Mirv> cjwatson: yes, the SRU:s have went through that and then it'd be copied from there to archives <ogra_> Saviq, i see no ill effects when using teh system but i suspect some new greeter features i dont know about might be broken then <cjwatson> Mirv: no sign of any copies after that <Mirv> I wonder if that step never really happened <Mirv> cjwatson: that's what I thought too <ogra_> there must be a reason why dbus-x11 was added as a dep <Mirv> maybe didrocks could again find it this time... <Mirv> (moved to #ubuntu-touch) <sil2100> ogra_: we need mterry to explain himself then... <ogra_> sil2100, well, it would most likely only break new greeter features ... not existing ones <ogra_> sil2100, so if he is off and doesnt show up today we should remove the dep and he can add it himself later again <ogra_> that would prevent us from rolling back the whole ... in case he is off the whole week or some such <ogra_> this is to big to easily roll it back ... but we also cant live without working AP tests for multiple days <ogra_> (or even a week) <sil2100> I wouldn't want to roll it back <ogra_> sigh sigh sigh ... and *none* of his changelogs mention *any* of the dependency changes <sil2100> Especially that it doesn't cause visible regressions on the phone itself <sil2100> geh <sil2100> I'm really considering automating this dependency changes <ogra_> note that i wont ack any packages anymore where there are dep changes that are not mentioned ... <sil2100> I mean, including a forceful changelog entry about deps being changed <ogra_> this really makes life to hard <sil2100> Right * sil2100 adds this to his todo list for today <ogra_> sil2100, right, the ci should preferably do that ... but until thats possible people really need to care for tehir shit <sil2100> ogra_: I wonder if we would have any means (if we got the right people) to try and you know, forcefully remove the dbus-x11 package from one of the machines and running some failing tests again <ogra_> sil2100, first of all we need to find people where it breaks for them at home :p <ogra_> seems Saviq cant reproduce the failures at home <sil2100> Yeah ;) I can't as well <sil2100> The 5 failures I got seem to be unrelated <ogra_> right <ogra_> and additionally i might even be on the wrong track ... while it is wrong to have multiple session dbuses running that might not cause the infra issue at all (it is very likely but we can only prove it when fiddling with the infra) <ogra_> i would like to have at least some proof ... * ogra_ hopes plars is not off today ... that would help a lot <ogra_> as he can log in to a device, manually remmove the file and re-run <sil2100> Right <sil2100> We would need at least one person with direct access to some of the machines <ogra_> yep <ogra_> hmm, so looking at the greeter code mike makes a lot use of gdbus <dbarth> o/ line 39 for a new silo request <ogra_> aha ... <ogra_> /usr/bin/unity8-greeter-wrapper ... line 55 to 58 ... there he needs to use dbus to set some ofnono/pulse stuff ... and for that he uses the session bus <ogra_> (which is the wrong session bus anyway i guess) <ogra_> ah, no, that talks to the lightdm session bus <ogra_> sigh, that landing is so gigantic * ogra_ tries to read the unity8 source changes but thats a *lot* <sil2100> Yeah... I wonder if it wouldn't just be better to wait for mterry <sil2100> But hm, we have no means of checking if he'll be around today <ogra_> well <sil2100> Robert or Kevin would know <ogra_> sil2100, so here is another quick and dirty idea <ogra_> since the silos and landing are stuck anyway ... how about i do a dput upload that only drops the dep ... we build an image and see if it fixes it <ogra_> that costs 3-4h and we'll know for sure after that <ogra_> and is easy to roll back <popey> sil2100: have you droped mterry a mail? He replied to one of my bugs 4 hours ago. <ogra_> would need Saviq to merge my change back and forth i guess ... <sil2100> popey: oh, let me try that, he might be in transit now though <popey> calendar shows him being a patch pilot today. <sil2100> ogra_: hmmm... right, but this seems REALLY dirty <sil2100> Not sure if it's worth it? <ogra_> well, it wastes the least time <Saviq> mterry should be around soon * ogra_ was just thinking we could do something while no landings can happen anyway <ogra_> ah, cool <Saviq> like within an hour or two <sil2100> ogra_: let's wait a little bit then <ogra_> then ignore my hackish idea :) <sil2100> Ok, I have to jump now for that lunch <Saviq> let me see what were his travel plans <Saviq> oh actually <dbarth> sil2100: hey, not sure if you saw my request above <Saviq> he only *just* flew out of Malta <sil2100> dbarth: let me get back to you after lunch <Saviq> mterry is here: http://flightaware.com/live/flight/DLH1277/history/20140602/1100Z/LMML/EDDF <Saviq> :| <sil2100> Might take a bit <Saviq> ogra_, back to your idea <sil2100> Saviq: *sigh* <ogra_> heh <sil2100> Ok, ogra_ I leave the decision up to you if you think it's worth it, I would try getting some info on if Paul will be here today <ogra_> i could quickly drive to frankfurt and ask him in person :) <Saviq> he _will_ have 3h layover in FRA <sil2100> Since if Paul would be around, we could do this without any hacks even <dbarth> sil2100: ok * sil2100 goes now REALLY <Saviq> I knew landing this on a Friday evening was not a great idea! <ogra_> even if he has a 3h layover expecting him to do any work during travelling is evil <Saviq> of course <Saviq> I just meant that he might pop in, not that we should rely on that <ogra_> Saviq, yeah, thats why i wanted to land on mon/tue ... execting fallout <ogra_> (i think i told you in the bar already) <Saviq> ogra_, when sil2100 and robru rounded me up at Friday afternoon, it didn't look like they wanted to wait :D <ogra_> heh, yeah, my pusyness infected everyone ... sorry for that <ogra_> i should have stopped asking for it on thursday <Saviq> ogra_, in any case, I can't identify the need for dbus-x11 in that merge <ogra_> *pushyness ... not pussyness <ogra_> :P <Saviq> ;) <Saviq> ogra_, so let's just go for it - upload a dbus-x11-- version, I'll merge around <ogra_> ok <dpm> sil2100, Mirv, bzoltan tells me that the UITK fix for https://bugs.launchpad.net/ubuntu/+source/ubuntu-ui-toolkit/+bug/1322630 is already in the archives. Am I correct to assume it'll land in the next image (61)? <ubot5> Ubuntu bug 1322630 in ubuntu-ui-toolkit (Ubuntu) "Setting i18n.domain breaks translation" [Undecided,Confirmed] <ogra_> dpm, if it is not in 60 it wont be in 61 either ... landing is broekn <ogra_> we are in TRAINCON-0 <ogra_> (see the ML) <plars> ogra_: sil2100: I'm off today, but I'll be around on and off. I'll get a run going right now on 60 with that. Basically I should just remove dbus-x11 right? <ogra_> plars, rm /etc/X11/Xsession.d/75dbus_dbus-launch <plars> ogra_: ok, will do <ogra_> plars, that should be sufficiaent <ogra_> removing the package would get you dependency issues i guess <ogra_> plars, i think sudoku_app should be a good test to re-run ... it was rock solid before for quite some time and only broke after this landing <ogra_> Saviq, i'll hold back the upload for that ^^^ <plars> ogra_: do you want a full rerun or just a few to spot check it? <ogra_> since we can get some real data <dpm> ok, thanks ogra_ <ogra_> plars, whatever causes theleast work for you <Saviq> ogra_, I'll do a run too <ogra_> plars, if a full re-run is easier do that ... just one app that was known stable before the split would do too though <plars> ogra_, sil2100: ok, I have sudoku running on one with that file removed right now <ogra_> great <plars> looking good so far :) <ogra_> yay <plars> ogra_, sil2100: sudoku passed completely <ogra_> ok <ogra_> i guess that proves the theory <Mirv> dpm: sil2100: nope, there's just some confusion, the UITK build is still in the silo <plars> I'll run a few other tests but looks good <ogra_> thanks ! <plars> ogra_, sil2100: I tried a couple more and it seems good still :) <ogra_> ok, i'll upload the dependency change and we can roll a new image <ogra_> mterry can then sort out any possible fallout when he is back at work <tedg> ogra_, With the dependency change do we drop out of traincon-0 ? <ogra_> tedg, if we see the tests pass ... but even then there are jenkins issues with the silos <tedg> Uhg, okay. <ogra_> yeah, it came all together :) <ogra_> (if the shit hits the fan it pours ... or how they say) <ogra_> murphys monday ;) <tedg> Heh, I just want to finally land the UAL name change. <tedg> Tired of blocking other fixes. <cjwatson> tedg: I haven't quite landed click yet for that but it should hopefully be on its way soon; there's an entry in the spreadsheet now <tedg> cjwatson, Why don't you put it in silo 18? <tedg> cjwatson, Then we can land them johncrosby@example.net. It has to be retested anyway. <tedg> I'm guessing that silo will be blocked by traincon though. So that'll have to get resolves. <tedg> resolved <cjwatson> tedg: have a couple of other things in the same landing, ideally <cjwatson> tedg: would rather test it independently <tedg> cjwatson, Okay, if you want to test with the ubuntu-app* binaries you can get them in silo 18. <Wellark> thostr_: landing-015 TESTING DONE. <Wellark> now, we would really need that landed even though we are on TRAINCON-0 <Wellark> dpm and cwayne are preparing an image for upcoming expo <Wellark> and getting that MP in would update the translations for chinese (tested that also) <Wellark> the MP's are really small <Wellark> so getting them through QA review would be easy <ogra_> Wellark, even if we wouldnt be in TARINCON-0 ... jenkins is messed up <Wellark> :( <Wellark> dpm: ^ <ogra_> he knows <dpm> thanks Wellark, it seems we'll need to wait until the infrastructure is back online <ogra_> the unity8 fix to lift TRAINCON-0 is already building ... once we have test results we can lift TRAINCON-0 <t1mp> Wellark, dpm phablet-config writable-image --ppa ppa:ci-train-ppa-service/landing-018 for the demo images? <t1mp> not ideal.. <ogra_> (but thats still a few hours) * Wellark sees what is in landing-018 <Wellark> dpm: but anyway. once landing-015 lands you will have the chinese translations <Wellark> I tested them and they at least seem to work :) <dpm> ok, excellent, thanks <Wellark> dpm: seems that the pin unlock dialog is not translated though <Wellark> but all those strings are coming from unity8 ATM <Wellark> + I'm sure you have the sims unlocked anyway at the expo <sil2100> Jenkins messed up again? <ogra_> sil2100, dunno, was it ever fixed ? * ogra_ hanst heard any info on it except that it was broken this morning <ogra_> sil2100, but worse, the archive is broken ... unity8 FTBFS due to broken deps of dbus-test-runner on gvfs-backends <sil2100> ogra_: Mirv mentioned later that it got fixed, so I updated the header to indicate it being fixed <ogra_> oh, ok <sil2100> Well, before changing to TRAINCON-0 ;p <sil2100> uh <ogra_> seems -proposed is briken <ogra_> *broken <cjwatson> the archive isn't broken, ldb/libsmbclient is broken <sil2100> Sucks, since I would like this to propagate already! <cjwatson> just you know to clarify <ogra_> i dont really get the relation when looking at the excuses page but accoing to pitti and cjwatson it has to do with libsmbclient <sil2100> cjwatson: thanks :) <ogra_> well, cant propagate something that doesnt build <cjwatson> you can see it on http://people.canonical.com/~ubuntu-archive/proposed-migration/update_output.txt by searching for ": ldb" <cjwatson> this sort of thing is not on excuses <ogra_> ah <cjwatson> You can see the real error by attempting "apt-get install dbus-test-runner gvfs-backends libsmbclient samba-libs" in utopic-proposed <cjwatson> samba-libs : Depends: libldb1 (< 1:1.1.17~) but 1:1.1.17-1 is to be installed <cjwatson> => samba merge needed <ogra_> :( <ogra_> ans zul just mentioned he wont have time to do that soon <cjwatson> but if you can get a silo working then we can do a unity8 build that way <ogra_>
{'channel': 'ubuntu-ci-eng', 'month': 6}