2051-2100 of 10000 results (27ms)
2014-07-31 §
11:37 <hashar> Jenkins: upgrading almost all jobs to use a new label 'UbuntuPrecise' {{bug|68340}} {{gerrit|150785}} [production]
10:49 <hashar> Jenkins: attempting to poll a Trusty slave (integration-slave1004-trusty [10.68.17.148] with label <tt>UbuntuTrusty</tt>). [production]
10:32 <hashar> Jenkins: tweaking jobs labels, that might eventually screw up Zuul/Jenkins entirely. [production]
08:43 <_joe_> start rolling reload of nginx to catch up with the new ssl config [production]
06:50 <springle> labsdb1001 migration complete, should be all systems go [production]
03:19 <LocalisationUpdate> ResourceLoader cache refresh completed at Thu Jul 31 03:18:07 UTC 2014 (duration 18m 6s) [production]
02:36 <LocalisationUpdate> completed (1.24wmf15) at 2014-07-31 02:35:29+00:00 [production]
02:20 <LocalisationUpdate> completed (1.24wmf14) at 2014-07-31 02:19:17+00:00 [production]
02:06 <springle> labsdb1001 migrating to mariadb 10, expect read-only and downtime, see labs-l [production]
2014-07-30 §
23:27 <maxsem> Synchronized php-1.24wmf15/extensions/MwEmbedSupport/: (no message) (duration: 00m 03s) [production]
23:27 <maxsem> Synchronized php-1.24wmf15/extensions/Wikidata/: (no message) (duration: 00m 08s) [production]
23:27 <maxsem> Synchronized php-1.24wmf15/extensions/SyntaxHighlight_GeSHi/: (no message) (duration: 00m 05s) [production]
23:24 <maxsem> Synchronized php-1.24wmf14/extensions/Wikidata: (no message) (duration: 00m 11s) [production]
23:13 <maxsem> Synchronized wmf-config: (no message) (duration: 00m 05s) [production]
21:04 <AaronSchulz> Started populateBacklinkNamespace.php on wikidata and commons [production]
21:02 <bblack> turned icinga email/sms back on [production]
20:24 <bblack> icinga back online again [production]
19:58 <bblack> shutting off icinga to make some optimizations [production]
19:20 <bblack> icinga is now substantially back online. email/sms still disabled for now, and downtimes/acks need to be re-added for known issues [production]
19:06 <csteipp> Synchronized php-1.24wmf14/includes/: (no message) (duration: 00m 05s) [production]
19:04 <csteipp> Synchronized php-1.24wmf15/includes/: (no message) (duration: 00m 07s) [production]
18:59 <bblack> icinga coming back up again for the first time, expect random strangeness to be ignored [production]
18:46 <bblack> temporarily hard-disabling email/sms from icinga via 'mv /usr/bin/mail /usr/bin/mail-disabled' on neon to prevent icinga spam on next startup attempt [production]
17:55 <bblack> stopping icinga service for now while working out other details [production]
17:25 <tacotuesday> repooled elastic1018 and elastic1019 as well [production]
17:21 <Coren> labmon1001 rebooting (final check for proper raid+lvm autodetection) [production]
17:08 <bblack> working on bringing up new neon install (first puppet run, etc) [production]
17:01 <Coren> labmon1001 rebooting (partitioning changes on primary disks) [production]
16:53 <tacotuesday> elastic1017 repooled, shards allocating [production]
16:13 <bd808> scap and dologmsg from tin won't work until neon is back up and running tcpircbot [production]
16:07 <bd808|deploy> Synchronized touch: no-op sync to test scap update (duration: 00m 05s) [production]
16:06 <bd808|deploy> scap announce failed -- timeout connecting to tcpircbot on neon.wikimedia.org [production]
16:04 <bd808|deploy> Updated scap to 4871208 (rely on $PATH for scap scripts) [production]
15:21 <hoo> Synchronized php-1.24wmf15/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.js: touch (duration: 00m 20s) [production]
15:17 <hashar> upgrading php5 on jenkins slaves [production]
15:07 <cmjohnson1> shutting down neon [production]
14:46 <demon> Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 04s) [production]
14:35 <demon> Synchronized wmf-config/PrivateSettings.php: Swift config for Cirrus (duration: 00m 08s) [production]
14:30 <godog> rolling restart of ms-fe* to pick up search backup user [production]
14:17 <bblack> rebooting neon again, trying to fix the disk situation [production]
14:11 <Coren> reinstalling labmon1001 -> change disk partitioning scheme [production]
13:50 <springle> neon read-only fs. fsck + reboot [production]
13:17 <manybubbles> rebuiding Cirrus index for commons to pick up weighted all field [production]
11:17 <_joe_> enabling puppet on all mw* servers [production]
11:15 <_joe_> re-enabling puppet on mw1019, last bunch of tests, then re-enabling globally [production]
10:58 <_joe_> re-enabling puppet on mw1018, testwiki upgraded to the new config and looks fine [production]
09:25 <godog> set weight for ms-be1014 and ms-be1015 to 2300 [production]
08:58 <_joe_> stopping puppet on the appservers, in preparation for releasing change 148099 [production]
08:30 <_joe_> powercycling neon, doesn't respond to requests, ssh hangs, console dark [production]
06:41 <springle> labsdb1001 work in progress; it may misbehave. see labs-l for updates [production]