2851-2900 of 10000 results (67ms)
2019-06-20 ยง
13:22 <ema@cumin1001> END (FAIL) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=99) [production]
13:21 <jijiki> depool mw1311 [production]
13:20 <marostegui> Reload haproxy on dbproxy1010 and dbproxy1011 to depool labsdb1011 - T222978 [production]
13:16 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
13:11 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
13:04 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:59 <jijiki> Disable puppet on jobrunners to merge 518023 and 518018 [production]
12:56 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:50 <ema> powercycle cp2017, stuck rebooting [production]
12:44 <hashar> Upgrading packages on contint1001 [production]
12:44 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:40 <hashar> Upgrading java/jenkins on releases* hosts # T226159 [production]
12:37 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:36 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:36 <moritzm> updated jenkins package on apt.wikimedia.org to 2.176.1 for jessie and stretch (T226159) [production]
11:54 <Amir1> EU SWAT is done [production]
11:49 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:517820|Switch property terms migration to WRITE_BOTH on test wikidata (T225051)]] (duration: 00m 58s) [production]
11:42 <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:517819|Introduce config variables for new terms store in mediawiki-config (T226086)]], Part II (duration: 00m 57s) [production]
11:39 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:517819|Introduce config variables for new terms store in mediawiki-config (T226086)]] (duration: 00m 57s) [production]
11:20 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:504261|Remove ExternalGuidanceEnableContentDetection]] (T219819) (duration: 01m 00s) [production]
11:14 <moritzm> rebooting mw2235, mw2255, mw2271 for MDS kernel update [production]
11:12 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:516053|Fix import group name]] (duration: 00m 57s) [production]
11:09 <mlitn@deploy1001> Finished scap: [SDC] Enable depicts qualifiers on Commons & increase rate limits (duration: 20m 34s) [production]
10:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:58 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:58 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:58 <moritzm> rebooting scb100[12], mw2139 for MDS kernel update (their CPUs were previously unsupported by Intel, but are now covered with the new release) [production]
10:48 <mlitn@deploy1001> Started scap: [SDC] Enable depicts qualifiers on Commons & increase rate limits [production]
10:33 <marostegui> Deploy schema change on the fishbowl wikis list on T225643 [production]
10:31 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:24 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:23 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:17 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:11 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:10 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:58 <_joe_> upgraded service-checker T225707 [production]
09:56 <ema@cumin1001> END (FAIL) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=99) [production]
09:51 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:50 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:44 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:30 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:25 <marostegui> Remove dbprov1001:/srv/backups/tmp/db1112 - T225981 [production]
09:24 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:21 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:17 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:17 <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 T225998 T226048 [production]
08:39 <marostegui> Stop Mysql on db1124: s1, s3, s5 and s8 to upgrade mysql, this will generate lag on labs [production]
07:59 <marostegui> Stop MYSQL and reboot db2084 [production]
07:15 <marostegui> Transfer dbprov1001:/srv/backups/tmp/db1112/sqldata to db1077 T225981 [production]