3601-3650 of 10000 results (98ms)
2019-05-20 ยง
15:26 <marostegui> Stop replication on labsdb1011 to start compressing tables - T222978 [production]
15:13 <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on group 0 (T188327) (duration: 00m 55s) [production]
14:54 <bblack> rebooting lvs1013, lvs1014, lvs1015 (not in active service, yet) [production]
14:43 <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@89b0ad0]: Migrating RecordLintJob to PHP7 - T219148 (duration: 00m 55s) [production]
14:42 <jiji@deploy1001> Started deploy [cpjobqueue/deploy@89b0ad0]: Migrating RecordLintJob to PHP7 - T219148 [production]
14:21 <marostegui> Reload haproxy on dbroxy1010 to depool labsdb1011 [production]
14:14 <marostegui> Reload haproxy on dbroxy1010 to repool labsdb1010 [production]
13:58 <mobrovac> bootstrap restbase1026-b - T219404 [production]
12:43 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1126 and db1134 (duration: 00m 50s) [production]
11:44 <fsero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:44 <fsero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:28 <fsero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:28 <fsero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:21 <fsero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:21 <fsero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:17 <mobrovac> bootstrap restbase1026-a - T219404 [production]
11:16 <fsero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:15 <fsero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:01 <arturo> icinga downtime toolschecker for 3h for T223332 [production]
10:45 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:44 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:43 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:511398| Bumping portals to master (T128546)]] (duration: 00m 49s) [production]
10:42 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:511398| Bumping portals to master (T128546)]] (duration: 00m 50s) [production]
10:27 <moritzm> rebooting contint1001 for kernel update [production]
10:25 <hashar> contint1001: docker image prune -f | Total reclaimed space: 7.115GB | T207707 [production]
10:20 <hashar> Stopped Zuul gracefully [production]
10:18 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:18 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:18 <fsero> puppet reenabled certs renewed - T221346 [production]
10:08 <fsero> rolling over certs into mcrouter proxies codfw - T221346 [production]
10:03 <fsero> rolling over certs into mcrouter proxies eqiad - T221346 [production]
09:42 <marostegui> Remove db2036 from tendril and zarcillo - T223885 [production]
09:39 <marostegui> Stop MySQL on db2036 T223885 [production]
09:38 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2036, going to be decommissioned T223885 (duration: 00m 49s) [production]
09:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2036, going to be decommissioned T223885 (duration: 00m 49s) [production]
09:35 <fsero> rolling over new certs to all mcrouter hosts except proxys - T221346 [production]
09:26 <fsero> continue to rolling over new certs - T221346 [production]
09:01 <fsero> disabling puppet on mcrouter hosts for regenerating certs - T221346 [production]
08:49 <moritzm> installing atftpd security updates [production]
08:43 <mobrovac> bootstrap restbase1025-c - T219404 [production]
08:38 <moritzm> installing samba security updates [production]
08:36 <moritzm> installing ghostscript security updates on jessie [production]
08:25 <moritzm> installing cups-filter security updates on jessie (prerequisite for ghostscript security update) [production]
07:48 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1126 and db1134 (duration: 00m 48s) [production]
07:26 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2046 (duration: 00m 50s) [production]
06:25 <elukey> rebuild and upload memkeys 20181031-1 to stretch-wikimedia [production]
06:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1126 and db1134 (duration: 00m 49s) [production]
06:20 <elukey> upgrade memkeys to version 20181031-1 on all the mc* hosts (was deployied only on a few of them) - T208376 [production]
06:11 <mobrovac> bootstrap restbase1025-b - T219404 [production]
06:00 <elukey> powercycle analytics1071 - soft lockups error messages in the dmesg [production]