751-800 of 10000 results (62ms)
2019-07-03 §
08:44 <moritzm> rolling reboot of kubernetes masters in codfw to pick up MDS-enabled qemu [production]
08:44 <moritzm> rolling reboot of kubernetes masters in codfw [production]
08:43 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:43 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:45 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:45 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:34 <godog> reenable puppet fleetwide [production]
07:33 <marostegui> Upgrade db2078 (s8 codfw master) [production]
07:25 <marostegui> Upgrade db2100 (snapshots on that hosts are finished) [production]
07:24 <godog> temporarily disable puppet to test/apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/520012 [production]
07:23 <moritzm> updated buster installer d-i image to RC3 [production]
07:10 <marostegui> Drop secret and scratch_tokens from labswiki (wikitech) and labstestwiki - T226826 [production]
07:06 <marostegui> Drop secret and scratch_tokens from fishbowl wiki list T226826 [production]
07:05 <godog> add 150G to graphite hosts lv, was at 94% utilization [production]
06:55 <godog> depool and roll-restart swift proxy - T209182 [production]
06:42 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1069 status (duration: 00m 28s) [production]
06:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Switchover x1 master eqiad from db1069 to db1120 T226358 (duration: 00m 27s) [production]
06:00 <marostegui> Starting x1 failover from db1069 to db1120 - T226358 [production]
06:00 <elukey> move the zookeeper puppet submodule into operations/puppet - T226466 [production]
05:52 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:52 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:21 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:21 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:03 <vgutierrez> restarting pybal on lvs4006 [production]
05:02 <marostegui> Start pre-failover steps for x1 - T226358 [production]
04:47 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:47 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:34 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:34 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:24 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:24 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:23 <vgutierrez> rebooting primary lvs servers for MDS security updates [production]
00:14 <eileen> process-control config revision is 8e215d07f2 (renable jobs) [production]
00:08 <eileen> civicrm revision is 8a4451f390, config revision is ec8c43ee86 Redis [production]
00:05 <eileen> process-control config revision is ec8c43ee86 (Redis turned on) [production]
2019-07-02 §
23:42 <eileen> civicrm revision is 8a4451f390, config revision is c02a038331 (mysql locks enabled) [production]
23:36 <catrope@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/Echo/: T226594 (duration: 00m 51s) [production]
23:34 <catrope@deploy1001> Synchronized php-1.34.0-wmf.11/skins/MonoBook/: T226594 (duration: 00m 50s) [production]
22:35 <eileen> civicrm revision changed from 96985fcc4b to 8a4451f390, config revision is af9e657134 [production]
20:35 <mutante> contint1001 - created new partitions on /dev/sdc and /dev/sdd; created new RAID 1 over /dev/sdc1 and /dev/sdd1 [production]
20:28 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy (duration: 14m 43s) [production]
20:20 <mutante> contint1001 - temp installing parted for labeling new disks sdc and sdd for raid for docker images (T207707) [production]
20:13 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy [production]
19:37 <krinkle@deploy1001> Finished scap: l10n sync did not work as expected, try full scap to fix missing i18n message for 9963d843622 (duration: 18m 24s) [production]
19:18 <krinkle@deploy1001> Started scap: l10n sync did not work as expected, try full scap to fix missing i18n message for 9963d843622 [production]
19:07 <krinkle@deploy1001> scap sync-l10n completed (1.34.0-wmf.11) (duration: 00m 47s) [production]
19:05 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/AbuseFilter/: 9963d843622b / T227095 (duration: 00m 51s) [production]
19:03 <krinkle@deploy1001> scap sync-l10n completed (1.34.0-wmf.11) (duration: 00m 48s) [production]
19:00 <mholloway-shell@deploy1001> Finished deploy [recommendation-api/deploy@a29da76]: Update recommendation-api to 4f50c71 (duration: 02m 50s) [production]
18:57 <mholloway-shell@deploy1001> Started deploy [recommendation-api/deploy@a29da76]: Update recommendation-api to 4f50c71 [production]