2051-2100 of 10000 results (62ms)
2019-07-03 ยง
11:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:520174|Configuring Namespaces at pawikisource]] (T226959) (duration: 00m 52s) [production]
11:05 <moritzm> rebooting krypton nodes to pick up MDS-enabled qemu [production]
11:05 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:05 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
11:04 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:04 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:36 <Amir1> start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wiktionary extensions/Cognate/maintenance/populateCognatePages.php (T226358) [production]
10:12 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:11 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:11 <moritzm> rolling reboot of eventschema service hosts to pick up MDS-enabled qemu [production]
10:00 <marostegui> Drop secret and stratch_tokens columns from the private wiki list T226826 [production]
09:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:58 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:54 <moritzm> rebooting netmon2001 for kernel security update [production]
09:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:52 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:47 <moritzm> rebooting debmonitor nodes to pick up MDS-enabled qemu [production]
09:46 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:46 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:27 <moritzm> rebooting failoid nodes to pick up MDS-enabled qemu [production]
09:25 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:25 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:01 <moritzm> rolling reboot of kubernetes masters in eqiad to pick up MDS-enabled qemu [production]
08:44 <moritzm> rolling reboot of kubernetes masters in codfw to pick up MDS-enabled qemu [production]
08:44 <moritzm> rolling reboot of kubernetes masters in codfw [production]
08:43 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:43 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:45 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:45 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:34 <godog> reenable puppet fleetwide [production]
07:33 <marostegui> Upgrade db2078 (s8 codfw master) [production]
07:25 <marostegui> Upgrade db2100 (snapshots on that hosts are finished) [production]
07:24 <godog> temporarily disable puppet to test/apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/520012 [production]
07:23 <moritzm> updated buster installer d-i image to RC3 [production]
07:10 <marostegui> Drop secret and scratch_tokens from labswiki (wikitech) and labstestwiki - T226826 [production]
07:06 <marostegui> Drop secret and scratch_tokens from fishbowl wiki list T226826 [production]
07:05 <godog> add 150G to graphite hosts lv, was at 94% utilization [production]
06:55 <godog> depool and roll-restart swift proxy - T209182 [production]
06:42 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1069 status (duration: 00m 28s) [production]
06:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Switchover x1 master eqiad from db1069 to db1120 T226358 (duration: 00m 27s) [production]
06:00 <marostegui> Starting x1 failover from db1069 to db1120 - T226358 [production]
06:00 <elukey> move the zookeeper puppet submodule into operations/puppet - T226466 [production]
05:52 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:52 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:21 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:21 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:03 <vgutierrez> restarting pybal on lvs4006 [production]
05:02 <marostegui> Start pre-failover steps for x1 - T226358 [production]
04:47 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:47 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]