3201-3250 of 10000 results (37ms)
2021-07-22 ยง
16:27 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2010.codfw.wmnet with reason: REIMAGE [production]
16:25 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2010.codfw.wmnet with reason: REIMAGE [production]
16:25 <mbsantos@deploy1002> Finished deploy [kartotherian/deploy@0a38bc5]: Rollback maps2007 mirroring (duration: 00m 20s) [production]
16:24 <mbsantos@deploy1002> Started deploy [kartotherian/deploy@0a38bc5]: Rollback maps2007 mirroring [production]
16:20 <mbsantos@deploy1002> Finished deploy [kartotherian/deploy@fb4bc10]: Preparing maps2007 to mirror traffic to the Tegola service (no-op) (duration: 00m 20s) [production]
16:20 <mbsantos@deploy1002> Started deploy [kartotherian/deploy@fb4bc10]: Preparing maps2007 to mirror traffic to the Tegola service (no-op) [production]
16:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db2091 (re)pooling @ 5%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P16860 and previous config saved to /var/cache/conftool/dbconfig/20210722-161333-root.json [production]
15:45 <marostegui> Stop db2091 for onsite maintenance [production]
15:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2091', diff saved to https://phabricator.wikimedia.org/P16859 and previous config saved to /var/cache/conftool/dbconfig/20210722-154408-marostegui.json [production]
15:22 <moritzm> installing dnspython bugfix updates from Buster 10.10 point release [production]
15:14 <mmandere> pool lvs1015 - T286065 [production]
15:14 <jynus> shutdown db2097 for hw servicing T287072 [production]
15:11 <moritzm> re-enabled puppet after row C switch maintenance completed [production]
15:11 <mmandere> pool cp108[3-6].eqiad.wmnet - T286065 [production]
14:58 <moritzm> disabled puppet temporarily for Row C switch maintenance [production]
14:50 <mmandere@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs1015.eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:50 <mmandere@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs1015.eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:47 <mmandere> depool lvs1015 - T286065 [production]
14:40 <mmandere@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp[1083-1086].eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:40 <mmandere@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on cp[1083-1086].eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:37 <mmandere> depool cp108[3-6].eqiad.wmnet - T286065 [production]
14:29 <effie> restarting pybal in lvs2009 and lvs1015 [production]
14:27 <moritzm> installing libwebp security updates on stretch [production]
14:25 <effie> restarting pybal in lvs2010 and lvs1016 [production]
14:22 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
14:20 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 0208fc2b71863c91c3e767373d4bea1a2eaf178d: Growth: Add mentor dashboard related config (T278920) (duration: 00m 55s) [production]
13:52 <kormat@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:47 <kormat@cumin1001> START - Cookbook sre.dns.netbox [production]
13:04 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.37.0-wmf.15 [production]
12:50 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1014.eqiad.wmnet with reason: REIMAGE [production]
12:48 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc1014.eqiad.wmnet with reason: REIMAGE [production]
12:40 <Amir1> cleaning flaggedrevs auto-approve logs in dewiki [production]
12:17 <Amir1> cleaning rest of auto-approve logs of ruwiki [production]
12:01 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[1421-1422].eqiad.wmnet with reason: new host [production]
12:01 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[1421-1422].eqiad.wmnet with reason: new host [production]
11:36 <Lucas_WMDE> EU backport+config window done [production]
11:35 <hnowlan_> removing maps2010 from old maps cassandra cluster [production]
11:35 <lucaswerkmeister-wmde@deploy1002> Synchronized w/touch.php: Config: [[gerrit:705690|Avoid using MWHttpRequest::factory()]] (2/2) (duration: 01m 04s) [production]
11:34 <lucaswerkmeister-wmde@deploy1002> Synchronized w/favicon.php: Config: [[gerrit:705690|Avoid using MWHttpRequest::factory()]] (1/2) (duration: 01m 04s) [production]
11:23 <lucaswerkmeister-wmde@deploy1002> Synchronized w/robots.php: Config: [[gerrit:705682|Avoid using WikiPage::factory()]] (duration: 01m 06s) [production]
10:59 <mutante> mw1421, mw1422 - puppetmaster - cleaning certs, reimaged hosts [production]
10:45 <effie> restart pybal on lvs2009 and lvs1015 [production]
10:45 <jiji@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mwdebug [production]
10:42 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: REIMAGE [production]
10:42 <effie> restart pybal on lvs2010 and lvs1016 [production]
10:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: REIMAGE [production]
10:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: REIMAGE [production]
10:35 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: REIMAGE [production]
10:19 <mutante> mw1421, mw1422 - converting from app to API server for balance in row A [production]
10:09 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1422.eqiad.wmnet [production]