1651-1700 of 10000 results (34ms)
2021-07-22 §
14:47 <mmandere> depool lvs1015 - T286065 [production]
14:40 <mmandere@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp[1083-1086].eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:40 <mmandere@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on cp[1083-1086].eqiad.wmnet with reason: Eqiad row C maintenance [production]
14:37 <mmandere> depool cp108[3-6].eqiad.wmnet - T286065 [production]
14:29 <effie> restarting pybal in lvs2009 and lvs1015 [production]
14:27 <moritzm> installing libwebp security updates on stretch [production]
14:25 <effie> restarting pybal in lvs2010 and lvs1016 [production]
14:22 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
14:20 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 0208fc2b71863c91c3e767373d4bea1a2eaf178d: Growth: Add mentor dashboard related config (T278920) (duration: 00m 55s) [production]
13:52 <kormat@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:47 <kormat@cumin1001> START - Cookbook sre.dns.netbox [production]
13:04 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.37.0-wmf.15 [production]
12:50 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1014.eqiad.wmnet with reason: REIMAGE [production]
12:48 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc1014.eqiad.wmnet with reason: REIMAGE [production]
12:40 <Amir1> cleaning flaggedrevs auto-approve logs in dewiki [production]
12:17 <Amir1> cleaning rest of auto-approve logs of ruwiki [production]
12:01 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[1421-1422].eqiad.wmnet with reason: new host [production]
12:01 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[1421-1422].eqiad.wmnet with reason: new host [production]
11:36 <Lucas_WMDE> EU backport+config window done [production]
11:35 <hnowlan_> removing maps2010 from old maps cassandra cluster [production]
11:35 <lucaswerkmeister-wmde@deploy1002> Synchronized w/touch.php: Config: [[gerrit:705690|Avoid using MWHttpRequest::factory()]] (2/2) (duration: 01m 04s) [production]
11:34 <lucaswerkmeister-wmde@deploy1002> Synchronized w/favicon.php: Config: [[gerrit:705690|Avoid using MWHttpRequest::factory()]] (1/2) (duration: 01m 04s) [production]
11:23 <lucaswerkmeister-wmde@deploy1002> Synchronized w/robots.php: Config: [[gerrit:705682|Avoid using WikiPage::factory()]] (duration: 01m 06s) [production]
10:59 <mutante> mw1421, mw1422 - puppetmaster - cleaning certs, reimaged hosts [production]
10:45 <effie> restart pybal on lvs2009 and lvs1015 [production]
10:45 <jiji@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mwdebug [production]
10:42 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: REIMAGE [production]
10:42 <effie> restart pybal on lvs2010 and lvs1016 [production]
10:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: REIMAGE [production]
10:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: REIMAGE [production]
10:35 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: REIMAGE [production]
10:19 <mutante> mw1421, mw1422 - converting from app to API server for balance in row A [production]
10:09 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1422.eqiad.wmnet [production]
10:01 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue [production]
10:01 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue [production]
09:11 <XioNoX> depool eqiad to reduce load on one codfw-eqiad link - T287110 [production]
08:48 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet [production]
08:45 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet [production]
08:34 <XioNoX> cr2-codfw> request chassis fpc slot 0 offline - T287110 [production]
07:24 <hashar@deploy1002> Finished deploy [integration/docroot@b3e39b0]: build: Updating mediawiki/mediawiki-codesniffer to 37.0.0 (duration: 00m 09s) [production]
07:24 <hashar@deploy1002> Started deploy [integration/docroot@b3e39b0]: build: Updating mediawiki/mediawiki-codesniffer to 37.0.0 [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1170 (s2, s7), pool db1105 (s2) and db1098 (s7) into dump T286888', diff saved to https://phabricator.wikimedia.org/P16844 and previous config saved to /var/cache/conftool/dbconfig/20210722-070114-marostegui.json [production]
06:20 <ryankemper> [WDQS] Pooled `wdqs1006` (was still depooled following data-transfer cookbook runs from several hours ago) [production]
05:41 <ryankemper> [WDQS] Restarted `wdqs-blazegraph` on `wdqs1013` [production]
05:31 <ryankemper> T281327 [Elastic] Unbanned `elastic2043.codfw.wmnet` from all 3 cirrus/elasticsearch clusters; node is back in the fleet [production]
00:52 <ryankemper@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2043.codfw.wmnet with reason: REIMAGE [production]
00:50 <ryankemper@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2043.codfw.wmnet with reason: REIMAGE [production]
2021-07-21 §
23:15 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable Score on enwikisource, plwikisource. Disable on all private/lockeddown wikis (T257066) (duration: 01m 03s) [production]
22:44 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:41 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` [production]