|
2021-04-21
§
|
| 06:49 |
<elukey> |
stop all services on an-coord1001 as prep step for reimage |
[analytics] |
| 06:45 |
<elukey> |
PURGE BINARY LOGS BEFORE '2021-04-14 00:00:00'; on an-coord1001 to free some space before the reimage |
[analytics] |
| 06:42 |
<elukey> |
upload hue_4.9.0-2+deb10u1 to buster-wikimedia |
[production] |
| 06:11 |
<marostegui> |
Stop MySQL on db1074 to clone db1156 (there will be lag in s2 in wiki replicas) T258361 |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 to clone db1156 T258361', diff saved to https://phabricator.wikimedia.org/P15491 and previous config saved to /var/cache/conftool/dbconfig/20210421-061019-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:00 |
<elukey> |
stop timers on an-launcher1002 as prep step for an-coord1001 reimage |
[analytics] |
| 05:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1086.eqiad.wmnet |
[production] |
| 05:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1086.eqiad.wmnet |
[production] |
| 00:38 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:36 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:15 |
<ryankemper> |
[WDQS] Pooled `wdqs1003` |
[production] |
| 00:14 |
<ryankemper> |
[WDQS] Pooled `wdqs2008` |
[production] |
| 00:06 |
<ryankemper> |
`sudo -i wmf-auto-reimage-host -p T280382 wdqs1006.eqiad.wmnet` |
[production] |
| 00:04 |
<ryankemper> |
[WDQS] pooled `wdqs1004` |
[production] |
|
2021-04-20
§
|
| 23:46 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) |
[production] |
| 23:44 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/urwiki.yaml: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) |
[production] |
| 23:42 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 58s) |
[production] |
| 23:38 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=urwiki GrowthExperiments # T280067 |
[production] |
| 23:38 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 3/3) (duration: 00m 56s) |
[production] |
| 23:36 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/elwiki.yaml: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 2/3) (duration: 00m 57s) |
[production] |
| 23:35 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 1/3) (duration: 00m 57s) |
[production] |
| 23:34 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript deleteEqualMessages.php --wiki=hrwiki --delete |
[production] |
| 23:32 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=elwiki GrowthExperiments # T280172 |
[production] |
| 23:31 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 3/3) (duration: 00m 57s) |
[production] |
| 23:28 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist growthexperiments sql.php --cluster=extension1 /srv/mediawiki/php-1.37.0-wmf.1/extensions/GrowthExperiments/maintenance/schemas/mysql/growthexperiments_mentee_data.sql # T279587 |
[production] |
| 23:28 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/cawiki.yaml: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 2/3) (duration: 00m 57s) |
[production] |
| 23:27 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 1/3) (duration: 00m 57s) |
[production] |
| 23:24 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=cawiki GrowthExperiments # T280673 |
[production] |
| 23:18 |
<bstorm> |
deploying updated paws-hub image for https://github.com/toolforge/paws/pull/63 |
[paws] |
| 23:11 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on furud.codfw.wmnet with reason: REIMAGE |
[production] |
| 23:09 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on furud.codfw.wmnet with reason: REIMAGE |
[production] |
| 23:05 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:03 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:20 |
<bd808> |
`clush -w @all -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` |
[tools] |
| 22:19 |
<bd808> |
`clush -w @exec -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` |
[tools] |
| 22:14 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:10 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:52 |
<bd808> |
Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad1.wikimedia.cloud`. Was using wrong domain name in prior update. |
[tools] |
| 21:49 |
<bstorm> |
tagged the latest maintain-kubeusers and deployed to toolforge (with kustomize changes to rbac) after testing in toolsbeta T280300 |
[tools] |
| 21:46 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:27 |
<bd808> |
Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad.wmflabs`. was -2 which is decommed. |
[tools] |
| 20:52 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=ruwiki # T279853 |
[production] |
| 20:48 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd1020.wikimedia.org |
[production] |
| 20:41 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=viwiki # T279853 |
[production] |
| 20:36 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd1020.wikimedia.org |
[production] |