|
2021-04-21
§
|
| 07:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2005.codfw.wmnet |
[production] |
| 07:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2003.codfw.wmnet |
[production] |
| 07:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2004.codfw.wmnet |
[production] |
| 07:50 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2004.codfw.wmnet |
[production] |
| 07:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2003.codfw.wmnet |
[production] |
| 07:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet |
[production] |
| 07:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet |
[production] |
| 07:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet |
[production] |
| 07:38 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet |
[production] |
| 06:49 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 06:49 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 06:42 |
<elukey> |
upload hue_4.9.0-2+deb10u1 to buster-wikimedia |
[production] |
| 06:11 |
<marostegui> |
Stop MySQL on db1074 to clone db1156 (there will be lag in s2 in wiki replicas) T258361 |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 to clone db1156 T258361', diff saved to https://phabricator.wikimedia.org/P15491 and previous config saved to /var/cache/conftool/dbconfig/20210421-061019-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1086.eqiad.wmnet |
[production] |
| 05:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1086.eqiad.wmnet |
[production] |
| 00:38 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:36 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:15 |
<ryankemper> |
[WDQS] Pooled `wdqs1003` |
[production] |
| 00:14 |
<ryankemper> |
[WDQS] Pooled `wdqs2008` |
[production] |
| 00:06 |
<ryankemper> |
`sudo -i wmf-auto-reimage-host -p T280382 wdqs1006.eqiad.wmnet` |
[production] |
| 00:04 |
<ryankemper> |
[WDQS] pooled `wdqs1004` |
[production] |
|
2021-04-20
§
|
| 23:46 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) |
[production] |
| 23:44 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/urwiki.yaml: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) |
[production] |
| 23:42 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 58s) |
[production] |
| 23:38 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=urwiki GrowthExperiments # T280067 |
[production] |
| 23:38 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 3/3) (duration: 00m 56s) |
[production] |
| 23:36 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/elwiki.yaml: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 2/3) (duration: 00m 57s) |
[production] |
| 23:35 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 1/3) (duration: 00m 57s) |
[production] |
| 23:34 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript deleteEqualMessages.php --wiki=hrwiki --delete |
[production] |
| 23:32 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=elwiki GrowthExperiments # T280172 |
[production] |
| 23:31 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 3/3) (duration: 00m 57s) |
[production] |
| 23:28 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist growthexperiments sql.php --cluster=extension1 /srv/mediawiki/php-1.37.0-wmf.1/extensions/GrowthExperiments/maintenance/schemas/mysql/growthexperiments_mentee_data.sql # T279587 |
[production] |
| 23:28 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/cawiki.yaml: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 2/3) (duration: 00m 57s) |
[production] |
| 23:27 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 425d77b73f48b3e16a5aa2c0086f292d370cd17e: cawiki: Enable Growth team features in stealth mode (T280673; 1/3) (duration: 00m 57s) |
[production] |
| 23:24 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=cawiki GrowthExperiments # T280673 |
[production] |
| 23:11 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on furud.codfw.wmnet with reason: REIMAGE |
[production] |
| 23:09 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on furud.codfw.wmnet with reason: REIMAGE |
[production] |
| 23:05 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:03 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:14 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:10 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:46 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |