|
2022-08-02
ยง
|
| 13:19 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2028.codfw.wmnet with reason: host reimage |
[production] |
| 13:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:17 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:15 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a4499e5ac23a0558bed276e2b74134590afc5c95: Revert "testwiki: Add mediawiki.web_ui.interactions stream" (T314151, T311268) (duration: 03m 19s) |
[production] |
| 13:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:09 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: c2fb8a58d8f62e29a15ebee26198e79e4597d24c: Enable RealtimePreview on Group 0 wikis (T314150) (duration: 03m 21s) |
[production] |
| 13:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32154 and previous config saved to /var/cache/conftool/dbconfig/20220802-130636-marostegui.json |
[production] |
| 13:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1096:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32153 and previous config saved to /var/cache/conftool/dbconfig/20220802-130428-marostegui.json |
[production] |
| 13:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T312972)', diff saved to https://phabricator.wikimedia.org/P32152 and previous config saved to /var/cache/conftool/dbconfig/20220802-130351-marostegui.json |
[production] |
| 13:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2013.codfw.wmnet with OS bullseye |
[production] |
| 13:00 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.reimage for host elastic2028.codfw.wmnet with OS bullseye |
[production] |
| 13:00 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on ganeti2013.codfw.wmnet with reason: Remove node for eventual reimage, T311686 |
[production] |
| 12:59 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on ganeti2013.codfw.wmnet with reason: Remove node for eventual reimage, T311686 |
[production] |
| 12:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P32151 and previous config saved to /var/cache/conftool/dbconfig/20220802-124845-marostegui.json |
[production] |
| 12:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P32150 and previous config saved to /var/cache/conftool/dbconfig/20220802-123338-marostegui.json |
[production] |
| 12:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T312972)', diff saved to https://phabricator.wikimedia.org/P32149 and previous config saved to /var/cache/conftool/dbconfig/20220802-121832-marostegui.json |
[production] |
| 12:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1180 (T312972)', diff saved to https://phabricator.wikimedia.org/P32148 and previous config saved to /var/cache/conftool/dbconfig/20220802-121624-marostegui.json |
[production] |
| 12:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 12:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 12:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 12:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 12:01 |
<marostegui> |
dbmaint x1@eqiad T314087 |
[production] |
| 11:57 |
<marostegui> |
dbmaint s7@eqiad T314377 |
[production] |
| 11:57 |
<marostegui> |
dbmaint s3@eqiad T314377 |
[production] |
| 11:57 |
<marostegui> |
dbmaint s8@eqiad T314377 |
[production] |
| 11:54 |
<marostegui> |
dbmait s8@eqiad T314377 |
[production] |
| 11:54 |
<marostegui> |
dbmait s3@eqiad T314377 |
[production] |
| 11:50 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 11:48 |
<marostegui> |
dbmait s7@eqiad T314377 |
[production] |
| 11:46 |
<marostegui> |
dbmait s4@eqiad T314377 |
[production] |
| 11:35 |
<elukey> |
restart rsyslog on ml-serve1006 |
[production] |
| 10:50 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-worker1082.eqiad.wmnet with reason: T312626 btullis |
[production] |
| 10:50 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-worker1082.eqiad.wmnet with reason: T312626 btullis |
[production] |
| 10:49 |
<godog> |
grow sda3 by 100G on thanos-be2004 - T314275 |
[production] |
| 10:42 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: cluster=wikireplicas-b,name=dbproxy1018.eqiad.wmnet |
[production] |
| 10:42 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=wikireplicas-b,name=dbproxy1019.eqiad.wmnet |
[production] |
| 10:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |