|
2022-08-02
§
|
| 14:39 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32162 and previous config saved to /var/cache/conftool/dbconfig/20220802-143952-marostegui.json |
[production] |
| 14:37 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reimage for host puppetmaster1004.eqiad.wmnet with OS buster |
[production] |
| 14:32 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2060.codfw.wmnet with reason: host reimage |
[production] |
| 14:28 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2060.codfw.wmnet with reason: host reimage |
[production] |
| 14:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P32161 and previous config saved to /var/cache/conftool/dbconfig/20220802-142446-marostegui.json |
[production] |
| 14:23 |
<Emperor> |
shutdown ms-be20[30,45,52] for PDU work T309957 |
[production] |
| 14:22 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[2030,2045,2052].codfw.wmnet with reason: shutdown for PDU replacement |
[production] |
| 14:21 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be[2030,2045,2052].codfw.wmnet with reason: shutdown for PDU replacement |
[production] |
| 14:12 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.reimage for host elastic2060.codfw.wmnet with OS bullseye |
[production] |
| 14:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P32160 and previous config saved to /var/cache/conftool/dbconfig/20220802-140940-marostegui.json |
[production] |
| 14:05 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetmaster2004.codfw.wmnet with OS buster |
[production] |
| 14:04 |
<godog> |
grow sda/sdb 3 by 100G on thanos-be1001 - T314275 |
[production] |
| 14:03 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on centrallog2002.codfw.wmnet with reason: pdu |
[production] |
| 14:03 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on centrallog2002.codfw.wmnet with reason: pdu |
[production] |
| 14:01 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on prometheus2005.codfw.wmnet with reason: pdu |
[production] |
| 14:01 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on prometheus2005.codfw.wmnet with reason: pdu |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-tls |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2032.codfw.wmnet,service=ats-be |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2031.codfw.wmnet,service=ats-be |
[production] |
| 13:56 |
<godog> |
schedule poweroff for centrallog2002 at 16 utc - T310070 |
[production] |
| 13:54 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-be |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32159 and previous config saved to /var/cache/conftool/dbconfig/20220802-135435-marostegui.json |
[production] |
| 13:53 |
<godog> |
depool and poweroff prometheus2005 - T310070 |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-tls |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-tls |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=varnish-fe |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32158 and previous config saved to /var/cache/conftool/dbconfig/20220802-135226-marostegui.json |
[production] |
| 13:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:52 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=ats-tls |
[production] |
| 13:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32157 and previous config saved to /var/cache/conftool/dbconfig/20220802-135155-marostegui.json |
[production] |
| 13:51 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=ats-tls |
[production] |
| 13:51 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-be |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-be |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=ats-tls |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=ats-be |
[production] |
| 13:45 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: host reimage |
[production] |
| 13:42 |
<jbond@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: host reimage |
[production] |
| 13:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:42 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:41 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2013.codfw.wmnet with OS bullseye |
[production] |
| 13:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:40 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:754933|Enable usage tracking for statement for cebwiki (T296384)]] – expected to gradually increase number of wbc_entity_usage and probably recentchanges rows on cebwiki, but not too much, see task for details (duration: 03m 06s) |
[production] |
| 13:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:39 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2028.codfw.wmnet with OS bullseye |
[production] |