|
2022-08-02
§
|
| 14:04 |
<godog> |
grow sda/sdb 3 by 100G on thanos-be1001 - T314275 |
[production] |
| 14:03 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on centrallog2002.codfw.wmnet with reason: pdu |
[production] |
| 14:03 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on centrallog2002.codfw.wmnet with reason: pdu |
[production] |
| 14:01 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on prometheus2005.codfw.wmnet with reason: pdu |
[production] |
| 14:01 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on prometheus2005.codfw.wmnet with reason: pdu |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-tls |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2032.codfw.wmnet,service=ats-be |
[production] |
| 13:57 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2031.codfw.wmnet,service=ats-be |
[production] |
| 13:56 |
<godog> |
schedule poweroff for centrallog2002 at 16 utc - T310070 |
[production] |
| 13:54 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-be |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32159 and previous config saved to /var/cache/conftool/dbconfig/20220802-135435-marostegui.json |
[production] |
| 13:53 |
<godog> |
depool and poweroff prometheus2005 - T310070 |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-tls |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=ats-tls |
[production] |
| 13:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[3-4].codfw.wmnet,service=varnish-fe |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32158 and previous config saved to /var/cache/conftool/dbconfig/20220802-135226-marostegui.json |
[production] |
| 13:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:52 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=ats-tls |
[production] |
| 13:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T312972)', diff saved to https://phabricator.wikimedia.org/P32157 and previous config saved to /var/cache/conftool/dbconfig/20220802-135155-marostegui.json |
[production] |
| 13:51 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=ats-tls |
[production] |
| 13:51 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp203[1-2].codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-be |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2030.codfw.wmnet,service=ats-be |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=ats-tls |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=varnish-fe |
[production] |
| 13:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2029.codfw.wmnet,service=ats-be |
[production] |
| 13:45 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: host reimage |
[production] |
| 13:42 |
<jbond@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: host reimage |
[production] |
| 13:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:42 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:41 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2013.codfw.wmnet with OS bullseye |
[production] |
| 13:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:40 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:754933|Enable usage tracking for statement for cebwiki (T296384)]] – expected to gradually increase number of wbc_entity_usage and probably recentchanges rows on cebwiki, but not too much, see task for details (duration: 03m 06s) |
[production] |
| 13:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:39 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2028.codfw.wmnet with OS bullseye |
[production] |
| 13:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P32156 and previous config saved to /var/cache/conftool/dbconfig/20220802-133648-marostegui.json |
[production] |
| 13:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:34 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:754937|Introduce $wmgEntityUsageModifierLimitsStatement (T296384)]] (2/2) (duration: 03m 21s) |
[production] |
| 13:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:30 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:754937|Introduce $wmgEntityUsageModifierLimitsStatement (T296384)]] (1/2) (duration: 03m 16s) |
[production] |
| 13:30 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ganeti2028.codfw.wmnet with reason: Power down for PDU maintenance, T309957 |
[production] |
| 13:30 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on ganeti2028.codfw.wmnet with reason: Power down for PDU maintenance, T309957 |
[production] |
| 13:27 |
<jbond@cumin2002> |
START - Cookbook sre.hosts.reimage for host puppetmaster2004.codfw.wmnet with OS buster |
[production] |
| 13:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2013.codfw.wmnet with reason: host reimage |
[production] |
| 13:24 |
<vgutierrez> |
restarting ATS 9.x instances to apply https://gerrit.wikimedia.org/r/819585 - T309651 |
[production] |