|
2023-09-12
ยง
|
| 10:13 |
<moritzm> |
disabled nginx/puppetdb/postgresql/microservice on puppetdb1002/2002 to ensure nothing hits the old endpoints anymore |
[production] |
| 10:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on puppetdb2002.codfw.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway |
[production] |
| 10:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on puppetdb2002.codfw.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway |
[production] |
| 10:09 |
<jgiannelos@deploy1002> |
helmfile [staging] DONE helmfile.d/services/wikifeeds: apply |
[production] |
| 10:08 |
<jgiannelos@deploy1002> |
helmfile [staging] START helmfile.d/services/wikifeeds: apply |
[production] |
| 10:05 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on puppetdb1002.eqiad.wmnet with reason: Disable puppetdb/postgres on old nodes to ensure nothing hits them anyway |
[production] |
| 10:05 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on puppetdb1002.eqiad.wmnet with reason: Disable puppetdb/postgres on old nodes to ensure nothing hits them anyway |
[production] |
| 10:02 |
<hnowlan> |
enabling puppet on A:cp |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 1%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52465 and previous config saved to /var/cache/conftool/dbconfig/20230912-100138-root.json |
[production] |
| 09:59 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors |
[production] |
| 09:59 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors |
[production] |
| 09:53 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors |
[production] |
| 09:52 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors |
[production] |
| 09:52 |
<jmm@cumin2002> |
START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki |
[production] |
| 09:32 |
<hnowlan> |
disabled puppet on A:cp |
[production] |
| 09:26 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3317 (T343198)', diff saved to https://phabricator.wikimedia.org/P52464 and previous config saved to /var/cache/conftool/dbconfig/20230912-092639-arnaudb.json |
[production] |
| 09:26 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.pki.restart-reboot (exit_code=99) rolling reboot on A:pki |
[production] |
| 09:26 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:26 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:26 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T343198)', diff saved to https://phabricator.wikimedia.org/P52463 and previous config saved to /var/cache/conftool/dbconfig/20230912-092618-arnaudb.json |
[production] |
| 09:26 |
<jmm@cumin2002> |
START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki |
[production] |
| 09:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2115.codfw.wmnet with reason: Maintenance |
[production] |
| 09:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2115.codfw.wmnet with reason: Maintenance |
[production] |
| 09:12 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:12 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:11 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P52461 and previous config saved to /var/cache/conftool/dbconfig/20230912-091112-arnaudb.json |
[production] |
| 08:58 |
<claime> |
Running puppet on cp-text P:trafficserver::backend - T341780 |
[production] |
| 08:58 |
<claime> |
Sending 5% of global traffic to mw-on-k8s - T341780 |
[production] |
| 08:56 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P52460 and previous config saved to /var/cache/conftool/dbconfig/20230912-085606-arnaudb.json |
[production] |
| 08:51 |
<claime> |
mw-api-ext, mw-web: Raise total replicas to 14 - T341780 |
[production] |
| 08:51 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-web: apply |
[production] |
| 08:51 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-web: apply |
[production] |
| 08:51 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 08:50 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 08:50 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:50 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:50 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:50 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:49 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:49 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 08:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1028.eqiad.wmnet |
[production] |
| 08:41 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T343198)', diff saved to https://phabricator.wikimedia.org/P52459 and previous config saved to /var/cache/conftool/dbconfig/20230912-084059-arnaudb.json |
[production] |
| 08:39 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.26 refs T343728 |
[production] |
| 08:38 |
<moritzm> |
rebalance Ganeti cluster in codfw/C following node replacement |
[production] |
| 08:24 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] (duration: 09m 16s) |
[production] |
| 08:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet |
[production] |
| 08:18 |
<oblivian@deploy1002> |
oblivian: Continuing with sync |
[production] |
| 08:17 |
<oblivian@deploy1002> |
oblivian: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 08:15 |
<oblivian@deploy1002> |
Started scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] |
[production] |
| 08:13 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] (duration: 45m 23s) |
[production] |