|
2022-03-22
ยง
|
| 11:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 11:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 11:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:56 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:46 |
<mmandere> |
pool cp1077 with HAProxy as TLS termination layer - T290005 |
[production] |
| 10:41 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS buster |
[production] |
| 10:26 |
<_joe_> |
running check-restart-php on api appservers |
[production] |
| 10:22 |
<_joe_> |
running check-and-restart on mw-eqiad-appservers |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T298557)', diff saved to https://phabricator.wikimedia.org/P22940 and previous config saved to /var/cache/conftool/dbconfig/20220322-101354-marostegui.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298557)', diff saved to https://phabricator.wikimedia.org/P22939 and previous config saved to /var/cache/conftool/dbconfig/20220322-101346-marostegui.json |
[production] |
| 10:03 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.3 refs T300203 |
[production] |
| 09:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22938 and previous config saved to /var/cache/conftool/dbconfig/20220322-095841-marostegui.json |
[production] |
| 09:54 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage |
[production] |
| 09:54 |
<jnuche@deploy1002> |
Finished scap: testwikis wikis to 1.39.0-wmf.3 refs T300203 (duration: 62m 07s) |
[production] |
| 09:51 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage |
[production] |
| 09:46 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cloudcontrol1005.wikimedia.org with reason: dcaro testing backups |
[production] |
| 09:46 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cloudcontrol1005.wikimedia.org with reason: dcaro testing backups |
[production] |
| 09:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22937 and previous config saved to /var/cache/conftool/dbconfig/20220322-094335-marostegui.json |
[production] |
| 09:34 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS buster |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298557)', diff saved to https://phabricator.wikimedia.org/P22936 and previous config saved to /var/cache/conftool/dbconfig/20220322-092830-marostegui.json |
[production] |
| 09:25 |
<mmandere> |
depool cp1077 for reimage - T290005 |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P22935 and previous config saved to /var/cache/conftool/dbconfig/20220322-091718-root.json |
[production] |
| 09:11 |
<dcausse> |
restarted blazegraph on wdqs2002 (deadlocked) |
[production] |
| 09:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P22934 and previous config saved to /var/cache/conftool/dbconfig/20220322-090214-root.json |
[production] |
| 08:59 |
<XioNoX> |
drmrs propagate LVS med to core routers |
[production] |
| 08:52 |
<jnuche@deploy1002> |
Started scap: testwikis wikis to 1.39.0-wmf.3 refs T300203 |
[production] |
| 08:49 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1008.eqiad.wmnet with OS bullseye |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P22933 and previous config saved to /var/cache/conftool/dbconfig/20220322-084710-root.json |
[production] |
| 08:37 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes1008.eqiad.wmnet with reason: host reimage |
[production] |
| 08:35 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1008.eqiad.wmnet with reason: host reimage |
[production] |
| 08:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P22932 and previous config saved to /var/cache/conftool/dbconfig/20220322-083206-root.json |
[production] |
| 08:19 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1008.eqiad.wmnet with OS bullseye |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3317 (T298557)', diff saved to https://phabricator.wikimedia.org/P22931 and previous config saved to /var/cache/conftool/dbconfig/20220322-081806-marostegui.json |
[production] |
| 08:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298557)', diff saved to https://phabricator.wikimedia.org/P22930 and previous config saved to /var/cache/conftool/dbconfig/20220322-081758-marostegui.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P22929 and previous config saved to /var/cache/conftool/dbconfig/20220322-081702-root.json |
[production] |
| 08:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1132 some more weight T301879', diff saved to https://phabricator.wikimedia.org/P22928 and previous config saved to /var/cache/conftool/dbconfig/20220322-080713-marostegui.json |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P22927 and previous config saved to /var/cache/conftool/dbconfig/20220322-080253-marostegui.json |
[production] |
| 07:57 |
<urbanecm> |
UTC morning backport window completed |
[production] |
| 07:57 |
<urbanecm@deploy1002> |
Synchronized php-1.39.0-wmf.2/extensions/GrowthExperiments/modules/ext.growthExperiments.MentorDashboard/MenteeOverview/MenteeOverviewPresets.js: 84877bd: MenteeOverviewPresets.getUsersToShow: Fix typo (T304353) (duration: 00m 49s) |
[production] |
| 07:53 |
<elukey> |
restart php-fpm on mw1449 - opcache full after deployment |
[production] |
| 07:49 |
<elukey> |
restart php-fpm on mw1448 - high cpu usage right after yesterday's deployment at 21 UTC |
[production] |
| 07:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P22925 and previous config saved to /var/cache/conftool/dbconfig/20220322-074748-marostegui.json |
[production] |
| 07:47 |
<elukey> |
depool mw1448 manually on the node (high cpu usage from php-fpm) |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298557)', diff saved to https://phabricator.wikimedia.org/P22924 and previous config saved to /var/cache/conftool/dbconfig/20220322-073243-marostegui.json |
[production] |
| 07:26 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 8151bf2: Allow flooders to remove the group from themselves in viwiki (T303578) (duration: 00m 50s) |
[production] |
| 07:21 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1007.eqiad.wmnet with OS bullseye |
[production] |