|
2022-02-03
ยง
|
| 19:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P20133 and previous config saved to /var/cache/conftool/dbconfig/20220203-194712-marostegui.json |
[production] |
| 19:45 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster |
[production] |
| 19:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:41 |
<cmjohnson@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster |
[production] |
| 19:40 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:40 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster |
[production] |
| 19:39 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:39 |
<cmjohnson@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster |
[production] |
| 19:35 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster |
[production] |
| 19:34 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.20/skins/Vector/includes/Hooks.php: Backport: [[gerrit:759310|Pass skin name to Hooks::isSkinLegacy (T299971)]] (duration: 00m 49s) |
[production] |
| 19:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:33 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.20/extensions/ContentTranslation/modules/entrypoints/ext.cx.entrypoints.contributionsmenu.js: Backport: [[gerrit:759314|Update skin checks with new vector skin key. (T298916 T300814)]] (duration: 00m 50s) |
[production] |
| 19:33 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster |
[production] |
| 19:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T300402)', diff saved to https://phabricator.wikimedia.org/P20132 and previous config saved to /var/cache/conftool/dbconfig/20220203-193208-marostegui.json |
[production] |
| 19:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:29 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.20/extensions/WikiEditor/modules/ext.wikiEditor.js: Backport: [[gerrit:759316|New bucket for abtest data (T291308)]] (2/2) (duration: 00m 50s) |
[production] |
| 19:28 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.20/extensions/WikiEditor/includes/Hooks.php: Backport: [[gerrit:759316|New bucket for abtest data (T291308)]] (1/2) (duration: 00m 49s) |
[production] |
| 19:27 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: Backport: [[gerrit:759315|New bucket for abtest data (T291308)]] (duration: 00m 50s) |
[production] |
| 19:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:26 |
<taavi@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:759557|commonswiki: Add three domains to the wgCopyUploadsDomains allowlist (T299835 T300848)]] (duration: 00m 54s) |
[production] |
| 19:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 18:46 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:42 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1167 (T300402)', diff saved to https://phabricator.wikimedia.org/P20131 and previous config saved to /var/cache/conftool/dbconfig/20220203-183648-marostegui.json |
[production] |
| 18:36 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:36 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:36 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:36 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20130 and previous config saved to /var/cache/conftool/dbconfig/20220203-183634-marostegui.json |
[production] |
| 18:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P20129 and previous config saved to /var/cache/conftool/dbconfig/20220203-182129-marostegui.json |
[production] |
| 18:17 |
<dancy> |
restarted php7.2-fpm processes on mediawiki12 |
[production] |
| 18:10 |
<dancy> |
killed 8 spinning php7.2-fpm processes on mediawiki12 |
[production] |
| 18:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P20128 and previous config saved to /var/cache/conftool/dbconfig/20220203-180624-marostegui.json |
[production] |
| 17:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20127 and previous config saved to /var/cache/conftool/dbconfig/20220203-175120-marostegui.json |
[production] |
| 17:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20126 and previous config saved to /var/cache/conftool/dbconfig/20220203-174913-marostegui.json |
[production] |
| 17:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1114.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1114.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1111 (T300402)', diff saved to https://phabricator.wikimedia.org/P20125 and previous config saved to /var/cache/conftool/dbconfig/20220203-174905-marostegui.json |
[production] |
| 17:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P20122 and previous config saved to /var/cache/conftool/dbconfig/20220203-173400-marostegui.json |
[production] |
| 17:22 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts restbase2011.codfw.wmnet |
[production] |
| 17:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P20120 and previous config saved to /var/cache/conftool/dbconfig/20220203-171856-marostegui.json |
[production] |