3901-3950 of 10000 results (49ms)
2022-02-03 ยง
20:07 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/skins/Vector/includes/Hooks.php: Backport: [[gerrit:759313|Drop skin override (T300814)]] (2/2) (duration: 00m 49s) [production]
20:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:06 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/skins/Vector/skin.json: Backport: [[gerrit:759313|Drop skin override (T300814)]] (1/2) (duration: 00m 49s) [production]
20:05 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
20:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P20134 and previous config saved to /var/cache/conftool/dbconfig/20220203-200217-marostegui.json [production]
19:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P20133 and previous config saved to /var/cache/conftool/dbconfig/20220203-194712-marostegui.json [production]
19:45 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
19:41 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:41 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
19:40 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:40 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
19:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:39 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
19:35 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
19:34 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/skins/Vector/includes/Hooks.php: Backport: [[gerrit:759310|Pass skin name to Hooks::isSkinLegacy (T299971)]] (duration: 00m 49s) [production]
19:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:33 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/extensions/ContentTranslation/modules/entrypoints/ext.cx.entrypoints.contributionsmenu.js: Backport: [[gerrit:759314|Update skin checks with new vector skin key. (T298916 T300814)]] (duration: 00m 50s) [production]
19:33 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
19:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T300402)', diff saved to https://phabricator.wikimedia.org/P20132 and previous config saved to /var/cache/conftool/dbconfig/20220203-193208-marostegui.json [production]
19:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:29 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/extensions/WikiEditor/modules/ext.wikiEditor.js: Backport: [[gerrit:759316|New bucket for abtest data (T291308)]] (2/2) (duration: 00m 50s) [production]
19:28 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/extensions/WikiEditor/includes/Hooks.php: Backport: [[gerrit:759316|New bucket for abtest data (T291308)]] (1/2) (duration: 00m 49s) [production]
19:27 <taavi@deploy1002> Synchronized php-1.38.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: Backport: [[gerrit:759315|New bucket for abtest data (T291308)]] (duration: 00m 50s) [production]
19:26 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:26 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:759557|commonswiki: Add three domains to the wgCopyUploadsDomains allowlist (T299835 T300848)]] (duration: 00m 54s) [production]
19:25 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:16 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:46 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:42 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1167 (T300402)', diff saved to https://phabricator.wikimedia.org/P20131 and previous config saved to /var/cache/conftool/dbconfig/20220203-183648-marostegui.json [production]
18:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
18:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
18:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
18:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
18:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20130 and previous config saved to /var/cache/conftool/dbconfig/20220203-183634-marostegui.json [production]
18:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P20129 and previous config saved to /var/cache/conftool/dbconfig/20220203-182129-marostegui.json [production]
18:17 <dancy> restarted php7.2-fpm processes on mediawiki12 [production]
18:10 <dancy> killed 8 spinning php7.2-fpm processes on mediawiki12 [production]
18:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P20128 and previous config saved to /var/cache/conftool/dbconfig/20220203-180624-marostegui.json [production]
17:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20127 and previous config saved to /var/cache/conftool/dbconfig/20220203-175120-marostegui.json [production]
17:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1114 (T300402)', diff saved to https://phabricator.wikimedia.org/P20126 and previous config saved to /var/cache/conftool/dbconfig/20220203-174913-marostegui.json [production]
17:49 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1114.eqiad.wmnet with reason: Maintenance [production]