|
2022-01-12
ยง
|
| 21:19 |
<ryankemper> |
[WDQS] T299098 depooled `wdqs2003` so dc-ops can take a look at the PS2 failure |
[production] |
| 21:18 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@988b7d2] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@988b7d2] (duration: 06m 57s) |
[production] |
| 21:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P18707 and previous config saved to /var/cache/conftool/dbconfig/20220112-211248-marostegui.json |
[production] |
| 21:11 |
<joal@deploy1002> |
Started deploy [analytics/refinery@988b7d2] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@988b7d2] |
[production] |
| 21:11 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@988b7d2] (thin): Regular analytics weekly train THIN [analytics/refinery@988b7d2] (duration: 00m 07s) |
[production] |
| 21:11 |
<joal@deploy1002> |
Started deploy [analytics/refinery@988b7d2] (thin): Regular analytics weekly train THIN [analytics/refinery@988b7d2] |
[production] |
| 21:10 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@988b7d2]: Regular analytics weekly train [analytics/refinery@988b7d2] (duration: 24m 20s) |
[production] |
| 20:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164 (T297191)', diff saved to https://phabricator.wikimedia.org/P18706 and previous config saved to /var/cache/conftool/dbconfig/20220112-205744-marostegui.json |
[production] |
| 20:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1164 (T297191)', diff saved to https://phabricator.wikimedia.org/P18705 and previous config saved to /var/cache/conftool/dbconfig/20220112-205636-marostegui.json |
[production] |
| 20:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:56 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T297191)', diff saved to https://phabricator.wikimedia.org/P18704 and previous config saved to /var/cache/conftool/dbconfig/20220112-205629-marostegui.json |
[production] |
| 20:46 |
<joal@deploy1002> |
Started deploy [analytics/refinery@988b7d2]: Regular analytics weekly train [analytics/refinery@988b7d2] |
[production] |
| 20:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18703 and previous config saved to /var/cache/conftool/dbconfig/20220112-204124-marostegui.json |
[production] |
| 20:36 |
<dduvall> |
1.38.0-wmf.17 rolled back from group1 due to large spike in db read-only errors and slow queries (T293958) |
[production] |
| 20:33 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.38.0-wmf.17 |
[production] |
| 20:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18702 and previous config saved to /var/cache/conftool/dbconfig/20220112-202619-marostegui.json |
[production] |
| 20:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:21 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.38.0-wmf.17 refs T293958 (duration: 01m 21s) |
[production] |
| 20:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:19 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.17 refs T293958 |
[production] |
| 20:19 |
<jgleeson> |
updated payments from 939cb4bc to 8497eae9 |
[production] |
| 20:17 |
<mutante> |
applying firewall change on phabricator (VCS, git-ssh), second attempt, first codfw-only |
[production] |
| 20:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T297191)', diff saved to https://phabricator.wikimedia.org/P18701 and previous config saved to /var/cache/conftool/dbconfig/20220112-201114-marostegui.json |
[production] |
| 20:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3311 (T297191)', diff saved to https://phabricator.wikimedia.org/P18700 and previous config saved to /var/cache/conftool/dbconfig/20220112-200806-marostegui.json |
[production] |
| 20:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T297191)', diff saved to https://phabricator.wikimedia.org/P18699 and previous config saved to /var/cache/conftool/dbconfig/20220112-200759-marostegui.json |
[production] |
| 19:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P18698 and previous config saved to /var/cache/conftool/dbconfig/20220112-195254-marostegui.json |
[production] |
| 19:52 |
<hashar> |
Restarting CI Jenkins once more to apply the Gearman plugin update T298691 |
[production] |
| 19:44 |
<hashar> |
Clearing /srv partition on integration-castor03 |
[production] |
| 19:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P18697 and previous config saved to /var/cache/conftool/dbconfig/20220112-193749-marostegui.json |
[production] |
| 19:34 |
<hashar> |
Upgrading CI Jenkins and Gearman plugin T298691 |
[production] |
| 19:29 |
<mutante> |
wdqs2003 - one power supply failed so it's not redundant anymore, says Icinga |
[production] |
| 19:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 19:25 |
<cwhite> |
begin eqiad opensearch upgrade T288621 |
[production] |
| 19:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T297191)', diff saved to https://phabricator.wikimedia.org/P18696 and previous config saved to /var/cache/conftool/dbconfig/20220112-192244-marostegui.json |
[production] |
| 19:22 |
<mutante> |
deneb - for some reason the "package builder clean up build directory"-service fails T287222 |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 19:21 |
<cjming> |
end of UTC evening backport & config window |
[production] |
| 19:21 |
<mutante> |
[deneb:~] $ sudo systemctl start package_builder_Clean_up_build_directory.service |
[production] |