|
2023-03-14
§
|
| 07:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T329260)', diff saved to https://phabricator.wikimedia.org/P45840 and previous config saved to /var/cache/conftool/dbconfig/20230314-073149-marostegui.json |
[production] |
| 07:25 |
<marostegui> |
Migrate db1183 to mariadb m5 eqiad dbmaint 10.6 T322294 |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P45839 and previous config saved to /var/cache/conftool/dbconfig/20230314-071643-marostegui.json |
[production] |
| 07:13 |
<marostegui> |
Migrate db2135 to mariadb m5 codfw dbmaint 10.6 |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P45838 and previous config saved to /var/cache/conftool/dbconfig/20230314-070137-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T329260)', diff saved to https://phabricator.wikimedia.org/P45837 and previous config saved to /var/cache/conftool/dbconfig/20230314-064630-marostegui.json |
[production] |
| 06:42 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts centrallog1001 |
[production] |
| 06:42 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 06:42 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: centrallog1001 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" |
[production] |
| 06:41 |
<hashar> |
gerrit: changed `operations/puppet` merge strategy to allow "content merges" (see `ops` list for the rationale) |
[production] |
| 06:36 |
<denisse@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: centrallog1001 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" |
[production] |
| 06:34 |
<denisse@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:28 |
<denisse@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts centrallog1001 |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T329260)', diff saved to https://phabricator.wikimedia.org/P45836 and previous config saved to /var/cache/conftool/dbconfig/20230314-061633-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 06:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 06:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 06:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 05:05 |
<ryankemper@deploy2002> |
Finished deploy [wdqs/wdqs@61ef435]: 0.3.122 (duration: 08m 45s) |
[production] |
| 04:57 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.122` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 04:56 |
<ryankemper@deploy2002> |
Started deploy [wdqs/wdqs@61ef435]: 0.3.122 |
[production] |
| 04:56 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.122`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:55 |
<mwpresync@deploy2002> |
Pruned MediaWiki: 1.40.0-wmf.25 (duration: 02m 20s) |
[production] |
| 03:53 |
<mwpresync@deploy2002> |
Finished scap: testwikis wikis to 1.40.0-wmf.27 refs T330205 (duration: 51m 02s) |
[production] |
| 03:02 |
<mwpresync@deploy2002> |
Started scap: testwikis wikis to 1.40.0-wmf.27 refs T330205 |
[production] |
| 02:22 |
<legoktm> |
removed user's 2FA on wikitech for T331955 |
[production] |
| 02:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45835 and previous config saved to /var/cache/conftool/dbconfig/20230314-022023-marostegui.json |
[production] |
| 02:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P45834 and previous config saved to /var/cache/conftool/dbconfig/20230314-020517-marostegui.json |
[production] |
| 01:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P45833 and previous config saved to /var/cache/conftool/dbconfig/20230314-015011-marostegui.json |
[production] |
| 01:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45832 and previous config saved to /var/cache/conftool/dbconfig/20230314-013504-marostegui.json |
[production] |
| 01:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45831 and previous config saved to /var/cache/conftool/dbconfig/20230314-012442-marostegui.json |
[production] |
| 01:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 01:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 01:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45830 and previous config saved to /var/cache/conftool/dbconfig/20230314-012421-marostegui.json |
[production] |
| 01:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P45829 and previous config saved to /var/cache/conftool/dbconfig/20230314-010915-marostegui.json |
[production] |
| 00:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P45828 and previous config saved to /var/cache/conftool/dbconfig/20230314-005409-marostegui.json |
[production] |
| 00:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45827 and previous config saved to /var/cache/conftool/dbconfig/20230314-003903-marostegui.json |
[production] |
| 00:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45826 and previous config saved to /var/cache/conftool/dbconfig/20230314-002840-marostegui.json |
[production] |
| 00:28 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 00:28 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 00:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T329260)', diff saved to https://phabricator.wikimedia.org/P45825 and previous config saved to /var/cache/conftool/dbconfig/20230314-002819-marostegui.json |
[production] |
| 00:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P45824 and previous config saved to /var/cache/conftool/dbconfig/20230314-001313-marostegui.json |
[production] |