|
2023-03-14
§
|
| 06:41 |
<hashar> |
gerrit: changed `operations/puppet` merge strategy to allow "content merges" (see `ops` list for the rationale) |
[production] |
| 06:36 |
<denisse@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: centrallog1001 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" |
[production] |
| 06:34 |
<denisse@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:28 |
<denisse@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts centrallog1001 |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T329260)', diff saved to https://phabricator.wikimedia.org/P45836 and previous config saved to /var/cache/conftool/dbconfig/20230314-061633-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 06:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 06:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 06:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 05:07 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 05:05 |
<ryankemper@deploy2002> |
Finished deploy [wdqs/wdqs@61ef435]: 0.3.122 (duration: 08m 45s) |
[production] |
| 04:57 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.122` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 04:56 |
<ryankemper@deploy2002> |
Started deploy [wdqs/wdqs@61ef435]: 0.3.122 |
[production] |
| 04:56 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.122`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:55 |
<mwpresync@deploy2002> |
Pruned MediaWiki: 1.40.0-wmf.25 (duration: 02m 20s) |
[production] |
| 03:53 |
<mwpresync@deploy2002> |
Finished scap: testwikis wikis to 1.40.0-wmf.27 refs T330205 (duration: 51m 02s) |
[production] |
| 03:02 |
<mwpresync@deploy2002> |
Started scap: testwikis wikis to 1.40.0-wmf.27 refs T330205 |
[production] |
| 02:22 |
<legoktm> |
removed user's 2FA on wikitech for T331955 |
[production] |
| 02:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45835 and previous config saved to /var/cache/conftool/dbconfig/20230314-022023-marostegui.json |
[production] |
| 02:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P45834 and previous config saved to /var/cache/conftool/dbconfig/20230314-020517-marostegui.json |
[production] |
| 01:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P45833 and previous config saved to /var/cache/conftool/dbconfig/20230314-015011-marostegui.json |
[production] |
| 01:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45832 and previous config saved to /var/cache/conftool/dbconfig/20230314-013504-marostegui.json |
[production] |
| 01:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2176 (T329260)', diff saved to https://phabricator.wikimedia.org/P45831 and previous config saved to /var/cache/conftool/dbconfig/20230314-012442-marostegui.json |
[production] |
| 01:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 01:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 01:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45830 and previous config saved to /var/cache/conftool/dbconfig/20230314-012421-marostegui.json |
[production] |
| 01:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P45829 and previous config saved to /var/cache/conftool/dbconfig/20230314-010915-marostegui.json |
[production] |
| 00:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P45828 and previous config saved to /var/cache/conftool/dbconfig/20230314-005409-marostegui.json |
[production] |
| 00:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45827 and previous config saved to /var/cache/conftool/dbconfig/20230314-003903-marostegui.json |
[production] |
| 00:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2174 (T329260)', diff saved to https://phabricator.wikimedia.org/P45826 and previous config saved to /var/cache/conftool/dbconfig/20230314-002840-marostegui.json |
[production] |
| 00:28 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 00:28 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 00:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T329260)', diff saved to https://phabricator.wikimedia.org/P45825 and previous config saved to /var/cache/conftool/dbconfig/20230314-002819-marostegui.json |
[production] |
| 00:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P45824 and previous config saved to /var/cache/conftool/dbconfig/20230314-001313-marostegui.json |
[production] |
|
2023-03-13
§
|
| 23:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P45823 and previous config saved to /var/cache/conftool/dbconfig/20230313-235807-marostegui.json |
[production] |
| 23:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T329260)', diff saved to https://phabricator.wikimedia.org/P45822 and previous config saved to /var/cache/conftool/dbconfig/20230313-234301-marostegui.json |
[production] |
| 23:39 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1003.eqiad.wmnet |
[production] |
| 23:33 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host sessionstore1003.eqiad.wmnet |
[production] |
| 23:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2173 (T329260)', diff saved to https://phabricator.wikimedia.org/P45821 and previous config saved to /var/cache/conftool/dbconfig/20230313-233127-marostegui.json |
[production] |
| 23:31 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 23:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 23:31 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 23:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 23:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T329260)', diff saved to https://phabricator.wikimedia.org/P45820 and previous config saved to /var/cache/conftool/dbconfig/20230313-233050-marostegui.json |
[production] |
| 23:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P45819 and previous config saved to /var/cache/conftool/dbconfig/20230313-231544-marostegui.json |
[production] |
| 23:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P45818 and previous config saved to /var/cache/conftool/dbconfig/20230313-230038-marostegui.json |
[production] |
| 22:48 |
<zabe@deploy2002> |
Finished scap: [[gerrit:898037|noc: Switch default selection on db.php from eqiad to codfw]] (duration: 06m 56s) |
[production] |
| 22:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T329260)', diff saved to https://phabricator.wikimedia.org/P45817 and previous config saved to /var/cache/conftool/dbconfig/20230313-224532-marostegui.json |
[production] |