|
2023-05-17
§
|
| 06:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48269 and previous config saved to /var/cache/conftool/dbconfig/20230517-064313-root.json |
[production] |
| 06:40 |
<mvolz@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:39 |
<mvolz@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 06:39 |
<mvolz@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:38 |
<mvolz@deploy1002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 06:37 |
<mvolz@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:37 |
<mvolz@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48268 and previous config saved to /var/cache/conftool/dbconfig/20230517-062914-root.json |
[production] |
| 06:22 |
<mvolz@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:21 |
<mvolz@deploy1002> |
helmfile [eqiad] START helmfile.d/services/zotero: apply |
[production] |
| 06:20 |
<mvolz@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:20 |
<mvolz@deploy1002> |
helmfile [codfw] START helmfile.d/services/zotero: apply |
[production] |
| 06:19 |
<mvolz@deploy1002> |
helmfile [staging] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:18 |
<mvolz@deploy1002> |
helmfile [staging] START helmfile.d/services/zotero: apply |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48267 and previous config saved to /var/cache/conftool/dbconfig/20230517-061409-root.json |
[production] |
| 06:01 |
<volans> |
restarted ferm on ms-be1047 |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48265 and previous config saved to /var/cache/conftool/dbconfig/20230517-055904-root.json |
[production] |
| 05:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2096', diff saved to https://phabricator.wikimedia.org/P48264 and previous config saved to /var/cache/conftool/dbconfig/20230517-055310-root.json |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1115.eqiad.wmnet |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1115.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 05:48 |
<marostegui@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1115.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 05:46 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 05:41 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1115.eqiad.wmnet |
[production] |
| 05:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1112 from dbctl T336332', diff saved to https://phabricator.wikimedia.org/P48263 and previous config saved to /var/cache/conftool/dbconfig/20230517-052007-marostegui.json |
[production] |
| 05:16 |
<marostegui> |
Optimize s7 on dbstore1003 T336733 |
[production] |
| 00:21 |
<krinkle@deploy1002> |
Synchronized src/: I4cfa4a2474b4e (duration: 06m 01s) |
[production] |
| 00:15 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I4cfa4a2474b4e (duration: 06m 14s) |
[production] |
| 00:07 |
<krinkle@deploy1002> |
Synchronized lib/: I4cfa4a2474b4e (duration: 06m 51s) |
[production] |
|
2023-05-16
§
|
| 20:59 |
<jdrewniak@deploy1002> |
Finished scap: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] (duration: 07m 18s) |
[production] |
| 20:53 |
<jdrewniak@deploy1002> |
jdrewniak and matmarex: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:52 |
<jdrewniak@deploy1002> |
Started scap: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] |
[production] |
| 20:49 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.network.provision (exit_code=0) for device ssw1-a8-codfw.mgmt.codfw.wmnet |
[production] |
| 20:49 |
<jdrewniak@deploy1002> |
Finished scap: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] (duration: 09m 19s) |
[production] |
| 20:41 |
<jdrewniak@deploy1002> |
jdrewniak: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:39 |
<jdrewniak@deploy1002> |
Started scap: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] |
[production] |
| 20:36 |
<jdrewniak@deploy1002> |
Finished scap: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] (duration: 07m 44s) |
[production] |
| 20:30 |
<jdrewniak@deploy1002> |
jdrewniak: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:30 |
<brett> |
Rolling out maglev LVS scheduler in drmrs (for real this time) - T263797 |
[production] |
| 20:29 |
<jdrewniak@deploy1002> |
Started scap: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] |
[production] |
| 19:13 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:13 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" |
[production] |
| 19:12 |
<volans@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" |
[production] |
| 19:10 |
<volans@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:10 |
<volans@cumin2002> |
START - Cookbook sre.network.provision for device ssw1-a8-codfw.mgmt.codfw.wmnet |
[production] |
| 19:04 |
<sukhe> |
dummry run of authdns-update to confirm new hosts |
[production] |
| 19:00 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns2003.wikimedia.org |
[production] |
| 19:00 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:00 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" |
[production] |
| 18:59 |
<sukhe@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" |
[production] |