|
2023-05-17
ยง
|
| 08:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host krb1001.eqiad.wmnet |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 25%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48284 and previous config saved to /var/cache/conftool/dbconfig/20230517-081341-root.json |
[production] |
| 08:08 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 08:08 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 08:05 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 08:04 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 10%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48283 and previous config saved to /var/cache/conftool/dbconfig/20230517-075836-root.json |
[production] |
| 07:57 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 07:57 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 07:48 |
<moritzm> |
upgrading krb1001 to Bullseye T331695 |
[production] |
| 07:45 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1001.eqiad.wmnet with reason: Update to Bullseye |
[production] |
| 07:45 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on krb1001.eqiad.wmnet with reason: Update to Bullseye |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48278 and previous config saved to /var/cache/conftool/dbconfig/20230517-074332-root.json |
[production] |
| 07:36 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 37468 |
[production] |
| 07:35 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'clear' for AS: 37468 |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 4%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48277 and previous config saved to /var/cache/conftool/dbconfig/20230517-072827-root.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 for decommissioning', diff saved to https://phabricator.wikimedia.org/P48276 and previous config saved to /var/cache/conftool/dbconfig/20230517-072508-root.json |
[production] |
| 07:19 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:920625|Revert "Enable the new Special:Contribute page entry point for desktop on selected wikis"]] (duration: 07m 22s) |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48275 and previous config saved to /var/cache/conftool/dbconfig/20230517-071428-root.json |
[production] |
| 07:13 |
<kartik@deploy1002> |
trainbranchbot and kartik: Backport for [[gerrit:920625|Revert "Enable the new Special:Contribute page entry point for desktop on selected wikis"]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 07:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 3%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48274 and previous config saved to /var/cache/conftool/dbconfig/20230517-071322-root.json |
[production] |
| 07:11 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:920625|Revert "Enable the new Special:Contribute page entry point for desktop on selected wikis"]] |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1121 T336725', diff saved to https://phabricator.wikimedia.org/P48273 and previous config saved to /var/cache/conftool/dbconfig/20230517-071039-root.json |
[production] |
| 07:09 |
<kartik@deploy1002> |
Backport cancelled. |
[production] |
| 06:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48272 and previous config saved to /var/cache/conftool/dbconfig/20230517-065923-root.json |
[production] |
| 06:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 2%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48271 and previous config saved to /var/cache/conftool/dbconfig/20230517-065817-root.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48270 and previous config saved to /var/cache/conftool/dbconfig/20230517-064419-root.json |
[production] |
| 06:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48269 and previous config saved to /var/cache/conftool/dbconfig/20230517-064313-root.json |
[production] |
| 06:40 |
<mvolz@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:39 |
<mvolz@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 06:39 |
<mvolz@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:38 |
<mvolz@deploy1002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 06:37 |
<mvolz@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 06:37 |
<mvolz@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48268 and previous config saved to /var/cache/conftool/dbconfig/20230517-062914-root.json |
[production] |
| 06:22 |
<mvolz@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:21 |
<mvolz@deploy1002> |
helmfile [eqiad] START helmfile.d/services/zotero: apply |
[production] |
| 06:20 |
<mvolz@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:20 |
<mvolz@deploy1002> |
helmfile [codfw] START helmfile.d/services/zotero: apply |
[production] |
| 06:19 |
<mvolz@deploy1002> |
helmfile [staging] DONE helmfile.d/services/zotero: apply |
[production] |
| 06:18 |
<mvolz@deploy1002> |
helmfile [staging] START helmfile.d/services/zotero: apply |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48267 and previous config saved to /var/cache/conftool/dbconfig/20230517-061409-root.json |
[production] |
| 06:01 |
<volans> |
restarted ferm on ms-be1047 |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2096 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48265 and previous config saved to /var/cache/conftool/dbconfig/20230517-055904-root.json |
[production] |
| 05:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2096', diff saved to https://phabricator.wikimedia.org/P48264 and previous config saved to /var/cache/conftool/dbconfig/20230517-055310-root.json |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1115.eqiad.wmnet |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 05:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1115.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 05:48 |
<marostegui@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1115.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 05:46 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |