|
2023-06-15
§
|
| 09:01 |
<fabfur> |
reboot cp5018 and cp5026 for kernel upgrade (T335835) |
[production] |
| 09:01 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp5018.eqsin.wmnet |
[production] |
| 09:01 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp5026.eqsin.wmnet |
[production] |
| 09:00 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1001.eqiad.wmnet |
[production] |
| 09:00 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw1492.eqiad.wmnet |
[production] |
| 09:00 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for mw1492.eqiad.wmnet |
[production] |
| 08:59 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw1492.eqiad.wmnet |
[production] |
| 08:59 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for mw1492.eqiad.wmnet |
[production] |
| 08:54 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host krb1001.eqiad.wmnet |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS buster |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1001" |
[production] |
| 08:31 |
<fabfur@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp5025.eqsin.wmnet |
[production] |
| 08:31 |
<fabfur@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp5017.eqsin.wmnet |
[production] |
| 08:20 |
<fabfur> |
reboot cp5017 and cp5025 for kernel upgrade (T335835) |
[production] |
| 08:20 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp5017.eqsin.wmnet |
[production] |
| 08:20 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp5025.eqsin.wmnet |
[production] |
| 08:15 |
<jelto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 08:13 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.13 refs T337527 |
[production] |
| 08:13 |
<jelto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 08:11 |
<jelto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 08:10 |
<jelto@deploy1002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 07:55 |
<elukey@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1001" |
[production] |
| 07:34 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49438 and previous config saved to /var/cache/conftool/dbconfig/20230615-073248-root.json |
[production] |
| 07:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage |
[production] |
| 07:24 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49437 and previous config saved to /var/cache/conftool/dbconfig/20230615-071744-root.json |
[production] |
| 07:11 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS buster |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49436 and previous config saved to /var/cache/conftool/dbconfig/20230615-070239-root.json |
[production] |
| 06:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49435 and previous config saved to /var/cache/conftool/dbconfig/20230615-064734-root.json |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49434 and previous config saved to /var/cache/conftool/dbconfig/20230615-063230-root.json |
[production] |
| 06:31 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox interface ID 2066 |
[production] |
| 06:31 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.debug for Netbox interface ID 2066 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49433 and previous config saved to /var/cache/conftool/dbconfig/20230615-061725-root.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 2%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49432 and previous config saved to /var/cache/conftool/dbconfig/20230615-060220-root.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49431 and previous config saved to /var/cache/conftool/dbconfig/20230615-054716-root.json |
[production] |
| 05:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1168 to upgrade to 10.6.14 T338918', diff saved to https://phabricator.wikimedia.org/P49430 and previous config saved to /var/cache/conftool/dbconfig/20230615-053318-root.json |
[production] |
|
2023-06-14
§
|
| 23:38 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe |
[production] |
| 23:30 |
<mvernon@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe |
[production] |
| 21:38 |
<mutante> |
phabricator - made dancy (https://phabricator.wikimedia.org/people/manage/25411/) and administrator (T339174) |
[production] |
| 21:02 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] (duration: 10m 44s) |
[production] |
| 20:54 |
<taavi@deploy1002> |
arlolra and taavi: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:52 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] |
[production] |
| 20:34 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 20:34 |
<aokoth@cumin1001> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 20:29 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] (duration: 10m 23s) |
[production] |
| 20:21 |
<taavi@deploy1002> |
taavi and albertoleoncio: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:19 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] |
[production] |
| 20:12 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:930206|simplewiki: Remove "changetags" from registered user (T339124)]] (duration: 08m 55s) |
[production] |
| 20:10 |
<mutante> |
https://ticket.wikimedia.org down for migration |
[production] |