|
2023-06-15
§
|
| 08:15 |
<jelto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 08:13 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.13 refs T337527 |
[production] |
| 08:13 |
<jelto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 08:11 |
<jelto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 08:10 |
<jelto@deploy1002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 07:55 |
<elukey@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1001" |
[production] |
| 07:34 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49438 and previous config saved to /var/cache/conftool/dbconfig/20230615-073248-root.json |
[production] |
| 07:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage |
[production] |
| 07:24 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49437 and previous config saved to /var/cache/conftool/dbconfig/20230615-071744-root.json |
[production] |
| 07:11 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS buster |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49436 and previous config saved to /var/cache/conftool/dbconfig/20230615-070239-root.json |
[production] |
| 06:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49435 and previous config saved to /var/cache/conftool/dbconfig/20230615-064734-root.json |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49434 and previous config saved to /var/cache/conftool/dbconfig/20230615-063230-root.json |
[production] |
| 06:31 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox interface ID 2066 |
[production] |
| 06:31 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.debug for Netbox interface ID 2066 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49433 and previous config saved to /var/cache/conftool/dbconfig/20230615-061725-root.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 2%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49432 and previous config saved to /var/cache/conftool/dbconfig/20230615-060220-root.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49431 and previous config saved to /var/cache/conftool/dbconfig/20230615-054716-root.json |
[production] |
| 05:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1168 to upgrade to 10.6.14 T338918', diff saved to https://phabricator.wikimedia.org/P49430 and previous config saved to /var/cache/conftool/dbconfig/20230615-053318-root.json |
[production] |
|
2023-06-14
§
|
| 23:38 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe |
[production] |
| 23:30 |
<mvernon@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe |
[production] |
| 21:38 |
<mutante> |
phabricator - made dancy (https://phabricator.wikimedia.org/people/manage/25411/) and administrator (T339174) |
[production] |
| 21:02 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] (duration: 10m 44s) |
[production] |
| 20:54 |
<taavi@deploy1002> |
arlolra and taavi: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:52 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:930249|Fix thumb styling on file description page (T337804)]] |
[production] |
| 20:34 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 20:34 |
<aokoth@cumin1001> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 20:29 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] (duration: 10m 23s) |
[production] |
| 20:21 |
<taavi@deploy1002> |
taavi and albertoleoncio: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:19 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:929742|Enable mobile page tabs for everyone in ptwikisource. (T338974)]] |
[production] |
| 20:12 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:930206|simplewiki: Remove "changetags" from registered user (T339124)]] (duration: 08m 55s) |
[production] |
| 20:10 |
<mutante> |
https://ticket.wikimedia.org down for migration |
[production] |
| 20:06 |
<taavi@deploy1002> |
taavi and stang: Backport for [[gerrit:930206|simplewiki: Remove "changetags" from registered user (T339124)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:03 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:930206|simplewiki: Remove "changetags" from registered user (T339124)]] |
[production] |
| 20:01 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 20:01 |
<aokoth@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on otrs1001.eqiad.wmnet with reason: Replacing Host |
[production] |
| 18:29 |
<milimetric@deploy1002> |
Finished deploy [airflow-dags/analytics@3d6caed]: Deploying mostly to rerun druid loading for mediawiki history reduced (duration: 00m 09s) |
[production] |
| 18:29 |
<milimetric@deploy1002> |
Started deploy [airflow-dags/analytics@3d6caed]: Deploying mostly to rerun druid loading for mediawiki history reduced |
[production] |
| 18:11 |
<moritzm> |
installing libssh security updates on buster |
[production] |
| 17:04 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply |
[production] |
| 17:03 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-main: apply |
[production] |
| 17:03 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply |
[production] |
| 17:02 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-main: apply |
[production] |
| 17:01 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-main: apply |
[production] |
| 17:01 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-main: apply |
[production] |
| 16:52 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS bullseye |
[production] |
| 16:51 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1492.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 16:48 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host mw1492.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |