|
2022-01-18
§
|
| 02:54 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:48 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:23 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
|
2022-01-17
§
|
| 23:27 |
<jynus> |
forced session revocation on phab for a user T299315 |
[production] |
| 20:48 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics-test@27a4f7a]: (no justification provided) (duration: 00m 02s) |
[production] |
| 20:48 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics-test@27a4f7a]: (no justification provided) |
[production] |
| 18:47 |
<krinkle@deploy1002> |
Finished deploy [integration/docroot@1621c26]: (no justification provided) (duration: 01m 14s) |
[production] |
| 18:46 |
<krinkle@deploy1002> |
Started deploy [integration/docroot@1621c26]: (no justification provided) |
[production] |
| 16:30 |
<moritzm> |
installing python-virtualenv bugfix updates from bullseye 11.2 point release |
[production] |
| 16:21 |
<moritzm> |
installing wget bugfix updates from bullseye 11.2 point release |
[production] |
| 16:13 |
<moritzm> |
installing freeipmi bugfix updates from bullseye 11.2 point release |
[production] |
| 16:02 |
<moritzm> |
installing curl bugfix updates from bullseye 11.2 point release |
[production] |
| 15:54 |
<mutante> |
mw1414,mw1415,mw1416,mw1417,mw1418,mw1447,mw1448,mw1449,mw1450,mw1437,mw1438 (all canaries eqiad) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378) |
[production] |
| 15:46 |
<mutante> |
parse2001, parse2002, wtp1025, wtp1026 (all parsoid canaries - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378) |
[production] |
| 15:40 |
<mutante> |
mw2278, mw2279, mw2374, mw2376 (API and jobrunner canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378) |
[production] |
| 15:34 |
<mutante> |
mw2271, mw2272, mw2251, mw2252 (appserver and API canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378) |
[production] |
| 15:01 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-airflow1003.eqiad.wmnet |
[production] |
| 14:58 |
<btullis@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM an-airflow1003.eqiad.wmnet |
[production] |
| 14:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2132.codfw.wmnet with OS bullseye |
[production] |
| 14:50 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-airflow1002.eqiad.wmnet |
[production] |
| 14:48 |
<btullis@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM an-airflow1002.eqiad.wmnet |
[production] |
| 14:44 |
<moritzm> |
imported cassandra 3.11.11 to component/cassandradev for stretch-wikimedia and buster-wikimedia T298805 |
[production] |
| 14:41 |
<moritzm> |
systemctl reset-failed ifup@ens5.service on an-airflow1001 T273026 |
[production] |
| 14:39 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-airflow1001.eqiad.wmnet |
[production] |
| 14:37 |
<hnowlan> |
removing restbase2009 from cassandra configs |
[production] |
| 14:30 |
<btullis@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM an-airflow1001.eqiad.wmnet |
[production] |
| 14:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2132.codfw.wmnet with OS bullseye |
[production] |
| 14:15 |
<marostegui> |
Reimage db2132 to Bullseye T299344 |
[production] |
| 13:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove recentchanges group from s3 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18762 and previous config saved to /var/cache/conftool/dbconfig/20220117-134520-marostegui.json |
[production] |
| 12:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1151.eqiad.wmnet with OS bullseye |
[production] |
| 12:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1151.eqiad.wmnet with OS bullseye |
[production] |
| 12:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2142.codfw.wmnet with OS bullseye |
[production] |
| 11:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2142.codfw.wmnet with OS bullseye |
[production] |
| 11:30 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafkamon1002.eqiad.wmnet |
[production] |
| 11:26 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM kafkamon1002.eqiad.wmnet |
[production] |
| 11:08 |
<moritzm> |
switching kubetcd1006 to DRBD-backed storage (required for ganeti update) |
[production] |
| 11:03 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd1006.eqiad.wmnet with reason: switch to drbd storage |
[production] |
| 11:03 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd1006.eqiad.wmnet with reason: switch to drbd storage |
[production] |
| 11:00 |
<moritzm> |
systemctl reset-failed ifup@ens5.service on kubetcd1005 T273026 |
[production] |
| 10:56 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet |
[production] |
| 10:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove recentchangeslinked group from s3 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18761 and previous config saved to /var/cache/conftool/dbconfig/20220117-104801-marostegui.json |
[production] |
| 10:47 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet |
[production] |
| 10:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1152.eqiad.wmnet with OS bullseye |
[production] |
| 10:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18760 and previous config saved to /var/cache/conftool/dbconfig/20220117-104459-marostegui.json |
[production] |
| 10:44 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet |
[production] |