|
2022-01-18
ยง
|
| 16:45 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet |
[production] |
| 16:21 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase2010.codfw.wmnet with OS buster |
[production] |
| 16:14 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2010.codfw.wmnet with OS buster |
[production] |
| 16:13 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase2010.codfw.wmnet with OS buster |
[production] |
| 16:11 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2010.codfw.wmnet with OS buster |
[production] |
| 16:10 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 16:09 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 16:07 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase2010.codfw.wmnet with OS buster |
[production] |
| 16:03 |
<moritzm> |
installing xen security updates on buster (client-side libraries) |
[production] |
| 15:59 |
<hashar> |
Shutting down CI for maintenance on contint2001 # T283582 |
[production] |
| 15:54 |
<godog> |
update kartotherian certs on maps hosts and roll-reload nginx - T297604 |
[production] |
| 15:54 |
<moritzm> |
installing libssh2 security updates on stretch |
[production] |
| 15:50 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 09s) |
[production] |
| 15:50 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) |
[production] |
| 15:47 |
<andrewbogott> |
resizing the wikitech-static host for T298052 |
[production] |
| 15:45 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 02s) |
[production] |
| 15:45 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) |
[production] |
| 15:35 |
<godog> |
regenerate kartotherian certs via cergen - T297604 |
[production] |
| 14:33 |
<kormat> |
Deploying wmfmariadbpy 0.8 T299406 |
[production] |
| 14:33 |
<kormat> |
uploaded wmfmariadbpy 0.8 to apt.wm.o |
[production] |
| 14:31 |
<moritzm> |
installing rsync security updates on stretch |
[production] |
| 14:28 |
<moritzm> |
installing xorg-server security updates on stretch |
[production] |
| 14:10 |
<moritzm> |
installing vim security updates on stretch |
[production] |
| 14:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T285149)', diff saved to https://phabricator.wikimedia.org/P18780 and previous config saved to /var/cache/conftool/dbconfig/20220118-140540-marostegui.json |
[production] |
| 13:55 |
<XioNoX> |
update grafana-plugins on grafana hosts - T251184 |
[production] |
| 13:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P18779 and previous config saved to /var/cache/conftool/dbconfig/20220118-135036-marostegui.json |
[production] |
| 13:46 |
<XioNoX> |
add grafana-plugins 0.3 (with worldmap plugin) to reprepo |
[production] |
| 13:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P18778 and previous config saved to /var/cache/conftool/dbconfig/20220118-133531-marostegui.json |
[production] |
| 13:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 13:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 13:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 13:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 13:26 |
<Lucas_WMDE> |
UTC morning backport window done |
[production] |
| 13:24 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.38.0-wmf.17/extensions/GrowthExperiments/includes/HomepageHooks.php: Backport: [[gerrit:754605|Monitoring: Add '.Save' to distinguish from '.Click' events (T286366)]] (duration: 00m 54s) |
[production] |
| 13:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 13:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 13:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 13:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T285149)', diff saved to https://phabricator.wikimedia.org/P18777 and previous config saved to /var/cache/conftool/dbconfig/20220118-132026-marostegui.json |
[production] |
| 13:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 13:14 |
<moritzm> |
installing python-babel security updates on buster |
[production] |
| 13:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1168 (T285149)', diff saved to https://phabricator.wikimedia.org/P18776 and previous config saved to /var/cache/conftool/dbconfig/20220118-131215-marostegui.json |
[production] |
| 13:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T285149)', diff saved to https://phabricator.wikimedia.org/P18775 and previous config saved to /var/cache/conftool/dbconfig/20220118-131208-marostegui.json |
[production] |
| 13:06 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: update requirements - ayounsi@cumin1001 |
[production] |
| 13:05 |
<ayounsi@cumin1001> |
START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: update requirements - ayounsi@cumin1001 |
[production] |
| 13:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 13:04 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@0f02386]: update requirements (duration: 01m 27s) |
[production] |
| 13:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 13:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |