|
2021-12-02
§
|
| 02:15 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 02:14 |
<andrew@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 01:52 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 01:21 |
<ryankemper> |
T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared |
[production] |
| 01:17 |
<wm-bot> |
Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
| 01:16 |
<ryankemper> |
T280001 Pooled `wcqs200[1-3]` (had been left unpooled from when we last removed wcqs from production) |
[production] |
| 01:12 |
<ryankemper> |
T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'` |
[production] |
| 01:12 |
<ryankemper> |
T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'` |
[production] |
| 01:11 |
<ryankemper> |
T280001 Waited 120s and checked https://icinga.wikimedia.org/alerts, proceeding to primary low-traffic hosts `lvs2009` and `lvs1015` |
[production] |
| 01:08 |
<ryankemper> |
T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good (for ex `lvs1016` has `TCP 10.2.2.67:443 wrr`) |
[production] |
| 01:07 |
<ryankemper> |
T280001 Restarting pybal on low-traffic backups: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'` |
[production] |
| 01:02 |
<ryankemper> |
T280001 `ryankemper@cumin1001:~$ sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'` |
[production] |
| 01:01 |
<ryankemper> |
T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/742841 |
[production] |
| 01:00 |
<ryankemper> |
T280001 About to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/742841 to bring `wcqs` into state `lvs_setup`, after which I'll perform a rolling restart of pybal |
[production] |
| 00:56 |
<wm-bot> |
Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:56 |
<wm-bot> |
Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:56 |
<wm-bot> |
Draining 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
| 00:50 |
<wm-bot> |
Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:50 |
<wm-bot> |
Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:50 |
<wm-bot> |
Draining 'cloudvirt1026.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
| 00:28 |
<wm-bot> |
Drained 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
| 00:24 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.9/skins/Vector/: a7586cd4a2559248ea1fd29cf74de535de016501: Update scroll observer to allow event logging (T292586) (duration: 00m 57s) |
[production] |
| 00:03 |
<wm-bot> |
Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:02 |
<wm-bot> |
Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 00:02 |
<wm-bot> |
Draining 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
|
2021-12-01
§
|
| 23:59 |
<wm-bot> |
Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster |
[admin] |
| 23:59 |
<wm-bot> |
Draining 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster |
[admin] |
| 23:54 |
<andrewbogott> |
*correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around T296790 and T296792 |
[admin] |
| 23:53 |
<andrewbogott> |
adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around T296790 and T296792 |
[admin] |
| 22:15 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 07s) |
[production] |
| 22:15 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:13 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 07s) |
[production] |
| 22:13 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:12 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 07s) |
[production] |
| 22:12 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:12 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 01m 23s) |
[production] |
| 22:11 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:10 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 03s) |
[production] |
| 22:10 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:10 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 03s) |
[production] |
| 22:10 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 22:09 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 03s) |
[production] |
| 22:09 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 21:25 |
<bd808> |
Migrated from python2.5 to python3.9! |
[tools.tool-db-usage] |
| 21:12 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 03s) |
[production] |
| 21:12 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 21:11 |
<otto@deploy1002> |
Finished deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) (duration: 00m 16s) |
[production] |
| 21:10 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257] (hadoop-test): (no justification provided) |
[production] |
| 21:10 |
<otto@deploy1002> |
Started deploy [airflow-dags/analytics@2f59257]: (no justification provided) |
[production] |
| 21:09 |
<razzi@deploy1002> |
Finished deploy [analytics/refinery@3b1b794]: Regular analytics weekly train [analytics/refinery@3b1b794] (duration: 21m 18s) |
[production] |