|
2021-05-20
§
|
| 10:15 |
<marostegui> |
Deploy schema change on s1 codfw, lag will appear in codfw T266486 T268392 T273360 |
[production] |
| 10:10 |
<jiji@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:10 |
<jiji@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P16122 and previous config saved to /var/cache/conftool/dbconfig/20210520-093510-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16121 and previous config saved to /var/cache/conftool/dbconfig/20210520-093257-root.json |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16120 and previous config saved to /var/cache/conftool/dbconfig/20210520-091754-root.json |
[production] |
| 09:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16119 and previous config saved to /var/cache/conftool/dbconfig/20210520-090250-root.json |
[production] |
| 08:56 |
<godog> |
move icinga-wm to libera.chat |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16118 and previous config saved to /var/cache/conftool/dbconfig/20210520-084746-root.json |
[production] |
| 07:44 |
<jiji@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:41 |
<jiji@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P16117 and previous config saved to /var/cache/conftool/dbconfig/20210520-071723-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16116 and previous config saved to /var/cache/conftool/dbconfig/20210520-071432-root.json |
[production] |
| 06:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16115 and previous config saved to /var/cache/conftool/dbconfig/20210520-065928-root.json |
[production] |
| 06:50 |
<ryankemper@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 |
[production] |
| 06:50 |
<ryankemper> |
T283223 Write queue not draining fast enough for the next node to reboot, will finish reboot tomorrow |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16114 and previous config saved to /var/cache/conftool/dbconfig/20210520-064425-root.json |
[production] |
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16113 and previous config saved to /var/cache/conftool/dbconfig/20210520-062921-root.json |
[production] |
| 06:25 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.6/includes/PageProps.php: Backport: [[gerrit:693028|PageProps: be prepared that PageIdentity is not proper title (T283170)]] (duration: 01m 06s) |
[production] |
| 06:08 |
<elukey> |
powercycle ms-be2035 - no ssh available, no metrics since hours ago, I/O errors registered in the main tty on serial console |
[production] |
| 05:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1141 (re)pooling @ 100%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16112 and previous config saved to /var/cache/conftool/dbconfig/20210520-054402-root.json |
[production] |
| 05:33 |
<ryankemper> |
T283223 `sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223` on `ryankemper@cumin1001` tmux session `restart_cloudelastic` |
[production] |
| 05:33 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1141 (re)pooling @ 75%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16111 and previous config saved to /var/cache/conftool/dbconfig/20210520-052859-root.json |
[production] |
| 05:27 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 |
[production] |
| 05:24 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 |
[production] |
| 05:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts labsdb1011.eqiad.wmnet |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1141 (re)pooling @ 50%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16110 and previous config saved to /var/cache/conftool/dbconfig/20210520-051355-root.json |
[production] |
| 05:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts labsdb1011.eqiad.wmnet |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1143', diff saved to https://phabricator.wikimedia.org/P16109 and previous config saved to /var/cache/conftool/dbconfig/20210520-050025-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1166', diff saved to https://phabricator.wikimedia.org/P16108 and previous config saved to /var/cache/conftool/dbconfig/20210520-045919-marostegui.json |
[production] |
| 04:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1141 (re)pooling @ 25%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16107 and previous config saved to /var/cache/conftool/dbconfig/20210520-045852-root.json |
[production] |
| 01:01 |
<mutante> |
signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192) |
[production] |
| 00:14 |
<ejegg> |
updated fundraising CiviCRM from b3fb3c9cb0 to 35f5afb1b4 |
[production] |
| 00:13 |
<ejegg> |
updated payments-wiki from 9f51ace546 to 6fac77f60e |
[production] |
|
2021-05-19
§
|
| 22:44 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~/uploads]$ sleep 3600 && mwscript importImages.php --wiki=commonswiki --comment-ext=txt --sleep=7200 --user=Lusccasdeutsch . # T278856 # 3 video files |
[production] |
| 22:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh2002.wikimedia.org |
[production] |
| 22:27 |
<Urbanecm> |
Start server-side upload for 1 video file (T283186) |
[production] |
| 22:25 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:22 |
<Urbanecm> |
Start server-side upload for 3 video file (T283102, T283054) |
[production] |
| 22:22 |
<razzi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:21 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 22:18 |
<razzi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:12 |
<urbanecm@deploy1002> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 14s) |
[production] |
| 22:11 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh2001.wikimedia.org |
[production] |
| 22:09 |
<urbanecm@deploy1002> |
update-interwiki-cache aborted: Update interwiki cache (duration: 00m 11s) |
[production] |
| 22:07 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host doh2002.wikimedia.org |
[production] |
| 22:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host doh2002.wikimedia.org |
[production] |
| 22:00 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host doh2002.wikimedia.org |
[production] |
| 21:58 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host doh2002.wikimedia.org |
[production] |