1551-1600 of 10000 results (51ms)
2021-05-20 ยง
15:44 <jiji@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
15:44 <jiji@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:43 <jiji@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
15:33 <moritzm> installing graphviz security updates on buster [production]
15:31 <ryankemper> [cloudelastic] `ryankemper@cloudelastic1003:~$ sudo systemctl restart *search*` to clear `Check systemd state` alert on `cloudelastic1003` [production]
15:30 <_joe_> test [production]
15:23 <moritzm> installing graphviz security updates on buster [production]
15:21 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
15:21 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
15:21 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
15:21 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
14:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1118', diff saved to https://phabricator.wikimedia.org/P16128 and previous config saved to /var/cache/conftool/dbconfig/20210520-143825-marostegui.json [production]
13:58 <hashar@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.6 (duration: 01m 05s) [production]
13:57 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.6 [production]
13:52 <hashar@deploy1002> Synchronized php-1.37.0-wmf.6/includes/upload/UploadFromStash.php: UploadFromStash: convert default user from false to null - T283196 (duration: 01m 05s) [production]
13:50 <hashar@deploy1002> Synchronized php-1.37.0-wmf.6/includes/user/ActorStore.php: ActorStore: avoid throwing in case of invalid usernames T283167 (duration: 01m 05s) [production]
13:41 <volans@deploy1002> Finished deploy [debmonitor/deploy@444b931]: Release v0.3.0 (duration: 01m 20s) [production]
13:39 <volans@deploy1002> Started deploy [debmonitor/deploy@444b931]: Release v0.3.0 [production]
12:30 <kormat> Deploying wmfmariadbpy 0.7 T283228 [production]
11:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16126 and previous config saved to /var/cache/conftool/dbconfig/20210520-113529-root.json [production]
11:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16125 and previous config saved to /var/cache/conftool/dbconfig/20210520-112026-root.json [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16124 and previous config saved to /var/cache/conftool/dbconfig/20210520-110522-root.json [production]
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16123 and previous config saved to /var/cache/conftool/dbconfig/20210520-105018-root.json [production]
10:15 <marostegui> Deploy schema change on s1 codfw, lag will appear in codfw T266486 T268392 T273360 [production]
10:10 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:10 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P16122 and previous config saved to /var/cache/conftool/dbconfig/20210520-093510-marostegui.json [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16121 and previous config saved to /var/cache/conftool/dbconfig/20210520-093257-root.json [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16120 and previous config saved to /var/cache/conftool/dbconfig/20210520-091754-root.json [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16119 and previous config saved to /var/cache/conftool/dbconfig/20210520-090250-root.json [production]
08:56 <godog> move icinga-wm to libera.chat [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16118 and previous config saved to /var/cache/conftool/dbconfig/20210520-084746-root.json [production]
07:44 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
07:41 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P16117 and previous config saved to /var/cache/conftool/dbconfig/20210520-071723-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16116 and previous config saved to /var/cache/conftool/dbconfig/20210520-071432-root.json [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16115 and previous config saved to /var/cache/conftool/dbconfig/20210520-065928-root.json [production]
06:50 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
06:50 <ryankemper> T283223 Write queue not draining fast enough for the next node to reboot, will finish reboot tomorrow [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16114 and previous config saved to /var/cache/conftool/dbconfig/20210520-064425-root.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16113 and previous config saved to /var/cache/conftool/dbconfig/20210520-062921-root.json [production]
06:25 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.6/includes/PageProps.php: Backport: [[gerrit:693028|PageProps: be prepared that PageIdentity is not proper title (T283170)]] (duration: 01m 06s) [production]
06:08 <elukey> powercycle ms-be2035 - no ssh available, no metrics since hours ago, I/O errors registered in the main tty on serial console [production]
05:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 100%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16112 and previous config saved to /var/cache/conftool/dbconfig/20210520-054402-root.json [production]
05:33 <ryankemper> T283223 `sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223` on `ryankemper@cumin1001` tmux session `restart_cloudelastic` [production]
05:33 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 75%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16111 and previous config saved to /var/cache/conftool/dbconfig/20210520-052859-root.json [production]
05:27 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:24 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts labsdb1011.eqiad.wmnet [production]