1951-2000 of 10000 results (37ms)
2021-05-20 §
12:30 <kormat> Deploying wmfmariadbpy 0.7 T283228 [production]
11:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16126 and previous config saved to /var/cache/conftool/dbconfig/20210520-113529-root.json [production]
11:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16125 and previous config saved to /var/cache/conftool/dbconfig/20210520-112026-root.json [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16124 and previous config saved to /var/cache/conftool/dbconfig/20210520-110522-root.json [production]
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: Repool db1112', diff saved to https://phabricator.wikimedia.org/P16123 and previous config saved to /var/cache/conftool/dbconfig/20210520-105018-root.json [production]
10:15 <marostegui> Deploy schema change on s1 codfw, lag will appear in codfw T266486 T268392 T273360 [production]
10:10 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:10 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P16122 and previous config saved to /var/cache/conftool/dbconfig/20210520-093510-marostegui.json [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16121 and previous config saved to /var/cache/conftool/dbconfig/20210520-093257-root.json [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16120 and previous config saved to /var/cache/conftool/dbconfig/20210520-091754-root.json [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16119 and previous config saved to /var/cache/conftool/dbconfig/20210520-090250-root.json [production]
08:56 <godog> move icinga-wm to libera.chat [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repool db1179', diff saved to https://phabricator.wikimedia.org/P16118 and previous config saved to /var/cache/conftool/dbconfig/20210520-084746-root.json [production]
07:44 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
07:41 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P16117 and previous config saved to /var/cache/conftool/dbconfig/20210520-071723-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16116 and previous config saved to /var/cache/conftool/dbconfig/20210520-071432-root.json [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16115 and previous config saved to /var/cache/conftool/dbconfig/20210520-065928-root.json [production]
06:50 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
06:50 <ryankemper> T283223 Write queue not draining fast enough for the next node to reboot, will finish reboot tomorrow [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16114 and previous config saved to /var/cache/conftool/dbconfig/20210520-064425-root.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repool db1166', diff saved to https://phabricator.wikimedia.org/P16113 and previous config saved to /var/cache/conftool/dbconfig/20210520-062921-root.json [production]
06:25 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.6/includes/PageProps.php: Backport: [[gerrit:693028|PageProps: be prepared that PageIdentity is not proper title (T283170)]] (duration: 01m 06s) [production]
06:08 <elukey> powercycle ms-be2035 - no ssh available, no metrics since hours ago, I/O errors registered in the main tty on serial console [production]
05:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 100%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16112 and previous config saved to /var/cache/conftool/dbconfig/20210520-054402-root.json [production]
05:33 <ryankemper> T283223 `sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223` on `ryankemper@cumin1001` tmux session `restart_cloudelastic` [production]
05:33 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 75%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16111 and previous config saved to /var/cache/conftool/dbconfig/20210520-052859-root.json [production]
05:27 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:24 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
05:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts labsdb1011.eqiad.wmnet [production]
05:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 50%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16110 and previous config saved to /var/cache/conftool/dbconfig/20210520-051355-root.json [production]
05:13 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts labsdb1011.eqiad.wmnet [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1143', diff saved to https://phabricator.wikimedia.org/P16109 and previous config saved to /var/cache/conftool/dbconfig/20210520-050025-marostegui.json [production]
04:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1166', diff saved to https://phabricator.wikimedia.org/P16108 and previous config saved to /var/cache/conftool/dbconfig/20210520-045919-marostegui.json [production]
04:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1141 (re)pooling @ 25%: Repool db1141', diff saved to https://phabricator.wikimedia.org/P16107 and previous config saved to /var/cache/conftool/dbconfig/20210520-045852-root.json [production]
01:01 <mutante> signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192) [production]
00:14 <ejegg> updated fundraising CiviCRM from b3fb3c9cb0 to 35f5afb1b4 [production]
00:13 <ejegg> updated payments-wiki from 9f51ace546 to 6fac77f60e [production]
2021-05-19 §
22:44 <Urbanecm> [urbanecm@mwmaint1002 ~/uploads]$ sleep 3600 && mwscript importImages.php --wiki=commonswiki --comment-ext=txt --sleep=7200 --user=Lusccasdeutsch . # T278856 # 3 video files [production]
22:29 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh2002.wikimedia.org [production]
22:27 <Urbanecm> Start server-side upload for 1 video file (T283186) [production]
22:25 <razzi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:22 <Urbanecm> Start server-side upload for 3 video file (T283102, T283054) [production]
22:22 <razzi@cumin1001> START - Cookbook sre.dns.netbox [production]
22:21 <razzi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
22:18 <razzi@cumin1001> START - Cookbook sre.dns.netbox [production]
22:12 <urbanecm@deploy1002> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 14s) [production]
22:11 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh2001.wikimedia.org [production]