4401-4450 of 10000 results (98ms)
2023-04-13 ยง
13:23 <vgutierrez> restarting haproxy in cp5022 - T334448 [production]
13:19 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@a08f56d]: (no justification provided) (duration: 17m 02s) [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P46672 and previous config saved to /var/cache/conftool/dbconfig/20230413-131021-root.json [production]
13:04 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
13:02 <jgiannelos@deploy2002> Started deploy [restbase/deploy@a08f56d]: (no justification provided) [production]
12:57 <cgoubert@deploy2002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
12:57 <cgoubert@deploy2002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
12:56 <claime> Migrating cxserver to mw-api-int on kubernetes, take two - T334204 [production]
12:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P46671 and previous config saved to /var/cache/conftool/dbconfig/20230413-125516-root.json [production]
12:49 <jelto@cumin2002> START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye [production]
12:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P46670 and previous config saved to /var/cache/conftool/dbconfig/20230413-124011-root.json [production]
12:38 <moritzm> installing systemd security updates on buster [production]
12:33 <moritzm> installing Django security updates [production]
12:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P46669 and previous config saved to /var/cache/conftool/dbconfig/20230413-122506-root.json [production]
12:21 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3001.esams.wmnet [production]
12:21 <moritzm> remove imagemagick 8:6.9.10.23+dfsg-2.1+deb10u1+wmf1 from apt.wikimedia.org (obsoleted by 8:6.9.10.23+dfsg-2.1+deb10u4 from the Debian archive) T328901 [production]
12:15 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet [production]
12:11 <moritzm> installing imagemagick security updates for buster T328901 [production]
12:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 4%: Repooling', diff saved to https://phabricator.wikimedia.org/P46668 and previous config saved to /var/cache/conftool/dbconfig/20230413-121001-root.json [production]
11:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 3%: Repooling', diff saved to https://phabricator.wikimedia.org/P46667 and previous config saved to /var/cache/conftool/dbconfig/20230413-115456-root.json [production]
11:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 2%: Repooling', diff saved to https://phabricator.wikimedia.org/P46666 and previous config saved to /var/cache/conftool/dbconfig/20230413-113951-root.json [production]
11:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1120 from dbctl T334580', diff saved to https://phabricator.wikimedia.org/P46665 and previous config saved to /var/cache/conftool/dbconfig/20230413-113435-marostegui.json [production]
11:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1114 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P46664 and previous config saved to /var/cache/conftool/dbconfig/20230413-112446-root.json [production]
11:24 <moritzm> installing imagemagick security updates [production]
11:18 <cgoubert@deploy2002> Finished scap: Updating mw-on-k8s certificates (duration: 01m 56s) [production]
11:16 <cgoubert@deploy2002> Started scap: Updating mw-on-k8s certificates [production]
11:15 <claime> Re-deploying mw-on-k8s to update certificates - T334561 [production]
10:39 <claime> updating appservers and api certificates - T334561 [production]
10:23 <Emperor> clear old 2/22/Free-object-universal-property.svg thumbs from wikipedia-commons-local-thumb.22 T334303 [production]
10:15 <stevemunene@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1132.eqiad.wmnet with OS buster [production]
10:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46662 and previous config saved to /var/cache/conftool/dbconfig/20230413-101307-root.json [production]
10:07 <moritzm> installing tomcat security updates [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46661 and previous config saved to /var/cache/conftool/dbconfig/20230413-095802-root.json [production]
09:53 <taavi> taavi@mwmaint2002 ~ $ mwscript emptyUserGroup.php --wiki frwikinews editor # T333750 [production]
09:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46660 and previous config saved to /var/cache/conftool/dbconfig/20230413-094257-root.json [production]
09:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 25%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46659 and previous config saved to /var/cache/conftool/dbconfig/20230413-092752-root.json [production]
09:25 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dse-k8s-worker1001.eqiad.wmnet [production]
09:22 <stevemunene@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1132.eqiad.wmnet with OS buster [production]
09:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 10%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46658 and previous config saved to /var/cache/conftool/dbconfig/20230413-091247-root.json [production]
09:12 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1001.eqiad.wmnet [production]
09:04 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: centrallog1001.eqiad.wmnet [production]
09:04 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: centrallog1001.eqiad.wmnet [production]
09:01 <stevemunene@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1132.eqiad.wmnet with OS buster [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 5%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46657 and previous config saved to /var/cache/conftool/dbconfig/20230413-085742-root.json [production]
08:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 4%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46656 and previous config saved to /var/cache/conftool/dbconfig/20230413-084238-root.json [production]
08:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1221 (re)pooling @ 100%: Pooling', diff saved to https://phabricator.wikimedia.org/P46655 and previous config saved to /var/cache/conftool/dbconfig/20230413-084036-root.json [production]
08:36 <moritzm> installing git security updates [production]
08:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1210 (re)pooling @ 100%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46654 and previous config saved to /var/cache/conftool/dbconfig/20230413-083457-root.json [production]
08:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1223 (re)pooling @ 3%: Pooling db1223 T326669', diff saved to https://phabricator.wikimedia.org/P46653 and previous config saved to /var/cache/conftool/dbconfig/20230413-082732-root.json [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1221 (re)pooling @ 75%: Pooling', diff saved to https://phabricator.wikimedia.org/P46652 and previous config saved to /var/cache/conftool/dbconfig/20230413-082532-root.json [production]