2101-2150 of 10000 results (32ms)
2021-05-04 ยง
17:16 <dancy@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.4 [production]
17:13 <wm-bot> <bd808> Updated to use new *.db.svc.wikimedia.cloud naming scheme [tools.replag]
17:02 <brennen> 1.37.0-wmf.4 was branched at f069fd8b5a6c817f4860fa68ae2f56b71a139f4a for T281145 [production]
17:02 <Amir1> stop exim4 and upgrade it in deployment-mx02 [releng]
17:00 <volans> uploaded spicerack_0.0.51 to apt.wikimedia.org bullseye-wikimedia [production]
16:26 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@e6ae572]: Increase convert_to_esbulk memory overhead (duration: 01m 54s) [production]
16:25 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@e6ae572]: Increase convert_to_esbulk memory overhead [production]
16:16 <dzahn@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:15 <dzahn@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
16:13 <mutante> k8s: upgrading release=namespaces, helmfile apply to create miscweb namespace T281538 [production]
16:13 <dzahn@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
16:12 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:12 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:12 <dzahn@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
16:07 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:07 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:05 <wm-bot> Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [admin]
15:59 <hnowlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
15:59 <hnowlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:58 <arturo> shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04) [cloudinfra]
15:56 <arturo> relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04 [cloudinfra]
15:45 <wm-bot> Safe rebooting 'cloudvirt1028.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [admin]
15:44 <wm-bot> Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [admin]
15:33 <arturo> created VMs mx-out03/mx-out04 as debian buster [cloudinfra]
15:31 <arturo> bump instance quota from 12 to 14 [cloudinfra]
15:28 <arturo> created anti-affinity server group 'mx' [cloudinfra]
15:23 <arturo> upgrading exim4-daemon-heavy in tools-mail-03 [tools]
15:22 <wm-bot> Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [admin]
15:19 <wm-bot> Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [admin]
15:15 <wm-bot> Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [admin]
14:41 <jmm@cumin2001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
13:46 <moritzm> installing exim security updates on buster [production]
13:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Repool db1137 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15721 and previous config saved to /var/cache/conftool/dbconfig/20210504-133950-root.json [production]
13:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Repool db1137 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15720 and previous config saved to /var/cache/conftool/dbconfig/20210504-132446-root.json [production]
13:19 <dcaro> rebooting cloudmetrics1002, got stuck again (T275605) [admin]
13:14 <moritzm> upgrading linux-libc-dev on buster hosts (to version introduced by 10.9 point release) [production]
13:12 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
13:12 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
13:09 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Repool db1137 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15719 and previous config saved to /var/cache/conftool/dbconfig/20210504-130943-root.json [production]
13:01 <moritzm> installing debian-archive-keyring updates on buster [production]
12:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Repool db1137 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15718 and previous config saved to /var/cache/conftool/dbconfig/20210504-125439-root.json [production]
12:50 <marostegui> Upgrade mysql and kernel on db1137 T281212 [production]
12:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 to upgrade its mysql T281212', diff saved to https://phabricator.wikimedia.org/P15717 and previous config saved to /var/cache/conftool/dbconfig/20210504-124937-marostegui.json [production]
12:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1120 (re)pooling @ 100%: Repool db1120 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15716 and previous config saved to /var/cache/conftool/dbconfig/20210504-124848-root.json [production]
12:46 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after sanitarium master switch T280751', diff saved to https://phabricator.wikimedia.org/P15715 and previous config saved to /var/cache/conftool/dbconfig/20210504-124647-kormat.json [production]
12:35 <kormat@cumin1001> dbctl commit (dc=all): 'Depooling for sanitarium master switch T280751', diff saved to https://phabricator.wikimedia.org/P15714 and previous config saved to /var/cache/conftool/dbconfig/20210504-123537-kormat.json [production]
12:34 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Replace db1085 with db1165 T280751 [production]
12:34 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Replace db1085 with db1165 T280751 [production]
12:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1120 (re)pooling @ 75%: Repool db1120 after mysql upgrade', diff saved to https://phabricator.wikimedia.org/P15713 and previous config saved to /var/cache/conftool/dbconfig/20210504-123344-root.json [production]
12:27 <hnowlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]