3651-3700 of 10000 results (41ms)
2021-03-03 ยง
15:18 <liw@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.33 [production]
15:13 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/CentralAuth/: af899b6818223928e2da421122c19e64126370da: Transform the first parameter to string (T276316) (duration: 01m 11s) [production]
14:48 <effie> upgrade memcached on mc1027,mc2027 [production]
14:41 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1101.eqiad.wmnet with reason: REIMAGE [production]
14:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1100.eqiad.wmnet with reason: REIMAGE [production]
14:39 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1101.eqiad.wmnet with reason: REIMAGE [production]
14:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1099.eqiad.wmnet with reason: REIMAGE [production]
14:37 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1100.eqiad.wmnet with reason: REIMAGE [production]
14:35 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1099.eqiad.wmnet with reason: REIMAGE [production]
14:06 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1018.eqiad.wmnet [production]
13:58 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1018.eqiad.wmnet [production]
13:09 <godog> swift eqiad-prod: remove ssd weight for ms-be1034 - T276193 [production]
12:54 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1017.eqiad.wmnet [production]
12:48 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1017.eqiad.wmnet [production]
12:42 <Urbanecm> Deploy a security patch for T276306 [production]
12:29 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/GrowthExperiments/ServiceWiring.php: cf635b46a112433f78979b04eb21729783cb2033: Do not open DB connections during service initialization (T276307) (duration: 01m 11s) [production]
12:26 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1016.eqiad.wmnet [production]
12:26 <Urbanecm> urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 90a205ffd60c95c4e87b0da46d9d85138ab74455: Add ReferenceTooltips and other gadget names for ReferencePreviews (T274353) (duration: 01m 10s) [production]
12:20 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1016.eqiad.wmnet [production]
12:04 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1014.eqiad.wmnet [production]
11:58 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1014.eqiad.wmnet [production]
11:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14616 and previous config saved to /var/cache/conftool/dbconfig/20210303-113349-root.json [production]
11:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P14615 and previous config saved to /var/cache/conftool/dbconfig/20210303-111843-root.json [production]
11:07 <filippo@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw [production]
11:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P14614 and previous config saved to /var/cache/conftool/dbconfig/20210303-110339-root.json [production]
10:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 100%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14613 and previous config saved to /var/cache/conftool/dbconfig/20210303-105726-root.json [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P14612 and previous config saved to /var/cache/conftool/dbconfig/20210303-104836-root.json [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1175 for schema change', diff saved to https://phabricator.wikimedia.org/P14611 and previous config saved to /var/cache/conftool/dbconfig/20210303-104522-marostegui.json [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14610 and previous config saved to /var/cache/conftool/dbconfig/20210303-104302-root.json [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 90%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14609 and previous config saved to /var/cache/conftool/dbconfig/20210303-104223-root.json [production]
10:38 <jbond42> upload new wmf-laptop 0.5.0 package [production]
10:37 <vgutierrez> rolling restart of ats-tls on eqiad [production]
10:34 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:34 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:34 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P14608 and previous config saved to /var/cache/conftool/dbconfig/20210303-102758-root.json [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 75%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14607 and previous config saved to /var/cache/conftool/dbconfig/20210303-102719-root.json [production]
10:25 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:25 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P14605 and previous config saved to /var/cache/conftool/dbconfig/20210303-101255-root.json [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 60%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14604 and previous config saved to /var/cache/conftool/dbconfig/20210303-101215-root.json [production]
10:05 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet [production]
10:00 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet [production]
10:00 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P14602 and previous config saved to /var/cache/conftool/dbconfig/20210303-095751-root.json [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 50%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14601 and previous config saved to /var/cache/conftool/dbconfig/20210303-095712-root.json [production]
09:55 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cloudnet1003.eqiad.wmnet with reason: HW issue [production]
09:54 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on cloudnet1003.eqiad.wmnet with reason: HW issue [production]
09:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1166 for schema change', diff saved to https://phabricator.wikimedia.org/P14600 and previous config saved to /var/cache/conftool/dbconfig/20210303-095417-marostegui.json [production]
09:53 <marostegui@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14599 and previous config saved to /var/cache/conftool/dbconfig/20210303-095351-root.json [production]