2851-2900 of 10000 results (30ms)
2021-03-03 ยง
14:06 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1018.eqiad.wmnet [production]
13:58 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1018.eqiad.wmnet [production]
13:09 <godog> swift eqiad-prod: remove ssd weight for ms-be1034 - T276193 [production]
12:54 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1017.eqiad.wmnet [production]
12:48 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1017.eqiad.wmnet [production]
12:42 <Urbanecm> Deploy a security patch for T276306 [production]
12:29 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/GrowthExperiments/ServiceWiring.php: cf635b46a112433f78979b04eb21729783cb2033: Do not open DB connections during service initialization (T276307) (duration: 01m 11s) [production]
12:26 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1016.eqiad.wmnet [production]
12:26 <Urbanecm> urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 90a205ffd60c95c4e87b0da46d9d85138ab74455: Add ReferenceTooltips and other gadget names for ReferencePreviews (T274353) (duration: 01m 10s) [production]
12:20 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1016.eqiad.wmnet [production]
12:04 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1014.eqiad.wmnet [production]
11:58 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1014.eqiad.wmnet [production]
11:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14616 and previous config saved to /var/cache/conftool/dbconfig/20210303-113349-root.json [production]
11:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P14615 and previous config saved to /var/cache/conftool/dbconfig/20210303-111843-root.json [production]
11:07 <filippo@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw [production]
11:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P14614 and previous config saved to /var/cache/conftool/dbconfig/20210303-110339-root.json [production]
10:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 100%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14613 and previous config saved to /var/cache/conftool/dbconfig/20210303-105726-root.json [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P14612 and previous config saved to /var/cache/conftool/dbconfig/20210303-104836-root.json [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1175 for schema change', diff saved to https://phabricator.wikimedia.org/P14611 and previous config saved to /var/cache/conftool/dbconfig/20210303-104522-marostegui.json [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14610 and previous config saved to /var/cache/conftool/dbconfig/20210303-104302-root.json [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 90%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14609 and previous config saved to /var/cache/conftool/dbconfig/20210303-104223-root.json [production]
10:38 <jbond42> upload new wmf-laptop 0.5.0 package [production]
10:37 <vgutierrez> rolling restart of ats-tls on eqiad [production]
10:34 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:34 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:34 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P14608 and previous config saved to /var/cache/conftool/dbconfig/20210303-102758-root.json [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 75%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14607 and previous config saved to /var/cache/conftool/dbconfig/20210303-102719-root.json [production]
10:25 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:25 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P14605 and previous config saved to /var/cache/conftool/dbconfig/20210303-101255-root.json [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 60%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14604 and previous config saved to /var/cache/conftool/dbconfig/20210303-101215-root.json [production]
10:05 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet [production]
10:00 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet [production]
10:00 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P14602 and previous config saved to /var/cache/conftool/dbconfig/20210303-095751-root.json [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 50%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14601 and previous config saved to /var/cache/conftool/dbconfig/20210303-095712-root.json [production]
09:55 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cloudnet1003.eqiad.wmnet with reason: HW issue [production]
09:54 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on cloudnet1003.eqiad.wmnet with reason: HW issue [production]
09:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1166 for schema change', diff saved to https://phabricator.wikimedia.org/P14600 and previous config saved to /var/cache/conftool/dbconfig/20210303-095417-marostegui.json [production]
09:53 <marostegui@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14599 and previous config saved to /var/cache/conftool/dbconfig/20210303-095351-root.json [production]
09:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 30%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14598 and previous config saved to /var/cache/conftool/dbconfig/20210303-094208-root.json [production]
09:41 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker[1132,1135-1138].eqiad.wmnet [production]
09:39 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker[1132,1135-1138].eqiad.wmnet [production]
09:38 <marostegui@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P14597 and previous config saved to /var/cache/conftool/dbconfig/20210303-093847-root.json [production]
09:31 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet [production]
09:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1138.eqiad.wmnet with reason: REIMAGE [production]
09:28 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1138.eqiad.wmnet with reason: REIMAGE [production]
09:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1137.eqiad.wmnet with reason: REIMAGE [production]
09:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1164 (re)pooling @ 25%: Slowly repool db1164 in s1 for the first time', diff saved to https://phabricator.wikimedia.org/P14596 and previous config saved to /var/cache/conftool/dbconfig/20210303-092705-root.json [production]