2351-2400 of 10000 results (30ms)
2021-06-21 ยง
14:17 <XioNoX> reboot scs-a1-codfw - T285229 [production]
14:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1008.eqiad.wmnet [production]
14:16 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet [production]
14:14 <klausman> starting update of ML team's etcd machines in eqiad [production]
14:14 <volans> uploaded spicerack_0.0.54 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
14:11 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2003.codfw.wmnet [production]
14:11 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps1008.eqiad.wmnet [production]
14:06 <otto@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
14:05 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2003.codfw.wmnet [production]
14:04 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2004.codfw.wmnet [production]
13:58 <XioNoX> reboot scs-eqsin - T285229 [production]
13:58 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2004.codfw.wmnet [production]
13:57 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet [production]
13:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1006.eqiad.wmnet [production]
13:56 <otto@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
13:55 <jynus> stopping replication at db1171:s3 at db1123-bin.004363:906878073 [production]
13:51 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet [production]
13:51 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2002.codfw.wmnet [production]
13:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps1006.eqiad.wmnet [production]
13:48 <XioNoX> reboot scs-ulsfo [production]
13:45 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2002.codfw.wmnet [production]
13:40 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2001.codfw.wmnet [production]
13:38 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2001.codfw.wmnet [production]
13:35 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2002.codfw.wmnet [production]
13:28 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.9/extensions/MobileFrontend/includes/ExtMobileFrontend.php: Backport: [[gerrit:700344|Avoid loading the whole entity when it only needs description. (T269960)]] (duration: 00m 58s) [production]
13:28 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet [production]
13:24 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet [production]
13:21 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet [production]
13:21 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet [production]
13:19 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet [production]
13:17 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet [production]
13:14 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet [production]
13:12 <elukey> upload istioctl 1.9.5 to {buster,stretch}-wikimedia [production]
13:12 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 40 hosts with reason: Merged broken patch [production]
13:12 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 40 hosts with reason: Merged broken patch [production]
13:09 <klausman> starting update of ML team's etcd machines in codfw [production]
12:55 <godog> move librenms alerts with "max alerts" == -1 to "interval" being 15m - T285205 [production]
12:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16672 and previous config saved to /var/cache/conftool/dbconfig/20210621-124030-root.json [production]
12:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 100%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16671 and previous config saved to /var/cache/conftool/dbconfig/20210621-123906-root.json [production]
12:35 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.9/extensions/Wikibase: Backport: [[gerrit:700528|Rewrite SerializationModifier to be more efficient]] (duration: 01m 02s) [production]
12:25 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1010.eqiad.wmnet [production]
12:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16670 and previous config saved to /var/cache/conftool/dbconfig/20210621-122526-root.json [production]
12:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16669 and previous config saved to /var/cache/conftool/dbconfig/20210621-122403-root.json [production]
12:19 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps1010.eqiad.wmnet [production]
12:12 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2008.codfw.wmnet [production]
12:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16668 and previous config saved to /var/cache/conftool/dbconfig/20210621-121023-root.json [production]
12:10 <godog> bump space for k8s and ops prometheus on prometheus1004 (prometheus1003 has been expanded previously but not logged) [production]
12:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 50%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16667 and previous config saved to /var/cache/conftool/dbconfig/20210621-120859-root.json [production]
11:57 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps2008.codfw.wmnet [production]
11:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16665 and previous config saved to /var/cache/conftool/dbconfig/20210621-115519-root.json [production]