151-200 of 10000 results (27ms)
2021-02-11 ยง
11:35 <mvolz@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
11:35 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2001.codfw.wmnet [production]
11:25 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host thumbor2001.codfw.wmnet [production]
11:25 <jiji@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor1004.eqiad.wmnet [production]
11:17 <mvolz@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
11:13 <mvolz@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
11:06 <mvolz@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' . [production]
11:04 <kormat@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 100%: changed binlog_format T274472', diff saved to https://phabricator.wikimedia.org/P14315 and previous config saved to /var/cache/conftool/dbconfig/20210211-110447-kormat.json [production]
11:03 <moritzm> installing firejail security updates on Stretch [production]
10:56 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2004.codfw.wmnet [production]
10:50 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host thanos-be2004.codfw.wmnet [production]
10:49 <kormat@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 66%: changed binlog_format T274472', diff saved to https://phabricator.wikimedia.org/P14314 and previous config saved to /var/cache/conftool/dbconfig/20210211-104943-kormat.json [production]
10:48 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2003.codfw.wmnet [production]
10:40 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host thanos-be2003.codfw.wmnet [production]
10:39 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2002.codfw.wmnet [production]
10:34 <kormat@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 33%: changed binlog_format T274472', diff saved to https://phabricator.wikimedia.org/P14313 and previous config saved to /var/cache/conftool/dbconfig/20210211-103440-kormat.json [production]
10:33 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host thanos-be2002.codfw.wmnet [production]
10:25 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2001.codfw.wmnet [production]
10:20 <kormat@cumin1001> dbctl commit (dc=all): 'db1118 depooling: change binlog_format', diff saved to https://phabricator.wikimedia.org/P14312 and previous config saved to /var/cache/conftool/dbconfig/20210211-101959-kormat.json [production]
10:19 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1118.eqiad.wmnet with reason: Depooling to change binglog_format T274472 [production]
10:19 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1118.eqiad.wmnet with reason: Depooling to change binglog_format T274472 [production]
10:18 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host thanos-be2001.codfw.wmnet [production]
10:15 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1004.eqiad.wmnet [production]
10:14 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4025.ulsfo.wmnet [production]
10:14 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4031.ulsfo.wmnet [production]
10:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2035.codfw.wmnet [production]
10:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp1084.eqiad.wmnet [production]
10:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3059.esams.wmnet [production]
10:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp1083.eqiad.wmnet [production]
10:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3058.esams.wmnet [production]
10:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2036.codfw.wmnet [production]
10:10 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp5005.eqsin.wmnet [production]
10:08 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp5011.eqsin.wmnet [production]
10:07 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host thanos-be1004.eqiad.wmnet [production]
10:02 <jynus> switching db1118 to row_format=STATEMENT as new s1 master candidate [production]
10:00 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp5005.eqsin.wmnet [production]
10:00 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp5011.eqsin.wmnet [production]
10:00 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp4025.ulsfo.wmnet [production]
09:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp4031.ulsfo.wmnet [production]
09:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp3059.esams.wmnet [production]
09:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp3058.esams.wmnet [production]
09:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp2036.codfw.wmnet [production]
09:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp2035.codfw.wmnet [production]
09:58 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp1084.eqiad.wmnet [production]
09:58 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host cp1083.eqiad.wmnet [production]
09:53 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host thumbor1004.eqiad.wmnet [production]
09:44 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
09:44 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet [production]
09:43 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
09:38 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet [production]