2501-2550 of 10000 results (45ms)
2021-05-11 ยง
15:58 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mwlog1001.eqiad.wmnet [production]
15:55 <bstorm> restart haproxy on dbproxy1018/9 to remove old config [production]
15:47 <herron@cumin1001> START - Cookbook sre.hosts.decommission for hosts mwlog1001.eqiad.wmnet [production]
15:38 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mwlog2001.codfw.wmnet [production]
15:37 <dancy@deploy1002> scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
15:36 <dancy@deploy1002> sync-world aborted: testwikis wikis to 1.37.0-wmf.4 (duration: 02m 04s) [production]
15:34 <dancy@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.4 [production]
15:33 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:31 <dancy@deploy1002> scap failed: RuntimeError scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) (duration: 17m 36s) [production]
15:31 <dancy@deploy1002> scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
15:27 <herron@cumin1001> START - Cookbook sre.hosts.decommission for hosts mwlog2001.codfw.wmnet [production]
15:24 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
15:13 <dancy@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.5 [production]
15:03 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:01 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:59 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE [production]
14:57 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE [production]
14:49 <moritzm> installing busybox security updates [production]
14:38 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:31 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:29 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
14:27 <moritzm> installing cgal security updates [production]
14:26 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:14 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:14 <hashar> Restarted CI Jenkins with a snapshot of the Gearman Jenkins plugin # T281737 [production]
14:10 <hashar> Restarted CI Jenkins for plugin upgrade # T282433 [production]
14:05 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:01 <hashar> Restarted releases Jenkins for plugin upgrade # T282433 [production]
13:47 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1d4d00798bb24daa4e5b81b6c2ecda6143a6c6f0: enwiki: Growth features: Change help panel links (T281896) (duration: 01m 02s) [production]
13:39 <jbond42> rolling restart of ats-backend [production]
12:11 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mc1027.eqiad.wmnet [production]
12:11 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mc1027.eqiad.wmnet [production]
11:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15913 and previous config saved to /var/cache/conftool/dbconfig/20210511-114540-root.json [production]
11:35 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw1002.eqiad.wmnet [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15912 and previous config saved to /var/cache/conftool/dbconfig/20210511-113036-root.json [production]
11:16 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:688178|Add P2671 and P4839 to deprecated properties list (T280779)]] (duration: 00m 58s) [production]
11:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15911 and previous config saved to /var/cache/conftool/dbconfig/20210511-111532-root.json [production]
11:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15910 and previous config saved to /var/cache/conftool/dbconfig/20210511-110029-root.json [production]
10:52 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
10:46 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
10:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1162', diff saved to https://phabricator.wikimedia.org/P15909 and previous config saved to /var/cache/conftool/dbconfig/20210511-102303-marostegui.json [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15908 and previous config saved to /var/cache/conftool/dbconfig/20210511-102212-root.json [production]
10:13 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet [production]
10:13 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15907 and previous config saved to /var/cache/conftool/dbconfig/20210511-100708-root.json [production]
09:54 <aborrero@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudgw2002-dev.codfw.wmnet [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15904 and previous config saved to /var/cache/conftool/dbconfig/20210511-095204-root.json [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15903 and previous config saved to /var/cache/conftool/dbconfig/20210511-093701-root.json [production]
09:23 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet [production]