201-250 of 10000 results (33ms)
2021-04-07 ยง
19:35 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: no-op for Beta (disable LocalisationUpdate extension) (duration: 01m 06s) [production]
19:24 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.38 (duration: 01m 06s) [production]
19:23 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.38 [production]
19:21 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on conf2005.codfw.wmnet with reason: REIMAGE [production]
19:19 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on conf2005.codfw.wmnet with reason: REIMAGE [production]
18:54 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on conf2004.codfw.wmnet with reason: REIMAGE [production]
18:52 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on conf2004.codfw.wmnet with reason: REIMAGE [production]
17:40 <tgr@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
17:40 <tgr@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
17:29 <tgr@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
17:29 <tgr@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
17:20 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-fe2002.codfw.wmnet with reason: REIMAGE [production]
17:18 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2002.codfw.wmnet with reason: REIMAGE [production]
16:45 <tgr@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
15:47 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-fe2001.codfw.wmnet with reason: REIMAGE [production]
15:45 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: REIMAGE [production]
15:39 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
15:13 <Amir1> setting enwiki and enwikibooks to wmf.38 on mwdebug1002 to test flagged revs [production]
15:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 100%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15228 and previous config saved to /var/cache/conftool/dbconfig/20210407-150436-root.json [production]
14:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 75%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15227 and previous config saved to /var/cache/conftool/dbconfig/20210407-144933-root.json [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 50%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15226 and previous config saved to /var/cache/conftool/dbconfig/20210407-143429-root.json [production]
14:33 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2411.codfw.wmnet with reason: REIMAGE [production]
14:31 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2411.codfw.wmnet with reason: REIMAGE [production]
14:19 <effie> restarting pybal on lvs2009, lvs1015 [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 25%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15225 and previous config saved to /var/cache/conftool/dbconfig/20210407-141925-root.json [production]
14:16 <effie> restarting pybal on lvs2010, lvs1016 [production]
14:05 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2410.codfw.wmnet with reason: REIMAGE [production]
14:03 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2410.codfw.wmnet with reason: REIMAGE [production]
13:54 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2409.codfw.wmnet with reason: REIMAGE [production]
13:52 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2409.codfw.wmnet with reason: REIMAGE [production]
13:43 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:43 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:42 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:42 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:41 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:41 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:39 <moritzm> imported jenkins 2.277.2 to apt.wikimedia.org (thirdparty/ci) T279033 [production]
13:37 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:36 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
13:35 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:35 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
13:33 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:33 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
12:45 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2028.codfw.wmnet [production]
12:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 100%: Repool db1118 after schema change', diff saved to https://phabricator.wikimedia.org/P15224 and previous config saved to /var/cache/conftool/dbconfig/20210407-122304-root.json [production]
12:18 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
12:18 <marostegui> Upgrade db1173's kernel [production]
12:18 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host ms-be2028.codfw.wmnet [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1173', diff saved to https://phabricator.wikimedia.org/P15222 and previous config saved to /var/cache/conftool/dbconfig/20210407-121659-marostegui.json [production]