51-100 of 10000 results (28ms)
2021-04-07 ยง
15:45 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: REIMAGE [production]
15:39 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
15:13 <Amir1> setting enwiki and enwikibooks to wmf.38 on mwdebug1002 to test flagged revs [production]
15:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 100%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15228 and previous config saved to /var/cache/conftool/dbconfig/20210407-150436-root.json [production]
14:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 75%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15227 and previous config saved to /var/cache/conftool/dbconfig/20210407-144933-root.json [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 50%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15226 and previous config saved to /var/cache/conftool/dbconfig/20210407-143429-root.json [production]
14:33 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2411.codfw.wmnet with reason: REIMAGE [production]
14:31 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2411.codfw.wmnet with reason: REIMAGE [production]
14:19 <effie> restarting pybal on lvs2009, lvs1015 [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 25%: Repool db1173 after cloning db1180', diff saved to https://phabricator.wikimedia.org/P15225 and previous config saved to /var/cache/conftool/dbconfig/20210407-141925-root.json [production]
14:16 <effie> restarting pybal on lvs2010, lvs1016 [production]
14:05 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2410.codfw.wmnet with reason: REIMAGE [production]
14:03 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2410.codfw.wmnet with reason: REIMAGE [production]
13:54 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2409.codfw.wmnet with reason: REIMAGE [production]
13:52 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2409.codfw.wmnet with reason: REIMAGE [production]
13:43 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:43 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:42 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:42 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:41 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:41 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:39 <moritzm> imported jenkins 2.277.2 to apt.wikimedia.org (thirdparty/ci) T279033 [production]
13:37 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:36 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
13:35 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:35 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
13:33 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:33 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
12:45 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2028.codfw.wmnet [production]
12:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 100%: Repool db1118 after schema change', diff saved to https://phabricator.wikimedia.org/P15224 and previous config saved to /var/cache/conftool/dbconfig/20210407-122304-root.json [production]
12:18 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
12:18 <marostegui> Upgrade db1173's kernel [production]
12:18 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host ms-be2028.codfw.wmnet [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1173', diff saved to https://phabricator.wikimedia.org/P15222 and previous config saved to /var/cache/conftool/dbconfig/20210407-121659-marostegui.json [production]
12:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 75%: Repool db1118 after schema change', diff saved to https://phabricator.wikimedia.org/P15221 and previous config saved to /var/cache/conftool/dbconfig/20210407-120800-root.json [production]
12:05 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 50%: Repool db1118 after schema change', diff saved to https://phabricator.wikimedia.org/P15220 and previous config saved to /var/cache/conftool/dbconfig/20210407-115257-root.json [production]
11:39 <marostegui> Deploy schema change on s3 codfw, lag will appear T276150 T276156 [production]
11:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1118 (re)pooling @ 25%: Repool db1118 after schema change', diff saved to https://phabricator.wikimedia.org/P15219 and previous config saved to /var/cache/conftool/dbconfig/20210407-113753-root.json [production]
11:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1184 to s1 depooled T275633', diff saved to https://phabricator.wikimedia.org/P15218 and previous config saved to /var/cache/conftool/dbconfig/20210407-111708-marostegui.json [production]
11:15 <ladsgroup@deploy1002> Synchronized wmf-config/flaggedrevs.php: [[gerrit:677412|flaggedrevs: Disable quality and pristine tier in all wikis]] (T277883) (duration: 02m 15s) [production]
10:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1118', diff saved to https://phabricator.wikimedia.org/P15217 and previous config saved to /var/cache/conftool/dbconfig/20210407-105617-marostegui.json [production]
10:51 <marostegui> Stop apache on dbmonitor1001 T224589 [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P15216 and previous config saved to /var/cache/conftool/dbconfig/20210407-103404-marostegui.json [production]
10:01 <kormat@cumin1001> dbctl commit (dc=all): 'Repool db2106 and db2147 T279406', diff saved to https://phabricator.wikimedia.org/P15215 and previous config saved to /var/cache/conftool/dbconfig/20210407-100147-kormat.json [production]
09:58 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kraz.wikimedia.org [production]
09:58 <moritzm> reboot kraz to nudge reconnects to irc2001.w.o for remaining connected clients [production]
09:58 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host kraz.wikimedia.org [production]
09:40 <moritzm> imported git-lfs for bullseye/main (part of standard packages) T275873 [production]