2451-2500 of 10000 results (46ms)
2022-03-11 §
10:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2011.codfw.wmnet with reason: host reimage [production]
10:25 <vgutierrez> disable certspotter - T303593 [production]
10:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2011.codfw.wmnet with reason: host reimage [production]
10:24 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
10:19 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
10:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.reimage for host db2110.codfw.wmnet with OS bullseye [production]
10:16 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
10:14 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
10:09 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2011.codfw.wmnet with OS bullseye [production]
10:09 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
10:04 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 12 hosts with reason: Maintenance [production]
10:04 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 12 hosts with reason: Maintenance [production]
10:04 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance [production]
10:04 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance [production]
10:03 <dcausse> manually installed jvmquake to wdqs1010 (test machine) from https://people.wikimedia.org/~jmm/jvmquake/ [production]
09:54 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
09:47 <vgutierrez> stopping certspotter on alert1001 [production]
09:47 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
09:42 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
09:36 <ayounsi@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
09:35 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
09:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
09:29 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
09:15 <ayounsi@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
09:15 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
09:00 <jayme> kubernetes2011:~# systemctl restart rsyslog.service - T289766 [production]
08:52 <ayounsi@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
08:51 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cloudvirt1017.eqiad.wmnet [production]
08:43 <dcausse> restarting blazegraph on wdqs1012 (jvm stuck for 5hours) [production]
08:42 <jynus> upgrade and restart db2139 [production]
08:41 <ayounsi@cumin1001> START - Cookbook sre.hosts.dhcp for host cloudvirt1017.eqiad.wmnet [production]
08:30 <jynus> upgrade and restart db1145 [production]
08:23 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cloudvirt1017.eqiad.wmnet [production]
08:21 <ayounsi@cumin1001> START - Cookbook sre.hosts.dhcp for host cloudvirt1017.eqiad.wmnet [production]
08:19 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1017.eqiad.wmnet with OS bullseye [production]
06:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P22364 and previous config saved to /var/cache/conftool/dbconfig/20220311-063921-root.json [production]
06:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P22363 and previous config saved to /var/cache/conftool/dbconfig/20220311-062417-root.json [production]
06:13 <marostegui> Reboot dbproxy1014 T303174 [production]
06:09 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P22362 and previous config saved to /var/cache/conftool/dbconfig/20220311-060913-root.json [production]
05:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P22361 and previous config saved to /var/cache/conftool/dbconfig/20220311-055409-root.json [production]
05:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106', diff saved to https://phabricator.wikimedia.org/P22360 and previous config saved to /var/cache/conftool/dbconfig/20220311-054514-marostegui.json [production]
02:54 <eileen> revision changed from 9fb68b24 to 252269c8 [production]
01:56 <eileen> civicrm revision changed from 8501c38c to 9fb68b24 [production]
01:31 <eileen> civicrm changed from 4cb2bdbc to 8501c38c [production]
00:33 <TimStarling> on mwmaint1002 running populateGlobalEditCount.php [production]
00:03 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
00:01 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
2022-03-10 §
23:58 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
23:55 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
23:08 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]