3751-3800 of 10000 results (59ms)
2022-05-05 ยง
11:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
11:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
11:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T307525)', diff saved to https://phabricator.wikimedia.org/P27697 and previous config saved to /var/cache/conftool/dbconfig/20220505-111934-ladsgroup.json [production]
11:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P27696 and previous config saved to /var/cache/conftool/dbconfig/20220505-111228-ladsgroup.json [production]
11:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase db1127 weight', diff saved to https://phabricator.wikimedia.org/P27695 and previous config saved to /var/cache/conftool/dbconfig/20220505-110940-marostegui.json [production]
11:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host theemin.codfw.wmnet [production]
11:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27694 and previous config saved to /var/cache/conftool/dbconfig/20220505-110821-ladsgroup.json [production]
11:05 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host theemin.codfw.wmnet [production]
11:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P27693 and previous config saved to /var/cache/conftool/dbconfig/20220505-110429-ladsgroup.json [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet [production]
11:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1025.eqiad.wmnet [production]
11:02 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet [production]
10:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2002.codfw.wmnet [production]
10:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people2002.codfw.wmnet [production]
10:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host restbase1025.eqiad.wmnet [production]
10:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T307525)', diff saved to https://phabricator.wikimedia.org/P27692 and previous config saved to /var/cache/conftool/dbconfig/20220505-105316-ladsgroup.json [production]
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase db1127 weight', diff saved to https://phabricator.wikimedia.org/P27691 and previous config saved to /var/cache/conftool/dbconfig/20220505-105049-marostegui.json [production]
10:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P27690 and previous config saved to /var/cache/conftool/dbconfig/20220505-104924-ladsgroup.json [production]
10:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1131 (T307525)', diff saved to https://phabricator.wikimedia.org/P27689 and previous config saved to /var/cache/conftool/dbconfig/20220505-104853-ladsgroup.json [production]
10:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
10:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
10:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T307525)', diff saved to https://phabricator.wikimedia.org/P27688 and previous config saved to /var/cache/conftool/dbconfig/20220505-104845-ladsgroup.json [production]
10:48 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1024.eqiad.wmnet [production]
10:45 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2049.codfw.wmnet with OS bullseye [production]
10:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2001.codfw.wmnet [production]
10:41 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host restbase1024.eqiad.wmnet [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1127 with low weight', diff saved to https://phabricator.wikimedia.org/P27687 and previous config saved to /var/cache/conftool/dbconfig/20220505-103723-marostegui.json [production]
10:36 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[1024-1026].eqiad.wmnet with reason: reboot [production]
10:36 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[1024-1026].eqiad.wmnet with reason: reboot [production]
10:36 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host build2001.codfw.wmnet [production]
10:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1023.eqiad.wmnet [production]
10:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T307525)', diff saved to https://phabricator.wikimedia.org/P27686 and previous config saved to /var/cache/conftool/dbconfig/20220505-103419-ladsgroup.json [production]
10:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27685 and previous config saved to /var/cache/conftool/dbconfig/20220505-103340-ladsgroup.json [production]
10:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1127 with low weight', diff saved to https://phabricator.wikimedia.org/P27684 and previous config saved to /var/cache/conftool/dbconfig/20220505-103111-marostegui.json [production]
10:30 <marostegui> Alter globalblocks on db1127 T307501 [production]
10:29 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2049.codfw.wmnet with reason: host reimage [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127', diff saved to https://phabricator.wikimedia.org/P27683 and previous config saved to /var/cache/conftool/dbconfig/20220505-102817-marostegui.json [production]
10:28 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host restbase1023.eqiad.wmnet [production]
10:26 <btullis@cumin1001> END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
10:26 <mvernon@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2049.codfw.wmnet with reason: host reimage [production]
10:24 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1022.eqiad.wmnet [production]
10:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: After the incident', diff saved to https://phabricator.wikimedia.org/P27682 and previous config saved to /var/cache/conftool/dbconfig/20220505-102056-root.json [production]
10:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27681 and previous config saved to /var/cache/conftool/dbconfig/20220505-101835-ladsgroup.json [production]
10:17 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host restbase1022.eqiad.wmnet [production]
10:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1174 (T307525)', diff saved to https://phabricator.wikimedia.org/P27680 and previous config saved to /var/cache/conftool/dbconfig/20220505-101400-ladsgroup.json [production]
10:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
10:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
10:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P27679 and previous config saved to /var/cache/conftool/dbconfig/20220505-101214-ladsgroup.json [production]
10:12 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
10:12 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]