5551-5600 of 10000 results (108ms)
2022-11-16 ยง
09:13 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 45899 [production]
09:13 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 45899 [production]
08:47 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1022.eqiad.wmnet to cluster eqiad and group D [production]
08:45 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1022.eqiad.wmnet to cluster eqiad and group D [production]
08:45 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet [production]
08:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
08:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
08:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39911 and previous config saved to /var/cache/conftool/dbconfig/20221116-083723-ladsgroup.json [production]
08:36 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet [production]
08:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39910 and previous config saved to /var/cache/conftool/dbconfig/20221116-082217-ladsgroup.json [production]
08:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39909 and previous config saved to /var/cache/conftool/dbconfig/20221116-080710-ladsgroup.json [production]
08:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2167:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39908 and previous config saved to /var/cache/conftool/dbconfig/20221116-080001-ladsgroup.json [production]
07:59 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
07:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
07:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39907 and previous config saved to /var/cache/conftool/dbconfig/20221116-075940-ladsgroup.json [production]
07:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39906 and previous config saved to /var/cache/conftool/dbconfig/20221116-075204-ladsgroup.json [production]
07:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39905 and previous config saved to /var/cache/conftool/dbconfig/20221116-074433-ladsgroup.json [production]
07:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39904 and previous config saved to /var/cache/conftool/dbconfig/20221116-072926-ladsgroup.json [production]
07:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39903 and previous config saved to /var/cache/conftool/dbconfig/20221116-071420-ladsgroup.json [production]
06:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
06:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
06:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:35 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39902 and previous config saved to /var/cache/conftool/dbconfig/20221116-062315-ladsgroup.json [production]
06:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]
06:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]
06:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39901 and previous config saved to /var/cache/conftool/dbconfig/20221116-062253-ladsgroup.json [production]
06:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39900 and previous config saved to /var/cache/conftool/dbconfig/20221116-060747-ladsgroup.json [production]
05:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39899 and previous config saved to /var/cache/conftool/dbconfig/20221116-055241-ladsgroup.json [production]
05:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39898 and previous config saved to /var/cache/conftool/dbconfig/20221116-054935-ladsgroup.json [production]
05:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance [production]
05:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance [production]
05:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318605)', diff saved to https://phabricator.wikimedia.org/P39897 and previous config saved to /var/cache/conftool/dbconfig/20221116-054914-ladsgroup.json [production]
05:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39896 and previous config saved to /var/cache/conftool/dbconfig/20221116-053734-ladsgroup.json [production]
05:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39895 and previous config saved to /var/cache/conftool/dbconfig/20221116-053407-ladsgroup.json [production]
05:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39894 and previous config saved to /var/cache/conftool/dbconfig/20221116-051901-ladsgroup.json [production]
05:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318605)', diff saved to https://phabricator.wikimedia.org/P39893 and previous config saved to /var/cache/conftool/dbconfig/20221116-050354-ladsgroup.json [production]
04:49 <TimStarling> on mwmaint1002: running storageTypeStats.php on dewiki [production]
04:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39892 and previous config saved to /var/cache/conftool/dbconfig/20221116-040652-ladsgroup.json [production]
04:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
04:06 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
04:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39891 and previous config saved to /var/cache/conftool/dbconfig/20221116-040630-ladsgroup.json [production]
03:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321130)', diff saved to https://phabricator.wikimedia.org/P39890 and previous config saved to /var/cache/conftool/dbconfig/20221116-035750-marostegui.json [production]
03:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P39889 and previous config saved to /var/cache/conftool/dbconfig/20221116-035124-ladsgroup.json [production]
03:46 <eileen> civicrm upgraded from fa71f219 to 85c98fc7 [production]
03:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P39888 and previous config saved to /var/cache/conftool/dbconfig/20221116-034243-marostegui.json [production]
03:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P39887 and previous config saved to /var/cache/conftool/dbconfig/20221116-033617-ladsgroup.json [production]
03:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P39886 and previous config saved to /var/cache/conftool/dbconfig/20221116-032737-marostegui.json [production]