3551-3600 of 10000 results (107ms)
2022-11-16 ยง
10:00 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
09:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
09:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39913 and previous config saved to /var/cache/conftool/dbconfig/20221116-094618-ladsgroup.json [production]
09:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39912 and previous config saved to /var/cache/conftool/dbconfig/20221116-093112-ladsgroup.json [production]
09:18 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 293 [production]
09:17 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 293 [production]
09:16 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30844 [production]
09:16 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 30844 [production]
09:13 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 45899 [production]
09:13 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 45899 [production]
08:47 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1022.eqiad.wmnet to cluster eqiad and group D [production]
08:45 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1022.eqiad.wmnet to cluster eqiad and group D [production]
08:45 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet [production]
08:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
08:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
08:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39911 and previous config saved to /var/cache/conftool/dbconfig/20221116-083723-ladsgroup.json [production]
08:36 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet [production]
08:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39910 and previous config saved to /var/cache/conftool/dbconfig/20221116-082217-ladsgroup.json [production]
08:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39909 and previous config saved to /var/cache/conftool/dbconfig/20221116-080710-ladsgroup.json [production]
08:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2167:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39908 and previous config saved to /var/cache/conftool/dbconfig/20221116-080001-ladsgroup.json [production]
07:59 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
07:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
07:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39907 and previous config saved to /var/cache/conftool/dbconfig/20221116-075940-ladsgroup.json [production]
07:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39906 and previous config saved to /var/cache/conftool/dbconfig/20221116-075204-ladsgroup.json [production]
07:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39905 and previous config saved to /var/cache/conftool/dbconfig/20221116-074433-ladsgroup.json [production]
07:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39904 and previous config saved to /var/cache/conftool/dbconfig/20221116-072926-ladsgroup.json [production]
07:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39903 and previous config saved to /var/cache/conftool/dbconfig/20221116-071420-ladsgroup.json [production]
06:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
06:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
06:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:35 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1132 (T318605)', diff saved to https://phabricator.wikimedia.org/P39902 and previous config saved to /var/cache/conftool/dbconfig/20221116-062315-ladsgroup.json [production]
06:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]
06:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]
06:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39901 and previous config saved to /var/cache/conftool/dbconfig/20221116-062253-ladsgroup.json [production]
06:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39900 and previous config saved to /var/cache/conftool/dbconfig/20221116-060747-ladsgroup.json [production]
05:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39899 and previous config saved to /var/cache/conftool/dbconfig/20221116-055241-ladsgroup.json [production]
05:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2153 (T318605)', diff saved to https://phabricator.wikimedia.org/P39898 and previous config saved to /var/cache/conftool/dbconfig/20221116-054935-ladsgroup.json [production]
05:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance [production]
05:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance [production]
05:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318605)', diff saved to https://phabricator.wikimedia.org/P39897 and previous config saved to /var/cache/conftool/dbconfig/20221116-054914-ladsgroup.json [production]
05:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39896 and previous config saved to /var/cache/conftool/dbconfig/20221116-053734-ladsgroup.json [production]
05:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39895 and previous config saved to /var/cache/conftool/dbconfig/20221116-053407-ladsgroup.json [production]
05:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39894 and previous config saved to /var/cache/conftool/dbconfig/20221116-051901-ladsgroup.json [production]
05:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318605)', diff saved to https://phabricator.wikimedia.org/P39893 and previous config saved to /var/cache/conftool/dbconfig/20221116-050354-ladsgroup.json [production]
04:49 <TimStarling> on mwmaint1002: running storageTypeStats.php on dewiki [production]
04:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1128 (T318605)', diff saved to https://phabricator.wikimedia.org/P39892 and previous config saved to /var/cache/conftool/dbconfig/20221116-040652-ladsgroup.json [production]
04:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]