1-50 of 10000 results (40ms)
2022-10-22 §
03:16 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
00:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
00:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
00:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T321312)', diff saved to https://phabricator.wikimedia.org/P35965 and previous config saved to /var/cache/conftool/dbconfig/20221022-000300-ladsgroup.json [production]
2022-10-21 §
23:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P35964 and previous config saved to /var/cache/conftool/dbconfig/20221021-234754-ladsgroup.json [production]
23:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P35963 and previous config saved to /var/cache/conftool/dbconfig/20221021-233247-ladsgroup.json [production]
23:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T321312)', diff saved to https://phabricator.wikimedia.org/P35962 and previous config saved to /var/cache/conftool/dbconfig/20221021-231741-ladsgroup.json [production]
23:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1203 (T321312)', diff saved to https://phabricator.wikimedia.org/P35961 and previous config saved to /var/cache/conftool/dbconfig/20221021-231026-ladsgroup.json [production]
23:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
23:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
23:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T321312)', diff saved to https://phabricator.wikimedia.org/P35960 and previous config saved to /var/cache/conftool/dbconfig/20221021-231001-ladsgroup.json [production]
22:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P35959 and previous config saved to /var/cache/conftool/dbconfig/20221021-225455-ladsgroup.json [production]
22:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P35958 and previous config saved to /var/cache/conftool/dbconfig/20221021-223948-ladsgroup.json [production]
22:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T321312)', diff saved to https://phabricator.wikimedia.org/P35957 and previous config saved to /var/cache/conftool/dbconfig/20221021-222442-ladsgroup.json [production]
22:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1193 (T321312)', diff saved to https://phabricator.wikimedia.org/P35956 and previous config saved to /var/cache/conftool/dbconfig/20221021-221826-ladsgroup.json [production]
22:18 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
22:18 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
22:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T321312)', diff saved to https://phabricator.wikimedia.org/P35955 and previous config saved to /var/cache/conftool/dbconfig/20221021-221802-ladsgroup.json [production]
22:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P35954 and previous config saved to /var/cache/conftool/dbconfig/20221021-220256-ladsgroup.json [production]
21:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P35953 and previous config saved to /var/cache/conftool/dbconfig/20221021-214749-ladsgroup.json [production]
21:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T321312)', diff saved to https://phabricator.wikimedia.org/P35952 and previous config saved to /var/cache/conftool/dbconfig/20221021-213242-ladsgroup.json [production]
21:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1192 (T321312)', diff saved to https://phabricator.wikimedia.org/P35951 and previous config saved to /var/cache/conftool/dbconfig/20221021-212629-ladsgroup.json [production]
21:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
21:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
21:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T321312)', diff saved to https://phabricator.wikimedia.org/P35950 and previous config saved to /var/cache/conftool/dbconfig/20221021-212604-ladsgroup.json [production]
21:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P35949 and previous config saved to /var/cache/conftool/dbconfig/20221021-211058-ladsgroup.json [production]
20:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P35948 and previous config saved to /var/cache/conftool/dbconfig/20221021-205551-ladsgroup.json [production]
20:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T321312)', diff saved to https://phabricator.wikimedia.org/P35947 and previous config saved to /var/cache/conftool/dbconfig/20221021-204045-ladsgroup.json [production]
20:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1178 (T321312)', diff saved to https://phabricator.wikimedia.org/P35946 and previous config saved to /var/cache/conftool/dbconfig/20221021-203430-ladsgroup.json [production]
20:34 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
20:34 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
20:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T321312)', diff saved to https://phabricator.wikimedia.org/P35945 and previous config saved to /var/cache/conftool/dbconfig/20221021-203406-ladsgroup.json [production]
20:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T321312)', diff saved to https://phabricator.wikimedia.org/P35944 and previous config saved to /var/cache/conftool/dbconfig/20221021-202721-ladsgroup.json [production]
20:20 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply updates - bking@cumin2002 - T321310 [production]
20:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P35943 and previous config saved to /var/cache/conftool/dbconfig/20221021-201900-ladsgroup.json [production]
20:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P35942 and previous config saved to /var/cache/conftool/dbconfig/20221021-201214-ladsgroup.json [production]
20:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P35941 and previous config saved to /var/cache/conftool/dbconfig/20221021-200353-ladsgroup.json [production]
19:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P35940 and previous config saved to /var/cache/conftool/dbconfig/20221021-195708-ladsgroup.json [production]
19:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T321312)', diff saved to https://phabricator.wikimedia.org/P35939 and previous config saved to /var/cache/conftool/dbconfig/20221021-194847-ladsgroup.json [production]
19:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1177 (T321312)', diff saved to https://phabricator.wikimedia.org/P35938 and previous config saved to /var/cache/conftool/dbconfig/20221021-194234-ladsgroup.json [production]
19:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
19:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
19:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T321312)', diff saved to https://phabricator.wikimedia.org/P35937 and previous config saved to /var/cache/conftool/dbconfig/20221021-194210-ladsgroup.json [production]
19:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T321312)', diff saved to https://phabricator.wikimedia.org/P35936 and previous config saved to /var/cache/conftool/dbconfig/20221021-194201-ladsgroup.json [production]
19:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2181 (T321312)', diff saved to https://phabricator.wikimedia.org/P35935 and previous config saved to /var/cache/conftool/dbconfig/20221021-193550-ladsgroup.json [production]
19:35 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]
19:35 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]
19:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T321312)', diff saved to https://phabricator.wikimedia.org/P35934 and previous config saved to /var/cache/conftool/dbconfig/20221021-193524-ladsgroup.json [production]
19:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P35933 and previous config saved to /var/cache/conftool/dbconfig/20221021-192704-ladsgroup.json [production]
19:24 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet [production]