2951-3000 of 10000 results (105ms)
2024-07-08 ยง
22:26 <bking@cumin2002> START - Cookbook sre.wdqs.reboot [production]
22:22 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P66005 and previous config saved to /var/cache/conftool/dbconfig/20240708-222234-arnaudb.json [production]
22:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P66004 and previous config saved to /var/cache/conftool/dbconfig/20240708-220727-arnaudb.json [production]
21:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169 (T367781)', diff saved to https://phabricator.wikimedia.org/P66003 and previous config saved to /var/cache/conftool/dbconfig/20240708-215220-arnaudb.json [production]
21:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2169 (T367781)', diff saved to https://phabricator.wikimedia.org/P66002 and previous config saved to /var/cache/conftool/dbconfig/20240708-214954-arnaudb.json [production]
21:49 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
21:49 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
21:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T367781)', diff saved to https://phabricator.wikimedia.org/P66001 and previous config saved to /var/cache/conftool/dbconfig/20240708-214932-arnaudb.json [production]
21:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P66000 and previous config saved to /var/cache/conftool/dbconfig/20240708-213425-arnaudb.json [production]
21:24 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:23 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P65999 and previous config saved to /var/cache/conftool/dbconfig/20240708-211918-arnaudb.json [production]
21:16 <catrope@deploy1002> Finished scap: Backport for [[gerrit:1052285|Enable VisualEditor by default on Italian Wikibooks (T369342)]] (duration: 09m 23s) [production]
21:10 <catrope@deploy1002> catrope, nmw03: Continuing with sync [production]
21:09 <catrope@deploy1002> catrope, nmw03: Backport for [[gerrit:1052285|Enable VisualEditor by default on Italian Wikibooks (T369342)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:06 <catrope@deploy1002> Started scap sync-world: Backport for [[gerrit:1052285|Enable VisualEditor by default on Italian Wikibooks (T369342)]] [production]
21:05 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic109[3-5]* for T348977 - bking@cumin2002 [production]
21:05 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic109[3-5]* for T348977 - bking@cumin2002 [production]
21:05 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1093-1095].eqiad.wmnet with reason: T348977 [production]
21:05 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1093-1095].eqiad.wmnet with reason: T348977 [production]
21:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T367781)', diff saved to https://phabricator.wikimedia.org/P65998 and previous config saved to /var/cache/conftool/dbconfig/20240708-210410-arnaudb.json [production]
21:02 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1023.eqiad.wmnet [production]
21:02 <fabfur@cumin1002> cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet [production]
21:01 <fabfur@cumin1002> cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet [production]
21:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2158 (T367781)', diff saved to https://phabricator.wikimedia.org/P65997 and previous config saved to /var/cache/conftool/dbconfig/20240708-210144-arnaudb.json [production]
21:01 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
21:01 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
21:01 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
21:01 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
21:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T367781)', diff saved to https://phabricator.wikimedia.org/P65996 and previous config saved to /var/cache/conftool/dbconfig/20240708-210106-arnaudb.json [production]
20:55 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host wdqs1023.eqiad.wmnet [production]
20:52 <catrope@deploy1002> Finished scap: Backport for [[gerrit:1051853|Graph extension: Add tracking for data sources used in <graph> tags]] (duration: 13m 00s) [production]
20:50 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1022.eqiad.wmnet [production]
20:47 <catrope@deploy1002> catrope: Continuing with sync [production]
20:45 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P65995 and previous config saved to /var/cache/conftool/dbconfig/20240708-204559-arnaudb.json [production]
20:43 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host wdqs1022.eqiad.wmnet [production]
20:42 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [production]
20:42 <catrope@deploy1002> catrope: Backport for [[gerrit:1051853|Graph extension: Add tracking for data sources used in <graph> tags]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1230 (T367856)', diff saved to https://phabricator.wikimedia.org/P65994 and previous config saved to /var/cache/conftool/dbconfig/20240708-204042-marostegui.json [production]
20:40 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance [production]
20:40 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance [production]
20:39 <catrope@deploy1002> Started scap sync-world: Backport for [[gerrit:1051853|Graph extension: Add tracking for data sources used in <graph> tags]] [production]
20:38 <bking@cumin2002> START - Cookbook sre.wdqs.reboot [production]
20:35 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [production]
20:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P65993 and previous config saved to /var/cache/conftool/dbconfig/20240708-203052-arnaudb.json [production]
20:28 <bking@cumin2002> START - Cookbook sre.wdqs.reboot [production]
20:27 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [production]
20:15 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T367781)', diff saved to https://phabricator.wikimedia.org/P65992 and previous config saved to /var/cache/conftool/dbconfig/20240708-201545-arnaudb.json [production]
20:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2151 (T367781)', diff saved to https://phabricator.wikimedia.org/P65991 and previous config saved to /var/cache/conftool/dbconfig/20240708-201318-arnaudb.json [production]
20:13 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]