3451-3500 of 10000 results (105ms)
2024-07-08 ยง
16:15 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1201 (T367781)', diff saved to https://phabricator.wikimedia.org/P65966 and previous config saved to /var/cache/conftool/dbconfig/20240708-161510-arnaudb.json [production]
16:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1201 (T367781)', diff saved to https://phabricator.wikimedia.org/P65965 and previous config saved to /var/cache/conftool/dbconfig/20240708-161302-arnaudb.json [production]
16:12 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
16:12 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
16:12 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T367781)', diff saved to https://phabricator.wikimedia.org/P65964 and previous config saved to /var/cache/conftool/dbconfig/20240708-161238-arnaudb.json [production]
16:09 <dcaro@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1011.eqiad.wmnet [production]
16:08 <root@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1011.eqiad.wmnet with OS bullseye [production]
15:57 <fabfur@cumin1002> cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet [production]
15:57 <fabfur@cumin1002> cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet [production]
15:57 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P65963 and previous config saved to /var/cache/conftool/dbconfig/20240708-155731-arnaudb.json [production]
15:51 <jdrewniak@deploy1002> Synchronized portals: Wikimedia Portals Update: [[gerrit:1046698| Bumping portals to master (T128546)]] (duration: 06m 28s) [production]
15:47 <pfischer@deploy1002> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:46 <pfischer@deploy1002> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:45 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:45 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:45 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:45 <swfrench@deploy1002> helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply [production]
15:44 <jdrewniak@deploy1002> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:1046698| Bumping portals to master (T128546)]] (duration: 07m 54s) [production]
15:44 <swfrench@deploy1002> helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply [production]
15:44 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:42 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P65962 and previous config saved to /var/cache/conftool/dbconfig/20240708-154224-arnaudb.json [production]
15:38 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
15:38 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
15:27 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T367781)', diff saved to https://phabricator.wikimedia.org/P65961 and previous config saved to /var/cache/conftool/dbconfig/20240708-152717-arnaudb.json [production]
15:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1187 (T367781)', diff saved to https://phabricator.wikimedia.org/P65960 and previous config saved to /var/cache/conftool/dbconfig/20240708-152508-arnaudb.json [production]
15:25 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
15:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
15:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T367781)', diff saved to https://phabricator.wikimedia.org/P65959 and previous config saved to /var/cache/conftool/dbconfig/20240708-152446-arnaudb.json [production]
15:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Bumping db1227 weight (T366852)', diff saved to https://phabricator.wikimedia.org/P65958 and previous config saved to /var/cache/conftool/dbconfig/20240708-152222-ladsgroup.json [production]
15:16 <root@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1011.eqiad.wmnet with reason: host reimage [production]
15:13 <root@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1011.eqiad.wmnet with reason: host reimage [production]
15:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P65957 and previous config saved to /var/cache/conftool/dbconfig/20240708-150939-arnaudb.json [production]
14:59 <root@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd1011.eqiad.wmnet with OS bullseye [production]
14:57 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1002.eqiad.wmnet [production]
14:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P65956 and previous config saved to /var/cache/conftool/dbconfig/20240708-145432-arnaudb.json [production]
14:53 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host search-loader1002.eqiad.wmnet [production]
14:53 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host search-loader1002.eqiad.wmnet [production]
14:53 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host search-loader1002.eqiad.wmnet [production]
14:52 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host search-loader1002.eqiad.wmnet [production]
14:51 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host search-loader1002.eqiad.wmnet [production]
14:51 <claime> cleaning up old shellbox files on mw1438 [production]
14:43 <root@cumin1002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cloudcephosd1011.eqiad.wmnet [production]
14:43 <root@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1011.eqiad.wmnet [production]
14:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T367781)', diff saved to https://phabricator.wikimedia.org/P65955 and previous config saved to /var/cache/conftool/dbconfig/20240708-143925-arnaudb.json [production]
14:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1180 (T367781)', diff saved to https://phabricator.wikimedia.org/P65954 and previous config saved to /var/cache/conftool/dbconfig/20240708-143716-arnaudb.json [production]
14:37 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
14:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
14:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T367781)', diff saved to https://phabricator.wikimedia.org/P65953 and previous config saved to /var/cache/conftool/dbconfig/20240708-143654-arnaudb.json [production]
14:34 <root@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1011.eqiad.wmnet [production]
14:31 <root@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1011.eqiad.wmnet [production]