251-300 of 10000 results (28ms)
2022-02-22 ยง
11:00 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1002.eqiad.wmnet with reason: host reimage [production]
10:59 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1093.eqiad.wmnet with reason: host reimage [production]
10:56 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1110 (T300774)', diff saved to https://phabricator.wikimedia.org/P21277 and previous config saved to /var/cache/conftool/dbconfig/20220222-105653-kormat.json [production]
10:56 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1002.eqiad.wmnet with reason: host reimage [production]
10:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P21276 and previous config saved to /var/cache/conftool/dbconfig/20220222-105632-marostegui.json [production]
10:56 <Lucas_WMDE> Deployed patch for T302192 [production]
10:48 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1093.eqiad.wmnet with OS bullseye [production]
10:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P21275 and previous config saved to /var/cache/conftool/dbconfig/20220222-104613-ladsgroup.json [production]
10:43 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-serve1002.eqiad.wmnet with OS bullseye [production]
10:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P21274 and previous config saved to /var/cache/conftool/dbconfig/20220222-104128-marostegui.json [production]
10:36 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1001.eqiad.wmnet with OS bullseye [production]
10:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P21273 and previous config saved to /var/cache/conftool/dbconfig/20220222-103109-ladsgroup.json [production]
10:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1182 (T300381)', diff saved to https://phabricator.wikimedia.org/P21272 and previous config saved to /var/cache/conftool/dbconfig/20220222-102623-marostegui.json [production]
10:24 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1001.eqiad.wmnet with reason: host reimage [production]
10:20 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1001.eqiad.wmnet with reason: host reimage [production]
10:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1182 (T300381)', diff saved to https://phabricator.wikimedia.org/P21271 and previous config saved to /var/cache/conftool/dbconfig/20220222-101710-marostegui.json [production]
10:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
10:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
10:16 <kormat@cumin1001> dbctl commit (dc=all): 'Depooling db1110 (T300774)', diff saved to https://phabricator.wikimedia.org/P21270 and previous config saved to /var/cache/conftool/dbconfig/20220222-101649-kormat.json [production]
10:16 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance [production]
10:16 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance [production]
10:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T302185)', diff saved to https://phabricator.wikimedia.org/P21269 and previous config saved to /var/cache/conftool/dbconfig/20220222-101604-ladsgroup.json [production]
10:12 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet [production]
10:07 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-serve1001.eqiad.wmnet with OS bullseye [production]
10:02 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1099.eqiad.wmnet with OS bullseye [production]
10:00 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet [production]
09:52 <XioNoX> restarting cr2-drmrs for software upgrade [production]
09:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1099.eqiad.wmnet with reason: host reimage [production]
09:47 <aqu@deploy1002> Finished deploy [analytics/refinery@ed5c9f9] (hadoop-test): Migrate aqs/hourly to Airflow TEST [analytics/refinery@ed5c9f9] (duration: 00m 03s) [production]
09:47 <aqu@deploy1002> Started deploy [analytics/refinery@ed5c9f9] (hadoop-test): Migrate aqs/hourly to Airflow TEST [analytics/refinery@ed5c9f9] [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T300381)', diff saved to https://phabricator.wikimedia.org/P21268 and previous config saved to /var/cache/conftool/dbconfig/20220222-094740-marostegui.json [production]
09:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1099.eqiad.wmnet with reason: host reimage [production]
09:43 <jayme@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:38 <aqu> Deploying analytics/refinery on hadoop-test only. [production]
09:38 <jayme@cumin1001> START - Cookbook sre.dns.netbox [production]
09:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.reimage for host db1099.eqiad.wmnet with OS bullseye [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P21267 and previous config saved to /var/cache/conftool/dbconfig/20220222-093235-marostegui.json [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P21266 and previous config saved to /var/cache/conftool/dbconfig/20220222-091730-marostegui.json [production]
09:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
09:04 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
09:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
09:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T300381)', diff saved to https://phabricator.wikimedia.org/P21265 and previous config saved to /var/cache/conftool/dbconfig/20220222-090226-marostegui.json [production]
08:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3318 (T302185)', diff saved to https://phabricator.wikimedia.org/P21264 and previous config saved to /var/cache/conftool/dbconfig/20220222-085835-ladsgroup.json [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1129 (T300381)', diff saved to https://phabricator.wikimedia.org/P21263 and previous config saved to /var/cache/conftool/dbconfig/20220222-085752-marostegui.json [production]
08:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
08:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
08:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3311 (T302185)', diff saved to https://phabricator.wikimedia.org/P21262 and previous config saved to /var/cache/conftool/dbconfig/20220222-085653-ladsgroup.json [production]
08:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
08:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]