2201-2250 of 10000 results (77ms)
2023-02-09 ยง
12:32 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage [production]
12:31 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc-gp1001.eqiad.wmnet with reason: host reimage [production]
12:27 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage [production]
12:22 <phedenskog@deploy1002> Finished deploy [performance/navtiming@bb224a1]: (no justification provided) (duration: 00m 08s) [production]
12:22 <phedenskog@deploy1002> Started deploy [performance/navtiming@bb224a1]: (no justification provided) [production]
12:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T329203)', diff saved to https://phabricator.wikimedia.org/P43997 and previous config saved to /var/cache/conftool/dbconfig/20230209-122036-marostegui.json [production]
12:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2178 (T328817)', diff saved to https://phabricator.wikimedia.org/P43996 and previous config saved to /var/cache/conftool/dbconfig/20230209-121923-marostegui.json [production]
12:18 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc-gp1001.eqiad.wmnet with OS buster [production]
12:17 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move Babel settings from IS.php to ext-Babel.php, part III (T308932) (duration: 06m 47s) [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2178 (T328817)', diff saved to https://phabricator.wikimedia.org/P43995 and previous config saved to /var/cache/conftool/dbconfig/20230209-121705-marostegui.json [production]
12:17 <jiji@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
12:16 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance [production]
12:16 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance [production]
12:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43994 and previous config saved to /var/cache/conftool/dbconfig/20230209-121644-marostegui.json [production]
12:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2124 (T329203)', diff saved to https://phabricator.wikimedia.org/P43993 and previous config saved to /var/cache/conftool/dbconfig/20230209-121507-marostegui.json [production]
12:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
12:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
12:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T329203)', diff saved to https://phabricator.wikimedia.org/P43992 and previous config saved to /var/cache/conftool/dbconfig/20230209-121446-marostegui.json [production]
12:12 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host puppetdb2003.codfw.wmnet with OS bullseye [production]
12:10 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetdb1003.eqiad.wmnet with OS bullseye [production]
12:10 <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move Babel settings from IS.php to ext-Babel.php, part II (T308932) (duration: 06m 40s) [production]
12:06 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:06 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:03 <ladsgroup@deploy1002> Synchronized wmf-config/ext-Babel.php: Move Babel settings from IS.php to ext-Babel.php, part I (T308932) (duration: 07m 06s) [production]
12:02 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1099.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:02 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1099.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:02 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1098.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:02 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1098.eqiad.wmnet with reason: Attempting to move some GPUs [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P43991 and previous config saved to /var/cache/conftool/dbconfig/20230209-120138-marostegui.json [production]
11:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P43990 and previous config saved to /var/cache/conftool/dbconfig/20230209-115940-marostegui.json [production]
11:57 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:57 <jiji@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:55 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb1003.eqiad.wmnet with reason: host reimage [production]
11:53 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb1003.eqiad.wmnet with reason: host reimage [production]
11:52 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:52 <jiji@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P43989 and previous config saved to /var/cache/conftool/dbconfig/20230209-114632-marostegui.json [production]
11:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P43988 and previous config saved to /var/cache/conftool/dbconfig/20230209-114434-marostegui.json [production]
11:40 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host puppetdb1003.eqiad.wmnet with OS bullseye [production]
11:34 <marostegui> Stop mariadb on db1098 (s6 and s7) T329171 [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43986 and previous config saved to /var/cache/conftool/dbconfig/20230209-113125-marostegui.json [production]
11:31 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1003.eqiad.wmnet with OS bullseye [production]
11:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T329203)', diff saved to https://phabricator.wikimedia.org/P43985 and previous config saved to /var/cache/conftool/dbconfig/20230209-112927-marostegui.json [production]
11:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2171:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43984 and previous config saved to /var/cache/conftool/dbconfig/20230209-112748-marostegui.json [production]
11:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]
11:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]
11:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2157 (T328817)', diff saved to https://phabricator.wikimedia.org/P43983 and previous config saved to /var/cache/conftool/dbconfig/20230209-112727-marostegui.json [production]
11:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T329203)', diff saved to https://phabricator.wikimedia.org/P43982 and previous config saved to /var/cache/conftool/dbconfig/20230209-112359-marostegui.json [production]
11:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
11:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]