951-1000 of 10000 results (76ms)
2023-02-13 ยง
21:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
21:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T328255)', diff saved to https://phabricator.wikimedia.org/P44502 and previous config saved to /var/cache/conftool/dbconfig/20230213-211932-ladsgroup.json [production]
21:18 <taavi@deploy1002> taavi: lmowiktionary: Create extendedmover group (T327340) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:16 <taavi@deploy1002> Started scap: lmowiktionary: Create extendedmover group (T327340) [production]
21:15 <taavi@deploy1002> Backport cancelled. [production]
21:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1114 (T328817)', diff saved to https://phabricator.wikimedia.org/P44501 and previous config saved to /var/cache/conftool/dbconfig/20230213-210738-marostegui.json [production]
21:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1114.eqiad.wmnet with reason: Maintenance [production]
21:07 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1114.eqiad.wmnet with reason: Maintenance [production]
21:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1111 (T328817)', diff saved to https://phabricator.wikimedia.org/P44500 and previous config saved to /var/cache/conftool/dbconfig/20230213-210717-marostegui.json [production]
21:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T329203)', diff saved to https://phabricator.wikimedia.org/P44499 and previous config saved to /var/cache/conftool/dbconfig/20230213-210513-marostegui.json [production]
21:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P44498 and previous config saved to /var/cache/conftool/dbconfig/20230213-210426-ladsgroup.json [production]
20:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2168:3317 (T329203)', diff saved to https://phabricator.wikimedia.org/P44497 and previous config saved to /var/cache/conftool/dbconfig/20230213-205905-marostegui.json [production]
20:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance [production]
20:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance [production]
20:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159 (T329203)', diff saved to https://phabricator.wikimedia.org/P44496 and previous config saved to /var/cache/conftool/dbconfig/20230213-205855-marostegui.json [production]
20:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P44495 and previous config saved to /var/cache/conftool/dbconfig/20230213-205211-marostegui.json [production]
20:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P44494 and previous config saved to /var/cache/conftool/dbconfig/20230213-204920-ladsgroup.json [production]
20:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P44493 and previous config saved to /var/cache/conftool/dbconfig/20230213-204348-marostegui.json [production]
20:39 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
20:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P44492 and previous config saved to /var/cache/conftool/dbconfig/20230213-203704-marostegui.json [production]
20:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T328255)', diff saved to https://phabricator.wikimedia.org/P44491 and previous config saved to /var/cache/conftool/dbconfig/20230213-203413-ladsgroup.json [production]
20:32 <dcausse> restarting blazegraph on wdqs1012 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
20:30 <cmooney@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
20:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P44490 and previous config saved to /var/cache/conftool/dbconfig/20230213-202842-marostegui.json [production]
20:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2109 (T328255)', diff saved to https://phabricator.wikimedia.org/P44489 and previous config saved to /var/cache/conftool/dbconfig/20230213-202656-ladsgroup.json [production]
20:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
20:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
20:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T328255)', diff saved to https://phabricator.wikimedia.org/P44488 and previous config saved to /var/cache/conftool/dbconfig/20230213-202635-ladsgroup.json [production]
20:24 <cmooney@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
20:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1111 (T328817)', diff saved to https://phabricator.wikimedia.org/P44487 and previous config saved to /var/cache/conftool/dbconfig/20230213-202157-marostegui.json [production]
20:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159 (T329203)', diff saved to https://phabricator.wikimedia.org/P44486 and previous config saved to /var/cache/conftool/dbconfig/20230213-201336-marostegui.json [production]
20:13 <cmooney@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
20:12 <elukey@cumin1001> END (PASS) - Cookbook sre.k8s.upgrade-cluster (exit_code=0) Upgrade K8s version: Upgrade ml-staging-codfw cluster to 1.23 [production]
20:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-staging2002.codfw.wmnet with OS bullseye [production]
20:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P44485 and previous config saved to /var/cache/conftool/dbconfig/20230213-201129-ladsgroup.json [production]
20:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2159 (T329203)', diff saved to https://phabricator.wikimedia.org/P44484 and previous config saved to /var/cache/conftool/dbconfig/20230213-200742-marostegui.json [production]
20:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]
20:07 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]
20:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance [production]
20:07 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance [production]
20:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2150 (T329203)', diff saved to https://phabricator.wikimedia.org/P44483 and previous config saved to /var/cache/conftool/dbconfig/20230213-200654-marostegui.json [production]
19:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1111 (T328817)', diff saved to https://phabricator.wikimedia.org/P44482 and previous config saved to /var/cache/conftool/dbconfig/20230213-195743-marostegui.json [production]
19:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1111.eqiad.wmnet with reason: Maintenance [production]
19:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1111.eqiad.wmnet with reason: Maintenance [production]
19:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1104 (T328817)', diff saved to https://phabricator.wikimedia.org/P44481 and previous config saved to /var/cache/conftool/dbconfig/20230213-195722-marostegui.json [production]
19:56 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
19:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P44480 and previous config saved to /var/cache/conftool/dbconfig/20230213-195623-ladsgroup.json [production]
19:56 <cmooney@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1001.eqiad.wmnet'] [production]
16:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P44436 and previous config saved to /var/cache/conftool/dbconfig/20230213-162456-marostegui.json [production]
16:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2438.mgmt.codfw.wmnet with reboot policy FORCED [production]