1901-1950 of 10000 results (76ms)
2023-02-08 ยง
07:38 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
07:38 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
07:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43776 and previous config saved to /var/cache/conftool/dbconfig/20230208-073816-marostegui.json [production]
07:38 <marostegui> dbmaint deploy schema change on s2 eqiad (with replication) T328807 T328828 [production]
07:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P43775 and previous config saved to /var/cache/conftool/dbconfig/20230208-072310-marostegui.json [production]
07:19 <marostegui> dbmaint deploy schema change on s5 eqiad (with replication) T328807 T328828 [production]
07:18 <marostegui> dbmaint deploy schema change on s4 eqiad (with replication) T328807 T328828 [production]
07:18 <marostegui> dbmaint deploy schema change on s8 eqiad (with replication) T328807 T328828 [production]
07:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P43774 and previous config saved to /var/cache/conftool/dbconfig/20230208-070803-marostegui.json [production]
07:07 <marostegui> Install 10.6.12 on pc2014 T329011 [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43773 and previous config saved to /var/cache/conftool/dbconfig/20230208-065257-marostegui.json [production]
06:52 <marostegui@deploy1002> Finished scap: Backport for [[gerrit:887445|ProductionServices.php: Promote pc2011 back to pc1 master]] (duration: 16m 01s) [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43772 and previous config saved to /var/cache/conftool/dbconfig/20230208-065149-marostegui.json [production]
06:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
06:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
06:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
06:48 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
06:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
06:48 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43771 and previous config saved to /var/cache/conftool/dbconfig/20230208-064405-root.json [production]
06:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43770 and previous config saved to /var/cache/conftool/dbconfig/20230208-064134-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43769 and previous config saved to /var/cache/conftool/dbconfig/20230208-064027-marostegui.json [production]
06:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
06:40 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
06:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
06:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
06:39 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
06:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
06:38 <marostegui@deploy1002> marostegui: Backport for [[gerrit:887445|ProductionServices.php: Promote pc2011 back to pc1 master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
06:36 <marostegui@deploy1002> Started scap: Backport for [[gerrit:887445|ProductionServices.php: Promote pc2011 back to pc1 master]] [production]
03:58 <AndyRussG> payments-wiki upgraded from 53d1a58d to 61ea310d, config revision changed from 5e707565 to 4d90c211 [production]
02:58 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2420.codfw.wmnet with OS buster [production]
01:43 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host mw2420.codfw.wmnet with OS buster [production]
01:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2435'] [production]
01:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2434'] [production]
01:00 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2435'] [production]
01:00 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2433'] [production]
01:00 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2434'] [production]
00:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2432'] [production]
00:52 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2433'] [production]
00:52 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2432'] [production]
00:50 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2431'] [production]
00:49 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2430'] [production]
00:43 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2431'] [production]
00:43 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2430'] [production]
00:39 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2429'] [production]
00:39 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2428'] [production]
00:32 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2429'] [production]
00:32 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2427'] [production]
00:32 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2428'] [production]