2001-2050 of 10000 results (92ms)
2023-09-29 §
01:40 <ejegg> payments-wiki upgraded from c4c9b938 to d6ad0376 [production]
01:33 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P52754 and previous config saved to /var/cache/conftool/dbconfig/20230929-013319-arnaudb.json [production]
01:18 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P52753 and previous config saved to /var/cache/conftool/dbconfig/20230929-011813-arnaudb.json [production]
01:03 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T343198)', diff saved to https://phabricator.wikimedia.org/P52752 and previous config saved to /var/cache/conftool/dbconfig/20230929-010306-arnaudb.json [production]
00:26 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1227.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:24 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db1227.mgmt.eqiad.wmnet with reboot policy FORCED [production]
2023-09-28 §
23:50 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2180 (T343198)', diff saved to https://phabricator.wikimedia.org/P52751 and previous config saved to /var/cache/conftool/dbconfig/20230928-235053-arnaudb.json [production]
23:50 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance [production]
23:50 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance [production]
23:50 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T343198)', diff saved to https://phabricator.wikimedia.org/P52750 and previous config saved to /var/cache/conftool/dbconfig/20230928-235032-arnaudb.json [production]
23:42 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2119 (T343198)', diff saved to https://phabricator.wikimedia.org/P52749 and previous config saved to /var/cache/conftool/dbconfig/20230928-234246-arnaudb.json [production]
23:42 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance [production]
23:42 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance [production]
23:42 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110 (T343198)', diff saved to https://phabricator.wikimedia.org/P52748 and previous config saved to /var/cache/conftool/dbconfig/20230928-234224-arnaudb.json [production]
23:35 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P52747 and previous config saved to /var/cache/conftool/dbconfig/20230928-233525-arnaudb.json [production]
23:27 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P52746 and previous config saved to /var/cache/conftool/dbconfig/20230928-232718-arnaudb.json [production]
23:20 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P52745 and previous config saved to /var/cache/conftool/dbconfig/20230928-232019-arnaudb.json [production]
23:12 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P52744 and previous config saved to /var/cache/conftool/dbconfig/20230928-231211-arnaudb.json [production]
23:05 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T343198)', diff saved to https://phabricator.wikimedia.org/P52743 and previous config saved to /var/cache/conftool/dbconfig/20230928-230512-arnaudb.json [production]
22:57 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110 (T343198)', diff saved to https://phabricator.wikimedia.org/P52742 and previous config saved to /var/cache/conftool/dbconfig/20230928-225705-arnaudb.json [production]
22:40 <wfan> payments-wiki change from c4c9b938 to 20828b07 [production]
22:03 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices2005-dev.codfw.wmnet with reason: host reimage [production]
22:02 <bking@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1007.eqiad.wmnet with OS bullseye [production]
22:00 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices2005-dev.codfw.wmnet with reason: host reimage [production]
21:58 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1033.eqiad.wmnet [production]
21:58 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1033.eqiad.wmnet [production]
21:58 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1030.eqiad.wmnet [production]
21:57 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1030.eqiad.wmnet [production]
21:57 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1027.eqiad.wmnet [production]
21:57 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1027.eqiad.wmnet [production]
21:57 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1026.eqiad.wmnet [production]
21:57 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1026.eqiad.wmnet [production]
21:56 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1025.eqiad.wmnet [production]
21:56 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1025.eqiad.wmnet [production]
21:56 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1032.eqiad.wmnet [production]
21:56 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1032.eqiad.wmnet [production]
21:55 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1029.eqiad.wmnet [production]
21:55 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1029.eqiad.wmnet [production]
21:55 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1029.eqiad.wmnet [production]
21:55 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1029.eqiad.wmnet [production]
21:54 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1024.eqiad.wmnet [production]
21:54 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1024.eqiad.wmnet [production]
21:54 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1023.eqiad.wmnet [production]
21:54 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1023.eqiad.wmnet [production]
21:54 <eevans@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts restbase1022.eqiad.wmnet [production]
21:54 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1022.eqiad.wmnet [production]
21:53 <wfan> payments-wiki change from 505a616d to 20828b07 [production]
21:52 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host cloudelastic1007.eqiad.wmnet with OS bullseye [production]
21:42 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts restbase1031.eqiad.wmnet [production]
21:42 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts restbase1031.eqiad.wmnet [production]