5701-5750 of 10000 results (91ms)
2022-12-07 §
09:10 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1108.eqiad.wmnet [production]
09:10 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 42 [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 100%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42439 and previous config saved to /var/cache/conftool/dbconfig/20221207-071831-root.json [production]
07:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 75%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42438 and previous config saved to /var/cache/conftool/dbconfig/20221207-070326-root.json [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 50%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42437 and previous config saved to /var/cache/conftool/dbconfig/20221207-064821-root.json [production]
06:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 25%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42436 and previous config saved to /var/cache/conftool/dbconfig/20221207-063316-root.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 10%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42435 and previous config saved to /var/cache/conftool/dbconfig/20221207-061810-root.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 5%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42434 and previous config saved to /var/cache/conftool/dbconfig/20221207-060305-root.json [production]
05:58 <marostegui> Drop phab1001 grants from m3 databases T323418 [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 1%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42433 and previous config saved to /var/cache/conftool/dbconfig/20221207-054759-root.json [production]
03:59 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1011.eqiad.wmnet with OS bullseye [production]
03:18 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1011.eqiad.wmnet with reason: host reimage [production]
03:15 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1011.eqiad.wmnet with reason: host reimage [production]
02:49 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1011.eqiad.wmnet with OS bullseye [production]
02:39 <cwhite@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['logstash1011'] [production]
02:33 <cwhite@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['logstash1011'] [production]
00:21 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1024.eqiad.wmnet with OS bullseye [production]
00:18 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1023.eqiad.wmnet with OS bullseye [production]
2022-12-06 §
23:08 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1024.eqiad.wmnet with OS bullseye [production]
23:06 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1023.eqiad.wmnet with OS bullseye [production]
23:04 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1024.eqiad.wmnet with OS bullseye [production]
22:52 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes1023.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:40 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host kubernetes1023.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:39 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:39 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubernetes1023 - cmjohnson@cumin1001" [production]
22:37 <tgr_> UTC late backports done [production]
22:36 <cmjohnson@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubernetes1023 - cmjohnson@cumin1001" [production]
22:30 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
22:26 <tgr@deploy1002> Finished scap: Backport for [[gerrit:865131|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]], [[gerrit:865130|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]] (duration: 18m 58s) [production]
22:09 <tgr@deploy1002> tgr and tgr: Backport for [[gerrit:865131|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]], [[gerrit:865130|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
22:07 <tgr@deploy1002> Started scap: Backport for [[gerrit:865131|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]], [[gerrit:865130|Fix UserDatabaseHelper::hasMainspaceEdits() (T324285)]] [production]
21:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1024.eqiad.wmnet with OS bullseye [production]
21:12 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cephosd1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:07 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1029.eqiad.wmnet with OS bullseye [production]
20:49 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cephosd1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:45 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1029.eqiad.wmnet with reason: host reimage [production]
20:42 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1029.eqiad.wmnet with reason: host reimage [production]
20:25 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1029.eqiad.wmnet with OS bullseye [production]
20:25 <cwhite@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logstash1029.eqiad.wmnet with OS bullseye [production]
20:24 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1029.eqiad.wmnet with OS bullseye [production]
20:24 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1028.eqiad.wmnet with OS bullseye [production]
20:22 <cwhite@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['logstash1029'] [production]
20:16 <cwhite@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['logstash1029'] [production]
20:13 <cwhite@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['logstash1029'] [production]
20:06 <eileen> civicrm upgraded from c9761fee to 3ae68ab4 [production]
20:06 <cwhite@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['logstash1029'] [production]
20:01 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1028.eqiad.wmnet with reason: host reimage [production]
20:00 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bullseye [production]
19:57 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1028.eqiad.wmnet with reason: host reimage [production]
19:56 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bullseye [production]