7551-7600 of 10000 results (97ms)
2022-12-07 ยง
09:59 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 8932 [production]
09:59 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13150 [production]
09:59 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 13150 [production]
09:53 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 138064 [production]
09:52 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 138064 [production]
09:52 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 16276 [production]
09:51 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 16276 [production]
09:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 32098 [production]
09:46 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 32098 [production]
09:45 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 31800 [production]
09:44 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 31800 [production]
09:44 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 31800 [production]
09:42 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 31800 [production]
09:42 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 45430 [production]
09:41 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 45430 [production]
09:41 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 7568 [production]
09:40 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 7568 [production]
09:37 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:36 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
09:36 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
09:36 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
09:36 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:35 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
09:35 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:35 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:34 <jiji@deploy1002> Finished scap: Backport for [[gerrit:865117|ProductionServices: Use redis_misc servers for LockManager (1/6) (T267581)]] (duration: 09m 08s) [production]
09:27 <jiji@deploy1002> jiji and jiji: Backport for [[gerrit:865117|ProductionServices: Use redis_misc servers for LockManager (1/6) (T267581)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
09:25 <jiji@deploy1002> Started scap: Backport for [[gerrit:865117|ProductionServices: Use redis_misc servers for LockManager (1/6) (T267581)]] [production]
09:23 <jiji@deploy1002> backport aborted: (duration: 00m 18s) [production]
09:17 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 395570 [production]
09:17 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 395570 [production]
09:14 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 397715 [production]
09:13 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 397715 [production]
09:12 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 42 [production]
09:11 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1108.eqiad.wmnet [production]
09:10 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1108.eqiad.wmnet [production]
09:10 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 42 [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 100%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42439 and previous config saved to /var/cache/conftool/dbconfig/20221207-071831-root.json [production]
07:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 75%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42438 and previous config saved to /var/cache/conftool/dbconfig/20221207-070326-root.json [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 50%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42437 and previous config saved to /var/cache/conftool/dbconfig/20221207-064821-root.json [production]
06:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 25%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42436 and previous config saved to /var/cache/conftool/dbconfig/20221207-063316-root.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 10%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42435 and previous config saved to /var/cache/conftool/dbconfig/20221207-061810-root.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 5%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42434 and previous config saved to /var/cache/conftool/dbconfig/20221207-060305-root.json [production]
05:58 <marostegui> Drop phab1001 grants from m3 databases T323418 [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1206 (re)pooling @ 1%: Testing new RAID controller', diff saved to https://phabricator.wikimedia.org/P42433 and previous config saved to /var/cache/conftool/dbconfig/20221207-054759-root.json [production]
03:59 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1011.eqiad.wmnet with OS bullseye [production]
03:18 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1011.eqiad.wmnet with reason: host reimage [production]
03:15 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1011.eqiad.wmnet with reason: host reimage [production]
02:49 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1011.eqiad.wmnet with OS bullseye [production]
02:39 <cwhite@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['logstash1011'] [production]