5901-5950 of 10000 results (114ms)
2023-03-22 §
10:36 <elukey@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:36 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:34 <elukey> `racadm racreset` for kafka-main2005 - http idrac not available (ssh on works fine) [production]
10:30 <elukey@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:29 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:27 <elukey@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:26 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main2005.codfw.wmnet [production]
10:23 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kafka-main2005.codfw.wmnet with reason: Stop kafka, update idrac/bios/nic-firmware [production]
10:22 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on kafka-main2005.codfw.wmnet with reason: Stop kafka, update idrac/bios/nic-firmware [production]
10:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1004.eqiad.wmnet with OS bullseye [production]
10:07 <stevemunene@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host an-test-client1002.eqiad.wmnet with OS bullseye [production]
09:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1004.eqiad.wmnet with reason: host reimage [production]
09:54 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1004.eqiad.wmnet with reason: host reimage [production]
09:38 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main1004.eqiad.wmnet with OS bullseye [production]
09:36 <elukey@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:27 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kafka-main1004.eqiad.wmnet [production]
09:27 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host kafka-main1004.eqiad.wmnet [production]
09:23 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:21 <elukey@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:12 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kafka-main1004.eqiad.wmnet [production]
09:12 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host kafka-main1004.eqiad.wmnet [production]
09:11 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:10 <elukey@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:02 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1004.eqiad.wmnet [production]
09:01 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1004.eqiad.wmnet with reason: Stop kafka, update idrac/bios/nic-firmware [production]
09:01 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on kafka-main1004.eqiad.wmnet with reason: Stop kafka, update idrac/bios/nic-firmware [production]
08:58 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pybal-test2003.codfw.wmnet with reason: Some tests with pybal/Bullseye [production]
08:58 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on pybal-test2003.codfw.wmnet with reason: Some tests with pybal/Bullseye [production]
08:52 <stevemunene@cumin1001> START - Cookbook sre.ganeti.reimage for host an-test-client1002.eqiad.wmnet with OS bullseye [production]
08:25 <elukey@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
08:25 <elukey@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
08:24 <XioNoX> deploy measure-$site.wikimedia.org CNAMES [production]
08:20 <elukey@deploy2002> helmfile [codfw] DONE helmfile.d/services/api-gateway: sync [production]
08:20 <elukey@deploy2002> helmfile [codfw] START helmfile.d/services/api-gateway: sync [production]
08:18 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/api-gateway: sync [production]
08:17 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/api-gateway: sync [production]
07:23 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 141082 [production]
07:22 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 141082 [production]
00:57 <zabe@deploy2002> Finished scap: update interwiki cache (duration: 07m 02s) [production]
00:50 <zabe@deploy2002> Started scap: update interwiki cache [production]
00:47 <zabe@deploy2002> Finished scap: T332115 (duration: 06m 56s) [production]
00:40 <zabe@deploy2002> Started scap: T332115 [production]
00:40 <zabe> create Wikipedia Angika (anpwiki) # T332115 [production]
00:38 <zabe@deploy2002> Finished scap: Backport for [[gerrit:901652|Add namespace translations for Angika (T332118)]], [[gerrit:901653|Add namespace translations for Angika (T332118)]], [[gerrit:901651|Add namespaces, linktrail and digit transform table for Angika (T332118)]] (duration: 27m 00s) [production]
00:29 <zabe@deploy2002> zabe: Backport for [[gerrit:901652|Add namespace translations for Angika (T332118)]], [[gerrit:901653|Add namespace translations for Angika (T332118)]], [[gerrit:901651|Add namespaces, linktrail and digit transform table for Angika (T332118)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
00:11 <zabe@deploy2002> Started scap: Backport for [[gerrit:901652|Add namespace translations for Angika (T332118)]], [[gerrit:901653|Add namespace translations for Angika (T332118)]], [[gerrit:901651|Add namespaces, linktrail and digit transform table for Angika (T332118)]] [production]
2023-03-21 §
23:46 <zabe@deploy2002> Finished scap: Backport for [[gerrit:901650|Add messages for Angika Wikipedia (anpwiki) (T332115)]], [[gerrit:901649|Add messages for Central Kurdish Wiktionary (ckbwiktionary) (T331831)]] (duration: 30m 08s) [production]
23:35 <zabe@deploy2002> zabe: Backport for [[gerrit:901650|Add messages for Angika Wikipedia (anpwiki) (T332115)]], [[gerrit:901649|Add messages for Central Kurdish Wiktionary (ckbwiktionary) (T331831)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
23:15 <zabe@deploy2002> Started scap: Backport for [[gerrit:901650|Add messages for Angika Wikipedia (anpwiki) (T332115)]], [[gerrit:901649|Add messages for Central Kurdish Wiktionary (ckbwiktionary) (T331831)]] [production]
23:07 <zabe@deploy2002> Finished scap: [[gerrit:901722|Revert "dewiki: Allow 'crats to remove sysopship and manage importers"]] (duration: 07m 10s) [production]