7901-7950 of 10000 results (115ms)
2023-11-28 ยง
11:41 <kamila@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
11:41 <kamila@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
11:33 <volans@cumin1001> END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox interface ID cr2-esams:xe-0/1/2 [production]
11:33 <volans@cumin1001> START - Cookbook sre.network.debug for Netbox interface ID cr2-esams:xe-0/1/2 [production]
11:22 <kamila@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
11:22 <kamila@deploy2002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
11:21 <kamila@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
11:21 <kamila@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
10:52 <vgutierrez> depool ncredir4001 [production]
10:45 <vgutierrez> repool ncredir4001 [production]
10:38 <klausman@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:37 <moritzm> installing lua5.3 security updates [production]
10:35 <vgutierrez> depool ncredir4001 [production]
10:21 <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008, effectively disabling IPIP encapsulation on ncredir@ulsfo - T351069 [production]
10:09 <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008, effectively enabling IPIP encapsulation on ncredir@ulsfo - T351069 [production]
10:01 <sg912@deploy2002> Finished deploy [airflow-dags/analytics@0283c11]: (no justification provided) (duration: 00m 47s) [production]
10:00 <sg912@deploy2002> Started deploy [airflow-dags/analytics@0283c11]: (no justification provided) [production]
09:58 <moritzm> installing intel-microcode security updates [production]
09:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet [production]
09:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
09:40 <hashar@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.7 refs T350083 [production]
09:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1024.eqiad.wmnet with OS bookworm [production]
08:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1024.eqiad.wmnet with reason: host reimage [production]
08:52 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1024.eqiad.wmnet with reason: host reimage [production]
08:47 <hashar@deploy2002> Finished scap: Backport for [[gerrit:977625|Revert "Parsoid DataAccess: Stop processing extensions as top-level docs"]] (duration: 07m 54s) [production]
08:41 <hashar@deploy2002> hashar and ssastry: Continuing with sync [production]
08:41 <hashar@deploy2002> hashar and ssastry: Backport for [[gerrit:977625|Revert "Parsoid DataAccess: Stop processing extensions as top-level docs"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:39 <hashar@deploy2002> Started scap: Backport for [[gerrit:977625|Revert "Parsoid DataAccess: Stop processing extensions as top-level docs"]] [production]
08:37 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1024.eqiad.wmnet with OS bookworm [production]
07:39 <XioNoX> add RPKI ROA for 193.46.90.0/24 - T309297 [production]
04:56 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.4 (duration: 02m 14s) [production]
04:53 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.7 refs T350083 (duration: 51m 11s) [production]
04:21 <eileen> civicrm upgraded from f3de1778 to c2eaa50e [production]
04:02 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.7 refs T350083 [production]
01:46 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:42 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:42 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:40 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
01:34 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic2100'] [production]
01:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic2091'] [production]
01:27 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2100'] [production]
01:26 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic2100.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:24 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2091'] [production]
01:24 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic2091.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:18 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2100.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:17 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic2094'] [production]
01:17 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic2101.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:17 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2101.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:17 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2091.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:16 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic2100.mgmt.codfw.wmnet with reboot policy FORCED [production]