2001-2050 of 10000 results (75ms)
2023-01-11 ยง
17:18 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
17:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2112 (T321391)', diff saved to https://phabricator.wikimedia.org/P42989 and previous config saved to /var/cache/conftool/dbconfig/20230111-171338-marostegui.json [production]
17:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2112 (T321391)', diff saved to https://phabricator.wikimedia.org/P42988 and previous config saved to /var/cache/conftool/dbconfig/20230111-171114-marostegui.json [production]
17:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 1%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42987 and previous config saved to /var/cache/conftool/dbconfig/20230111-171021-root.json [production]
17:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
17:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
17:04 <marostegui> dbmaint deploy schema change with replication on s7 eqiad T321391 [production]
17:03 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
17:03 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
16:38 <marostegui> dbmaint deploy schema change with replication on s5 eqiad T321391 [production]
16:31 <marostegui> dbmaint deploy schema change with replication on s4 eqiad T321391 [production]
16:25 <marostegui> dbmaint deploy schema change with replication on s8 eqiad T321391 [production]
16:22 <marostegui> dbmaint deploy schema change with replication on s6 eqiad T321391 [production]
16:06 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:06 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force update after eqsin outage is over - volans@cumin1001" [production]
16:05 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force update after eqsin outage is over - volans@cumin1001" [production]
16:03 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:01 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host mc1038.eqiad.wmnet with OS bullseye [production]
16:00 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:58 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:53 <zabe@deploy1002> Finished scap: T233004 (duration: 07m 54s) [production]
15:45 <zabe@deploy1002> Started scap: T233004 [production]
15:38 <zabe@deploy1002> backport aborted: (duration: 04m 25s) [production]
15:38 <zabe@deploy1002> sync-world aborted: Backport for [[gerrit:878870|Start reading from cul_actor everywhere (T233004)]] (duration: 04m 00s) [production]
15:36 <zabe@deploy1002> zabe and zabe: Backport for [[gerrit:878870|Start reading from cul_actor everywhere (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
15:34 <zabe@deploy1002> Started scap: Backport for [[gerrit:878870|Start reading from cul_actor everywhere (T233004)]] [production]
15:31 <pt1979@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
15:21 <marostegui> Stop mariadb on db1106 to reclone db1206 (there will be lag on s1 on wikireplicas) T326669 [production]
15:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106', diff saved to https://phabricator.wikimedia.org/P42982 and previous config saved to /var/cache/conftool/dbconfig/20230111-151712-marostegui.json [production]
14:56 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
14:47 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:46 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1005.eqiad.wmnet with OS bullseye [production]
14:46 <btullis@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" [production]
14:46 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.40.0-wmf.18/extensions/Wikibase/repo/tests/jest/wikibase.vector.searchClient.spec.js: Backport: [[gerrit:877972|Add missing parentheses to vector search match text (T326633)]] (2/2) (duration: 06m 46s) [production]
14:42 <btullis@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" [production]
14:39 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.40.0-wmf.18/extensions/Wikibase/repo/resources/wikibase.vector.searchClient.js: Backport: [[gerrit:877972|Add missing parentheses to vector search match text (T326633)]] (1/2) (duration: 07m 09s) [production]
14:28 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:877983|Fix test constructing HTMLFormField without parent (T326621)]] (duration: 08m 38s) [production]
14:25 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage [production]
14:22 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage [production]
14:21 <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and lucaswerkmeister-wmde: Backport for [[gerrit:877983|Fix test constructing HTMLFormField without parent (T326621)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
14:19 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:877983|Fix test constructing HTMLFormField without parent (T326621)]] [production]
14:14 <jelto@cumin1001> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) [production]
14:10 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd1001.eqiad.wmnet [production]
14:10 <moritzm> installing postgresql 11 security updates on maps/eqiad [production]
14:06 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host cephosd1005.eqiad.wmnet with OS bullseye [production]
14:02 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1004.eqiad.wmnet with OS bullseye [production]