4501-4550 of 10000 results (82ms)
2022-12-13 ยง
20:10 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge elasticsearch and plugin upgrade - ryankemper@cumin1001 - T322776 [production]
19:18 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts contint1001.wikimedia.org [production]
19:18 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:18 <dzahn@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: contint1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - dzahn@cumin2002" [production]
19:00 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:867609|ParserCache: fix metrics keys]], [[gerrit:867611|Don't write to parser cache from maintenance script]], [[gerrit:867613|Fix brittle test]] (duration: 07m 53s) [production]
18:59 <dzahn@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: contint1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - dzahn@cumin2002" [production]
18:57 <dzahn@cumin2002> START - Cookbook sre.dns.netbox [production]
18:54 <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:867609|ParserCache: fix metrics keys]], [[gerrit:867611|Don't write to parser cache from maintenance script]], [[gerrit:867613|Fix brittle test]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
18:52 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:867609|ParserCache: fix metrics keys]], [[gerrit:867611|Don't write to parser cache from maintenance script]], [[gerrit:867613|Fix brittle test]] [production]
18:51 <dzahn@cumin2002> START - Cookbook sre.hosts.decommission for hosts contint1001.wikimedia.org [production]
18:51 <mutante> decom'ing contint1001 (formerly prod CI) server, replaced by contint1002 T324698 [production]
18:47 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:867610|Don't write to parser cache from maintenance script]], [[gerrit:867608|ParserCache: fix metrics keys]] (duration: 09m 25s) [production]
18:40 <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:867610|Don't write to parser cache from maintenance script]], [[gerrit:867608|ParserCache: fix metrics keys]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
18:38 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:867610|Don't write to parser cache from maintenance script]], [[gerrit:867608|ParserCache: fix metrics keys]] [production]
18:21 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-stretch1001.eqiad.wmnet with OS bullseye [production]
17:27 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-stretch1001.eqiad.wmnet with OS bullseye [production]
17:26 <btullis> edited automation/proxies/ttyS1-115200.conf to remove `include "/etc/dhcp/automation/ttyS1-115200/kafka-stretch1001.conf";`and restarted isc-dhc-server [production]
17:22 <btullis> btullis@install1003:/etc/dhcp/automation/ttyS1-115200$ sudo systemctl restart isc-dhcp-server.service T314156 [production]
16:35 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-stretch1001.eqiad.wmnet with OS bullseye [production]
16:35 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-stretch1001.eqiad.wmnet with OS bullseye [production]
16:17 <hnowlan@puppetmaster1001> conftool action : set/weight=10:pooled=no; selector: service=thumbor,name=kubernetes1010.eqiad.wmnet [production]
16:15 <hnowlan@puppetmaster1001> conftool action : set/weight=10:pooled=yes; selector: service=thumbor,name=kubernetes1010.eqiad.wmnet [production]
16:12 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-stretch2002.codfw.wmnet with OS bullseye [production]
16:05 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: service=thumbor,name=kubernetes101[01234].eqiad.wmnet [production]
16:03 <moritzm> installing ruby-tzinfo security updates [production]
16:01 <hnowlan@puppetmaster1001> conftool action : set/weight=4:pooled=yes; selector: service=thumbor,name=kubernetes101[1234].eqiad.wmnet [production]
15:58 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20804 [production]
15:54 <hnowlan@puppetmaster1001> conftool action : set/weight=2:pooled=yes; selector: service=thumbor,name=kubernetes101[234].eqiad.wmnet [production]
15:51 <hnowlan@puppetmaster1001> conftool action : set/weight=2:pooled=yes; selector: service=thumbor,name=kubernetes1010.eqiad.wmnet [production]
15:35 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 20804 [production]
15:27 <derick@deploy1002> Finished scap: Backport for [[gerrit:867237|RangeChronologicalPager: Restore the compatibility with derived classes (T228431 T325034)]] (duration: 08m 59s) [production]
15:20 <derick@deploy1002> derick and func: Backport for [[gerrit:867237|RangeChronologicalPager: Restore the compatibility with derived classes (T228431 T325034)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
15:19 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/api-gateway: sync [production]
15:19 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/api-gateway: sync [production]
15:18 <derick@deploy1002> Started scap: Backport for [[gerrit:867237|RangeChronologicalPager: Restore the compatibility with derived classes (T228431 T325034)]] [production]
15:02 <derick@deploy1002> Finished scap: Backport for [[gerrit:867274|Log linter data while parsing full pages (T246403)]] (duration: 10m 28s) [production]
14:53 <derick@deploy1002> derick and arlolra: Backport for [[gerrit:867274|Log linter data while parsing full pages (T246403)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
14:52 <derick@deploy1002> Started scap: Backport for [[gerrit:867274|Log linter data while parsing full pages (T246403)]] [production]
14:50 <derick@deploy1002> backport aborted: (duration: 07m 09s) [production]
14:41 <derick@deploy1002> Finished scap: Backport for [[gerrit:866627|hewiki: set VisualEditor to direct mode (T320529)]] (duration: 14m 34s) [production]
14:32 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-stretch2002.codfw.wmnet with OS bullseye [production]
14:31 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: hardware diagnostics [production]
14:31 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: hardware diagnostics [production]
14:28 <derick@deploy1002> derick and daniel: Backport for [[gerrit:866627|hewiki: set VisualEditor to direct mode (T320529)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
14:26 <derick@deploy1002> Started scap: Backport for [[gerrit:866627|hewiki: set VisualEditor to direct mode (T320529)]] [production]
14:22 <moritzm> added smunene to pwstore [production]
14:17 <derick@deploy1002> Backport cancelled. [production]
14:04 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-stretch2001.codfw.wmnet with reason: Accessing BIOS on kafka-stretch2001 [production]
14:03 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on kafka-stretch2001.codfw.wmnet with reason: Accessing BIOS on kafka-stretch2001 [production]
13:59 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply [production]