3001-3050 of 10000 results (91ms)
2023-09-18 §
07:26 <taavi@deploy1002> Finished scap: Backport for [[gerrit:957873|robots.txt: Disable indexing user (talk) pages and draft (talk) pages on shwiki (T346589)]] (duration: 22m 24s) [production]
07:17 <taavi@deploy1002> aleksandar and taavi: Continuing with sync [production]
07:15 <moritzm> installing clamav security updates [production]
07:13 <taavi@deploy1002> aleksandar and taavi: Backport for [[gerrit:957873|robots.txt: Disable indexing user (talk) pages and draft (talk) pages on shwiki (T346589)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:03 <taavi@deploy1002> Started scap: Backport for [[gerrit:957873|robots.txt: Disable indexing user (talk) pages and draft (talk) pages on shwiki (T346589)]] [production]
2023-09-16 §
13:52 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [production]
13:52 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop: apply [production]
13:52 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop: apply [production]
13:52 <akosiaris> re-enable changeprop [production]
13:52 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop: apply [production]
12:57 <akosiaris> stop changeprop in eqiad [production]
01:44 <krinkle@deploy1002> Finished deploy [integration/docroot@9a1fb37]: (no justification provided) (duration: 00m 06s) [production]
01:44 <krinkle@deploy1002> Started deploy [integration/docroot@9a1fb37]: (no justification provided) [production]
2023-09-15 §
21:03 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1047.eqiad.wmnet with OS bullseye [production]
21:03 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1047.eqiad.wmnet with OS bullseye [production]
20:59 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1036.eqiad.wmnet with OS bullseye [production]
20:59 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1036.eqiad.wmnet with OS bullseye [production]
20:59 <tzatziki> removing 6 files for legal compliance [production]
20:58 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1038.eqiad.wmnet with OS bullseye [production]
20:58 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1038.eqiad.wmnet with OS bullseye [production]
20:58 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1036.eqiad.wmnet with OS bullseye [production]
20:58 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1036.eqiad.wmnet with OS bullseye [production]
17:56 <urandom> stopping Cassandra bootstrap, restbase1030-a — T331713 [production]
17:43 <urandom> initiate Cassandra bootstrap, restbase1030-a — T331713 [production]
17:11 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bookworm [production]
17:08 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bookworm [production]
16:51 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:47 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:46 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
16:44 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
16:37 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage [production]
16:35 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage [production]
16:32 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage [production]
16:32 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage [production]
16:12 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt2003-dev.codfw.wmnet with OS bookworm [production]
16:12 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bookworm [production]
16:09 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
15:51 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:51 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:50 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
15:50 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
15:50 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
15:50 <claime> raising mw-api-int replicas to 12+2 to cope with wdqs backfill [production]
15:43 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
15:43 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
15:42 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
15:42 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:42 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove old esams ranges and includes - cmooney@cumin1001" [production]
15:42 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
15:41 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove old esams ranges and includes - cmooney@cumin1001" [production]