4451-4500 of 10000 results (92ms)
2023-05-01 §
21:08 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic[2045-2048,2059,2065-2066,2071,2081-2083] for row C switch upgrade - bking@cumin1001 - T334049 [production]
21:08 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic[2045-2048,2059,2065-2066,2071,2081-2083] for row C switch upgrade - bking@cumin1001 - T334049 [production]
21:02 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: maintenance [production]
20:57 <eevans@cumin1001> START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: maintenance [production]
20:47 <urandom> upgrading sessionstore1003 to Cassandra 3.11.14 — T335383 [production]
20:45 <urandom> upgrading sessionstore1002 to Cassandra 3.11.14 — T335383 [production]
20:42 <urandom> upgrading sessionstore1001 to Cassandra 3.11.14 — T335383 [production]
20:38 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool sessionstore in eqiad: maintenance [production]
20:33 <eevans@cumin1001> START - Cookbook sre.discovery.service-route depool sessionstore in eqiad: maintenance [production]
20:13 <taavi@deploy1002> Finished scap: Backport for [[gerrit:912895|Point SyntaxHighlight at /srv/app/pygmentize (T320848)]] (duration: 08m 12s) [production]
20:06 <taavi@deploy1002> legoktm and taavi: Backport for [[gerrit:912895|Point SyntaxHighlight at /srv/app/pygmentize (T320848)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
20:05 <taavi@deploy1002> Started scap: Backport for [[gerrit:912895|Point SyntaxHighlight at /srv/app/pygmentize (T320848)]] [production]
19:42 <dancy@deploy1002> Installation of scap version "4.52.0" completed for 593 hosts [production]
19:41 <dancy@deploy1002> Installing scap version "4.52.0" for 593 hosts [production]
19:06 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:06 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:58 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:58 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
17:32 <sukhe> run authdns-update for CR 913966 [production]
16:54 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: maintenance [production]
16:49 <eevans@cumin1001> START - Cookbook sre.discovery.service-route pool sessionstore in codfw: maintenance [production]
16:44 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] (duration: 07m 22s) [production]
16:38 <urbanecm@deploy1002> urbanecm: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
16:37 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] [production]
16:33 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2001.codfw.wmnet [production]
16:26 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2001.codfw.wmnet [production]
16:22 <urandom> upgrading sessionstore2003 to Cassandra 3.11.14 — T335383 [production]
16:19 <urandom> upgrading sessionstore2002 to Cassandra 3.11.14 — T335383 [production]
16:03 <urandom> upgrading sessionstore2001 to Cassandra 3.11.14 — T335383 [production]
15:59 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool sessionstore in codfw: maintenance [production]
15:54 <eevans@cumin1001> START - Cookbook sre.discovery.service-route depool sessionstore in codfw: maintenance [production]
14:58 <sukhe> restart haproxy on cp1077: T334448 [production]
14:09 <sukhe> move backup routes for ns0 from dns2001 to dns2002: T334049 [production]
14:04 <sukhe> move ns1 from dns2001 to dns2002: T334049 [production]
13:19 <taavi@deploy1002> Finished scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] (duration: 07m 59s) [production]
13:12 <taavi@deploy1002> superpes and taavi: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:11 <taavi@deploy1002> Started scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] [production]
11:41 <zabe@deploy1002> Finished scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 21m 08s) [production]
11:31 <zabe@deploy1002> zabe: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
11:20 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
11:20 <zabe@deploy1002> sync-world aborted: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 02m 33s) [production]
11:17 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
2023-04-30 §
14:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
14:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
08:06 <elukey> powercycle ores1002 (mgmt console tty not usable, host frozen) [production]
2023-04-29 §
23:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
23:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
22:54 <rzl@cumin2002> dbctl commit (dc=all): 'Depool db1132', diff saved to https://phabricator.wikimedia.org/P47290 and previous config saved to /var/cache/conftool/dbconfig/20230429-225457-rzl.json [production]