4451-4500 of 10000 results (92ms)
2023-05-01 §
18:58 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:58 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
18:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
17:32 <sukhe> run authdns-update for CR 913966 [production]
16:54 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: maintenance [production]
16:49 <eevans@cumin1001> START - Cookbook sre.discovery.service-route pool sessionstore in codfw: maintenance [production]
16:44 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] (duration: 07m 22s) [production]
16:38 <urbanecm@deploy1002> urbanecm: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
16:37 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:912233|dewiki: Deploy Growth features to 100% of newcomers (T335385)]] [production]
16:33 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2001.codfw.wmnet [production]
16:26 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2001.codfw.wmnet [production]
16:22 <urandom> upgrading sessionstore2003 to Cassandra 3.11.14 — T335383 [production]
16:19 <urandom> upgrading sessionstore2002 to Cassandra 3.11.14 — T335383 [production]
16:03 <urandom> upgrading sessionstore2001 to Cassandra 3.11.14 — T335383 [production]
15:59 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool sessionstore in codfw: maintenance [production]
15:54 <eevans@cumin1001> START - Cookbook sre.discovery.service-route depool sessionstore in codfw: maintenance [production]
14:58 <sukhe> restart haproxy on cp1077: T334448 [production]
14:09 <sukhe> move backup routes for ns0 from dns2001 to dns2002: T334049 [production]
14:04 <sukhe> move ns1 from dns2001 to dns2002: T334049 [production]
13:19 <taavi@deploy1002> Finished scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] (duration: 07m 59s) [production]
13:12 <taavi@deploy1002> superpes and taavi: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:11 <taavi@deploy1002> Started scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] [production]
11:41 <zabe@deploy1002> Finished scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 21m 08s) [production]
11:31 <zabe@deploy1002> zabe: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
11:20 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
11:20 <zabe@deploy1002> sync-world aborted: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 02m 33s) [production]
11:17 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
2023-04-30 §
14:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
14:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
08:06 <elukey> powercycle ores1002 (mgmt console tty not usable, host frozen) [production]
2023-04-29 §
23:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
23:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
22:54 <rzl@cumin2002> dbctl commit (dc=all): 'Depool db1132', diff saved to https://phabricator.wikimedia.org/P47290 and previous config saved to /var/cache/conftool/dbconfig/20230429-225457-rzl.json [production]
2023-04-28 §
22:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for new frack nodes - pt1979@cumin2002" [production]
22:33 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for new frack nodes - pt1979@cumin2002" [production]
22:31 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
21:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
21:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
20:25 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit2002.wikimedia.org with reason: setup [production]
20:25 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit2002.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1003.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
19:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:16 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:16 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]