251-300 of 10000 results (72ms)
2023-05-01 §
16:19 <urandom> upgrading sessionstore2002 to Cassandra 3.11.14 — T335383 [production]
16:03 <urandom> upgrading sessionstore2001 to Cassandra 3.11.14 — T335383 [production]
15:59 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool sessionstore in codfw: maintenance [production]
15:54 <eevans@cumin1001> START - Cookbook sre.discovery.service-route depool sessionstore in codfw: maintenance [production]
14:58 <sukhe> restart haproxy on cp1077: T334448 [production]
14:09 <sukhe> move backup routes for ns0 from dns2001 to dns2002: T334049 [production]
14:04 <sukhe> move ns1 from dns2001 to dns2002: T334049 [production]
13:19 <taavi@deploy1002> Finished scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] (duration: 07m 59s) [production]
13:12 <taavi@deploy1002> superpes and taavi: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:11 <taavi@deploy1002> Started scap: Backport for [[gerrit:913687|[slwiki] Enable VisualEditor on Draft and Project namespaces (T335208)]], [[gerrit:913689|[frwikibooks] Change the logo for Vector legacy and add a wordmark for Vector 2022 (T335642)]], [[gerrit:913913|Close nawiki (T335674)]] [production]
11:41 <zabe@deploy1002> Finished scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 21m 08s) [production]
11:31 <zabe@deploy1002> zabe: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
11:20 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
11:20 <zabe@deploy1002> sync-world aborted: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] (duration: 02m 33s) [production]
11:17 <zabe@deploy1002> Started scap: Backport for [[gerrit:913915|Start writing to af_actor/afh_actor in group0 wikis (T334295)]] [production]
2023-04-30 §
14:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
14:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2184.codfw.wmnet with reason: Host down T335640 [production]
08:06 <elukey> powercycle ores1002 (mgmt console tty not usable, host frozen) [production]
2023-04-29 §
23:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
23:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint [production]
22:54 <rzl@cumin2002> dbctl commit (dc=all): 'Depool db1132', diff saved to https://phabricator.wikimedia.org/P47290 and previous config saved to /var/cache/conftool/dbconfig/20230429-225457-rzl.json [production]
2023-04-28 §
22:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for new frack nodes - pt1979@cumin2002" [production]
22:33 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for new frack nodes - pt1979@cumin2002" [production]
22:31 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
21:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
21:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
20:25 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit2002.wikimedia.org with reason: setup [production]
20:25 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit2002.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1003.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
20:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gerrit1001.wikimedia.org with reason: setup [production]
19:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:16 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:16 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:10 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:10 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:07 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:07 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
17:50 <htriedman@deploy1002> Finished deploy [airflow-dags/platform_eng@d56b7fb]: (no justification provided) (duration: 00m 10s) [production]
17:50 <htriedman@deploy1002> Started deploy [airflow-dags/platform_eng@d56b7fb]: (no justification provided) [production]
15:39 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
15:23 <jynus> update schema for backup1-codfw (mediabackups) T327157 [production]
15:07 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 2519 [production]
15:01 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 2519 [production]
14:57 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['stat1004'] [production]
14:57 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['stat1004'] [production]
14:51 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['stat1004'] [production]