1-50 of 10000 results (60ms)
2023-02-16 ยง
23:00 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2013-2014,2019,2021,2024].codfw.wmnet: Restarting Cassandra to apply JVM 1.8.0_362 - eevans@cumin1001 [production]
22:15 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
22:15 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
22:15 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
22:13 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
22:13 <rzl@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
22:13 <rzl@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
22:13 <rzl@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
22:13 <rzl@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
22:12 <rzl@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
22:12 <rzl@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
22:12 <rzl@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
22:12 <rzl@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
22:12 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
22:11 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
22:11 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
22:10 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
22:10 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
22:09 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
22:09 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
22:08 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
22:08 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
22:07 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2013-2014,2019,2021,2024].codfw.wmnet: Restarting Cassandra to apply JVM 1.8.0_362 - eevans@cumin1001 [production]
22:07 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
22:07 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
22:06 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
21:24 <TheresNoTime> close UTC late backport window [production]
21:21 <eileen> civicrm upgraded from efa4c485 to ffc16d2d [production]
21:11 <samtar@deploy1002> Finished scap: Backport for [[gerrit:865740|Remove Research Incentive survey from swwiki (T321252)]] (duration: 08m 13s) [production]
21:09 <SandraEbele> Added new field referer_data to wmf.webrequest table using the alter table statement [production]
21:04 <samtar@deploy1002> samtar and dani: Backport for [[gerrit:865740|Remove Research Incentive survey from swwiki (T321252)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:03 <samtar@deploy1002> Started scap: Backport for [[gerrit:865740|Remove Research Incentive survey from swwiki (T321252)]] [production]
19:29 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.23 refs T325586 [production]
19:03 <dancy@deploy1002> Installation of scap version "4.36.0" completed for 564 hosts [production]
19:03 <dancy@deploy1002> Installing scap version "4.36.0" for 564 hosts [production]
18:56 <ebysans@deploy1002> Finished deploy [analytics/refinery@0f1a930] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0f1a930] (duration: 01m 23s) [production]
18:54 <ebysans@deploy1002> Started deploy [analytics/refinery@0f1a930] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0f1a930] [production]
18:54 <ebysans@deploy1002> Finished deploy [analytics/refinery@0f1a930] (thin): Regular analytics weekly train THIN [analytics/refinery@0f1a930] (duration: 00m 07s) [production]
18:54 <ebysans@deploy1002> Started deploy [analytics/refinery@0f1a930] (thin): Regular analytics weekly train THIN [analytics/refinery@0f1a930] [production]
18:52 <ebysans@deploy1002> Finished deploy [analytics/refinery@0f1a930]: Regular analytics weekly train [analytics/refinery@0f1a930] (duration: 07m 11s) [production]
18:45 <ebysans@deploy1002> Started deploy [analytics/refinery@0f1a930]: Regular analytics weekly train [analytics/refinery@0f1a930] [production]
18:37 <SandraEbele> killed webrequest oozie bundle to deploy refinery changes. [production]
18:28 <bd808@deploy1002> helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [production]
18:26 <bd808@deploy1002> helmfile [eqiad] START helmfile.d/services/toolhub: apply [production]
18:25 <bd808@deploy1002> helmfile [codfw] DONE helmfile.d/services/toolhub: apply [production]
18:24 <bd808@deploy1002> helmfile [codfw] START helmfile.d/services/toolhub: apply [production]
18:22 <bd808@deploy1002> helmfile [staging] DONE helmfile.d/services/toolhub: apply [production]
18:21 <bd808@deploy1002> helmfile [staging] START helmfile.d/services/toolhub: apply [production]
18:08 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 [production]
18:08 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 [production]