251-300 of 10000 results (70ms)
2023-02-23 §
00:40 <zabe@deploy1002> aklapper and zabe: Backport for [[gerrit:886949|Fix interwiki prefix for generic wikimaniawiki (T327575)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
00:39 <zabe@deploy1002> Started scap: Backport for [[gerrit:886949|Fix interwiki prefix for generic wikimaniawiki (T327575)]] [production]
00:36 <zabe@deploy1002> Finished scap: Backport for [[gerrit:891380|throttle: Remove expired rules]] (duration: 08m 36s) [production]
00:29 <zabe@deploy1002> zabe: Backport for [[gerrit:891380|throttle: Remove expired rules]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
00:27 <zabe@deploy1002> Started scap: Backport for [[gerrit:891380|throttle: Remove expired rules]] [production]
2023-02-22 §
23:10 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:08 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
23:07 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
23:07 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:56 <zabe@deploy1002> Synchronized wmf-config/interwiki.php: T230382 (duration: 07m 06s) [production]
21:40 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
21:40 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
21:39 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
21:36 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:31 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:891293|Build backend for PersonalizedPraise (T322444)]] (duration: 07m 22s) [production]
21:24 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:891293|Build backend for PersonalizedPraise (T322444)]] [production]
21:10 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:891308|Growth: Set GEPersonalizedPraiseBackendEnabled to true on pilot wikis (T322444)]] (duration: 07m 33s) [production]
21:02 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:891308|Growth: Set GEPersonalizedPraiseBackendEnabled to true on pilot wikis (T322444)]] [production]
20:37 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase10[26-27].eqiad.wmnet: Replace expiring keys/certs - eevans@cumin1001 [production]
20:17 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase10[26-27].eqiad.wmnet: Replace expiring keys/certs - eevans@cumin1001 [production]
20:14 <eevans@cumin1001> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching restbase10[19-27].eqiad.wmnet: Replace expiring keys/certs - eevans@cumin1001 [production]
20:08 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
20:08 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
20:07 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
20:06 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P44743 and previous config saved to /var/cache/conftool/dbconfig/20230222-193422-root.json [production]
19:34 <mforns> restarted the following an-launcher1002 timers, which seemed stuck (next run = n/a): gobblin-webrequest.timer, reportupdater-browser.timer, reportupdater-reference-previews.timer, refine_event.timer, refine_eventlogging_legacy.timer [production]
19:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P44742 and previous config saved to /var/cache/conftool/dbconfig/20230222-191918-root.json [production]
19:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P44741 and previous config saved to /var/cache/conftool/dbconfig/20230222-190413-root.json [production]
18:58 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase10[19-27].eqiad.wmnet: Replace expiring keys/certs - eevans@cumin1001 [production]
18:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P44740 and previous config saved to /var/cache/conftool/dbconfig/20230222-184908-root.json [production]
18:33 <mutante> planet* - stopping and restarting all the timers for the various languages, commands from https://phabricator.wikimedia.org/P44739 [production]
18:24 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:24 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
18:23 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'depool db1112', diff saved to https://phabricator.wikimedia.org/P44738 and previous config saved to /var/cache/conftool/dbconfig/20230222-181046-ladsgroup.json [production]
18:08 <jbond> stop all failed timer servies and restart the corrosponding timer unit [production]
17:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: T329864', diff saved to https://phabricator.wikimedia.org/P44737 and previous config saved to /var/cache/conftool/dbconfig/20230222-175434-root.json [production]
17:50 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
17:41 <sukhe> force puppet run on stat1007 [production]
17:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: T329864', diff saved to https://phabricator.wikimedia.org/P44736 and previous config saved to /var/cache/conftool/dbconfig/20230222-173929-root.json [production]
17:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: T329864', diff saved to https://phabricator.wikimedia.org/P44734 and previous config saved to /var/cache/conftool/dbconfig/20230222-172424-root.json [production]
17:24 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns4004.wikimedia.org with OS bullseye [production]
17:17 <sukhe> running authdns-update for T152882 / CR 890908 [production]
17:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: T329864', diff saved to https://phabricator.wikimedia.org/P44733 and previous config saved to /var/cache/conftool/dbconfig/20230222-170920-root.json [production]
16:23 <hashar@deploy1002> Finished deploy [integration/docroot@b32e023]: doc: Add GrowthExperiments to MediaWiki components - T329034 (duration: 00m 07s) [production]
16:22 <hashar@deploy1002> Started deploy [integration/docroot@b32e023]: doc: Add GrowthExperiments to MediaWiki components - T329034 [production]
16:21 <hashar@deploy1002> Finished deploy [integration/docroot@956dd11]: zuul: Link to report_url if available (duration: 00m 15s) [production]
16:21 <hashar@deploy1002> Started deploy [integration/docroot@956dd11]: zuul: Link to report_url if available [production]
16:17 <aikochou@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]