101-150 of 10000 results (21ms)
2020-06-17 ยง
19:21 <marostegui> Deploy schema change on s6 codfw master T238966 [production]
19:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1094', diff saved to https://phabricator.wikimedia.org/P11572 and previous config saved to /var/cache/conftool/dbconfig/20200617-191723-marostegui.json [production]
19:11 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:08 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:05 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
18:57 <milimetric@deploy1001> Finished deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN) (duration: 00m 10s) [production]
18:57 <milimetric@deploy1001> Started deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN) [production]
18:52 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:49 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:44 <milimetric@deploy1001> Finished deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles (duration: 27m 55s) [production]
18:41 <Urbanecm> Morning B&C window done [production]
18:31 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 96153f9: Add temporary logging for mediamoderation (T247943) (duration: 00m 56s) [production]
18:24 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: REVERT: ae76450: Install DiscussionTools on all wikis (T252264; T253943) (duration: 00m 34s) [production]
18:22 <urbanecm@deploy1001> scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
18:21 <urbanecm@deploy1001> scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
18:16 <milimetric@deploy1001> Started deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles [production]
18:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c9f6452: Set DiscussionToolsEnableVisual to true by default (T251654) (duration: 00m 56s) [production]
18:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
18:04 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
16:57 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on group0 wikis - T249261 (duration: 00m 56s) [production]
16:00 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P11571 and previous config saved to /var/cache/conftool/dbconfig/20200617-160013-marostegui.json [production]
15:28 <godog> temp bump logstash7 workers to 8 and temp stop logstash - T255243 [production]
15:17 <jforrester@deploy1001> Synchronized private/PrivateSettings.php: T247943 Add API key and recipient config for MediaModeration (duration: 00m 55s) [production]
15:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2338.codfw.wmnet [production]
15:11 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet [production]
15:11 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T247943 Install MediaModeration extension - III: Install where enabled (duration: 00m 56s) [production]
15:10 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2335.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2336.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2337.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet [production]
15:08 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw233[5-9].codfw.wmnet [production]
14:58 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.37/extensions/GrowthExperiments/modules/help/ext.growthExperiments.HelpPanelProcessDialog.js: T255607 Fix help panel sizing logic (duration: 00m 56s) [production]
14:54 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:52 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:52 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:50 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:49 <mdholloway> rolled back recommendation-api deployment due to canary endpoint check failure (T255683) [production]
14:44 <mholloway-shell@deploy1001> Finished deploy [recommendation-api/deploy@c39d567]: Update recommendation-api to db97742 (duration: 01m 16s) [production]
14:43 <mholloway-shell@deploy1001> Started deploy [recommendation-api/deploy@c39d567]: Update recommendation-api to db97742 [production]
14:30 <akosiaris> redrain kubernetes1007-14 [production]
14:27 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:27 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:27 <mutante> disabling puppet on icinga to avoid alert spam when adding new appservers [production]
14:25 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
14:22 <akosiaris> uncordon kubernetes10{07..14} again [production]
14:13 <mutante> generating new mcrouter certs for mw2335 - mw2339 (T247021) [production]
14:02 <mutante> rebooting mw2335 through mw2339 (not in service) [production]
13:51 <XioNoX> cleanup msw1-codfw interfaces [production]
13:44 <akosiaris> redrain kubernetes1007-14 [production]
13:37 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]