151-200 of 10000 results (20ms)
2020-06-18 §
07:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
07:25 <ayounsi@cumin2001> START - Cookbook sre.dns.netbox [production]
07:24 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
07:22 <moritzm> rolling reboot of ganeti servers in codfw [production]
07:10 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
07:07 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
04:50 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11573 and previous config saved to /var/cache/conftool/dbconfig/20200618-045047-marostegui.json [production]
2020-06-17 §
23:25 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 0e7079d: Install DiscussionTools on all wikis (attempt 2) (T252264; T253943) (duration: 00m 56s) [production]
23:23 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/DiscussionTools/includes/Hooks.php: ff01083: Use $wgLocaltimezone global instead of request context (T255704) (duration: 00m 57s) [production]
23:21 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.37/extensions/DiscussionTools/includes/Hooks.php: 4551d29: Use $wgLocaltimezone global instead of request context (T252264; T253943; T255704) (duration: 00m 58s) [production]
23:01 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@79fb82f]: 0.3.39 (duration: 14m 38s) [production]
22:47 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@79fb82f]: 0.3.39 [production]
21:01 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
20:32 <hashar> Fixed up zuul-merger on contint1001 due to some faulty hotfix [production]
20:08 <hashar> Stopped zuul-merger on contint1001 [production]
19:21 <marostegui> Deploy schema change on s6 codfw master T238966 [production]
19:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1094', diff saved to https://phabricator.wikimedia.org/P11572 and previous config saved to /var/cache/conftool/dbconfig/20200617-191723-marostegui.json [production]
19:11 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:08 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:05 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
18:57 <milimetric@deploy1001> Finished deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN) (duration: 00m 10s) [production]
18:57 <milimetric@deploy1001> Started deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN) [production]
18:52 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:49 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:44 <milimetric@deploy1001> Finished deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles (duration: 27m 55s) [production]
18:41 <Urbanecm> Morning B&C window done [production]
18:31 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 96153f9: Add temporary logging for mediamoderation (T247943) (duration: 00m 56s) [production]
18:24 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: REVERT: ae76450: Install DiscussionTools on all wikis (T252264; T253943) (duration: 00m 34s) [production]
18:22 <urbanecm@deploy1001> scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
18:21 <urbanecm@deploy1001> scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
18:16 <milimetric@deploy1001> Started deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles [production]
18:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c9f6452: Set DiscussionToolsEnableVisual to true by default (T251654) (duration: 00m 56s) [production]
18:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
18:04 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
16:57 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on group0 wikis - T249261 (duration: 00m 56s) [production]
16:00 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P11571 and previous config saved to /var/cache/conftool/dbconfig/20200617-160013-marostegui.json [production]
15:28 <godog> temp bump logstash7 workers to 8 and temp stop logstash - T255243 [production]
15:17 <jforrester@deploy1001> Synchronized private/PrivateSettings.php: T247943 Add API key and recipient config for MediaModeration (duration: 00m 55s) [production]
15:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2338.codfw.wmnet [production]
15:11 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet [production]
15:11 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T247943 Install MediaModeration extension - III: Install where enabled (duration: 00m 56s) [production]
15:10 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2335.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2336.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2337.codfw.wmnet [production]
15:09 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet [production]
15:08 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw233[5-9].codfw.wmnet [production]
14:58 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.37/extensions/GrowthExperiments/modules/help/ext.growthExperiments.HelpPanelProcessDialog.js: T255607 Fix help panel sizing logic (duration: 00m 56s) [production]
14:54 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:52 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:52 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]