651-700 of 10000 results (57ms)
2019-11-22 §
08:54 <gehel> restarting blazegraph and updater on edqs1007 [production]
08:49 <ema> depool cp1081 and reimage as text_ats T227432 [production]
06:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Rebalance weights on s7 in preparation for s7 failover on Tuesday T238044', diff saved to https://phabricator.wikimedia.org/P9722 and previous config saved to /var/cache/conftool/dbconfig/20191122-063145-marostegui.json [production]
03:49 <shdubsh> restart prometheus@ops on prometheus1003 T238807 [production]
00:46 <mutante> xhgui1001/xhgui2001 - rsyncing /srv/mongod from tungsten to /srv/tungsten/mongod/ on both new machines (T158837) [production]
00:37 <mutante> tungsten - starting ferm service [production]
00:20 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Move newcomer tasks JSON config from mw.org to local wikis (T237301) (duration: 00m 52s) [production]
00:18 <catrope@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/GrowthExperiments/: Make non-remote titles work in RemotePageConfigurationLoader (T237301) (duration: 00m 54s) [production]
2019-11-21 §
23:09 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Remove unused CirrusSearch config variable (duration: 00m 52s) [production]
22:11 <Urbanecm> mwscript importImages.php --wiki=commonswiki --overwrite --user=Bürgerentscheid . (T238764) [production]
21:42 <mholloway-shell@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/UploadWizard: Revert "Add Machine Vision CTA to final step (T234960)", take 2 (duration: 00m 41s) [production]
21:36 <mholloway-shell@deploy1001> Scap failed!: 5/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
21:34 <mholloway-shell@deploy1001> Scap failed!: 4/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
21:29 <mholloway-shell@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/UploadWizard: Add Machine Vision CTA to final step (T234960) (duration: 00m 59s) [production]
21:16 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@70154b4]: Update mobileapps to c140e88 (duration: 06m 29s) [production]
21:09 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@70154b4]: Update mobileapps to c140e88 [production]
20:51 <mutante> puppetmaster1001 - revoking puppet certs for xhgui1001/xhgui2001 [production]
20:49 <mutante> ganeti1003 - switching boot order of xhgui1001 to network and reinstalling with stretch (T238098) [production]
20:16 <mforns@deploy1001> Finished deploy [analytics/refinery@97015e4]: add new projects to webrequest whitelist (duration: 08m 29s) [production]
20:14 <mutante> icinga1001 - systemctl reset-failed [production]
20:08 <mforns@deploy1001> Started deploy [analytics/refinery@97015e4]: add new projects to webrequest whitelist [production]
19:01 <andrewbogott> upgrading designate to 'ocata' on cloudservices1003 and 1004 [production]
18:49 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:45 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:42 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:13 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch private wikis back to Parsoid/JS - T229015 (duration: 00m 52s) [production]
18:03 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:02 <mobrovac@deploy1001> Synchronized wmf-config/ProductionServices.php: Use HTTPS for contacting Parsoid/PHP - T229015 (duration: 00m 53s) [production]
17:52 <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Switch private wikis to Parsoid/PHP; file 4/4 -- T229015 (duration: 00m 53s) [production]
17:51 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch private wikis to Parsoid/PHP; file 3/4 -- T229015 (duration: 00m 51s) [production]
17:50 <mobrovac@deploy1001> Synchronized wmf-config/ProductionServices.php: Switch private wikis to Parsoid/PHP; file 2/4 -- T229015 (duration: 00m 53s) [production]
17:48 <mobrovac@deploy1001> Synchronized wmf-config/LabsServices.php: Switch private wikis to Parsoid/PHP; file 1/4 -- T229015 (duration: 00m 53s) [production]
17:27 <mobrovac@deploy1001> Finished deploy [restbase/deploy@b987068]: Switch mw.org to Parsoid/PHP - T229015 (duration: 16m 43s) [production]
17:10 <mobrovac@deploy1001> Started deploy [restbase/deploy@b987068]: Switch mw.org to Parsoid/PHP - T229015 [production]
17:09 <mobrovac@deploy1001> Finished deploy [restbase/deploy@b987068] (dev-cluster): Switch mw.org to Parsoid/PHP (duration: 02m 38s) [production]
17:06 <mobrovac@deploy1001> Started deploy [restbase/deploy@b987068] (dev-cluster): Switch mw.org to Parsoid/PHP [production]
16:54 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
16:48 <sbassett@deploy1001> Finished scap: Deploying T238451 (ext:AbuseFilter), running scap sync for i18n issues. (duration: 16m 42s) [production]
16:31 <sbassett@deploy1001> Started scap: Deploying T238451 (ext:AbuseFilter), running scap sync for i18n issues. [production]
15:54 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
15:42 <mforns@deploy1001> Finished deploy [analytics/refinery@7f32472]: deploying analytics refinery (after refinery-source v0.0.107) (duration: 10m 50s) [production]
15:31 <mforns@deploy1001> Started deploy [analytics/refinery@7f32472]: deploying analytics refinery (after refinery-source v0.0.107) [production]
15:30 <ema> pool cp1079 with ATS backend T227432 [production]
15:22 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:19 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:13 <akosiaris> purge https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz, https://releases.wikimedia.org/charts/ and https://releases.wikimedia.org/charts/index.yaml [production]
15:09 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:07 <bblack> DONE testing deployment software changes on authdns cluster, back to normal [production]
15:07 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:49 <ema> depool cp1079 and reimage as text_ats T227432 [production]