301-350 of 10000 results (41ms)
2019-12-05 §
01:05 <mutante> phab1001 - powercycling [production]
01:04 <mutante> telling phab1001 to boot into BIOS next time it boots via mgmt console (https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_PowerEdge_RN30#Reboot_and_boot_into_BIOS_then_console) [production]
01:03 <twentyafterfour> phabricator switched back to phab1003 - reimaging phab1001 now [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=phab1001-vcs.eqiad.wmnet [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet [production]
00:43 <cwhite@cumin1001> dbctl commit (dc=all): 'Depool db1062 T239874', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20191205-004256-cwhite.json [production]
2019-12-04 §
23:38 <brennen@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.35.0-wmf.5" [production]
23:35 <brennen@deploy1001> Scap failed!: 9/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
23:30 <brennen@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.8 (duration: 01m 01s) [production]
23:29 <brennen@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.8 [production]
23:22 <jforrester@deploy1001> Synchronized wmf-config/logging.php: Keep test consistent w/ operations/puppet, for logging (duration: 01m 02s) [production]
23:21 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Keep test consistent w/ operations/puppet, for CS (duration: 01m 03s) [production]
22:49 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Disable VisualEditor on Wikitech (and Labs Wikitech) (duration: 01m 02s) [production]
22:45 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@e6afe36]: Update mobileapps to 9e9b042 (duration: 05m 48s) [production]
22:39 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@e6afe36]: Update mobileapps to 9e9b042 [production]
22:39 <bstorm_> powered off cloudstore1008, disabled sync from cloudstore1009, and downtimed both cloudstore1008 and cloudstore1009 for memory module replacement T239569 [production]
22:37 <bstorm_> poweroff cloudstore1008 for memory module replacement [production]
22:24 <RoanKattouw> T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php kowiki --cutoff 350 [production]
22:21 <RoanKattouw> T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php cswiki --cutoff 350 [production]
21:48 <eileen> civicrm revision changed from 6812488f3a to 4a72ad4e63, config revision is 9f4db1edad (CiviCRM security patches ) [production]
21:38 <rzl@cumin1001> conftool action : set/pooled=yes; selector: name=mw226[3456]\.codfw\.wmnet [production]
21:11 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:09 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:47 <milimetric@deploy1001> Finished deploy [analytics/refinery@fc710ec] (thin): Weekly train deploy to labs/notebooks (duration: 00m 07s) [production]
20:47 <milimetric@deploy1001> Started deploy [analytics/refinery@fc710ec] (thin): Weekly train deploy to labs/notebooks [production]
20:33 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:31 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:28 <milimetric@deploy1001> Finished deploy [analytics/refinery@fc710ec]: Weekly train deploy (duration: 07m 09s) [production]
20:28 <brennen@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.35.0-wmf.5" [production]
20:21 <milimetric@deploy1001> Started deploy [analytics/refinery@fc710ec]: Weekly train deploy [production]
20:15 <milimetric@deploy1001> Finished deploy [analytics/refinery@fc710ec]: Weekly train deploy (duration: 06m 37s) [production]
20:14 <brennen@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.8 (duration: 01m 29s) [production]
20:12 <brennen@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.8 [production]
20:09 <milimetric@deploy1001> Started deploy [analytics/refinery@fc710ec]: Weekly train deploy [production]
19:56 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:54 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:43 <twentyafterfour@deploy1001> Finished deploy [phabricator/deployment@e4e2b22]: deploy phabricator to phab2001.codfw.wmnet (duration: 00m 31s) [production]
19:43 <twentyafterfour@deploy1001> Started deploy [phabricator/deployment@e4e2b22]: deploy phabricator to phab2001.codfw.wmnet [production]
19:38 <milimetric@deploy1001> deploy aborted: Weekly train deploy (duration: 00m 21s) [production]
19:38 <milimetric@deploy1001> Started deploy [analytics/refinery@c8de2ab]: Weekly train deploy [production]
19:21 <Amir1> morning SWAT is done [production]
19:19 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:17 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.8/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php: SWAT: [[gerrit:554330|Remove no-op 'jquery.ui.core.styles' from FullEntityParserOutputGenerator]] (T219604 T239594) (duration: 01m 06s) [production]
19:16 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:55 <bblack> dns1001: back to normal again [production]
18:54 <bblack> dns1001: stop bird.service again, briefly [production]
18:52 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:50 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:49 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
18:46 <bblack> dns1001: restart bird.service [production]