3301-3350 of 10000 results (77ms)
2019-05-23 §
01:33 <mutante> run puppet on mx1001/mx2001 - switch mail route for phab to phab1003 [production]
01:30 <mutante> switched from phab1001 to phab1003 - applied on cp1008 varnish canary first [production]
01:28 <twentyafterfour> stopping phd on phab1001 [production]
01:18 <mutante> phabricator going readonly momentarily [production]
01:09 <twentyafterfour> extended phab downtime in icinga, actual downtime hasn't started yet, prep work taking longer than expected [production]
00:52 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@e040c6c]: Deploy GUI update (duration: 09m 54s) [production]
00:45 <mutante> phab1003 - rsyncing /srv/repos from phab1001 [production]
00:42 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@e040c6c]: Deploy GUI update [production]
00:33 <ejegg> updated payments-wiki from fa005a0640 to 786d76e212 [production]
2019-05-22 §
23:30 <twentyafterfour> scheduling downtime for phabricator from 0:00 to 1:00 utc [production]
23:10 <maxsem@deploy1001> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511889/ (duration: 00m 55s) [production]
22:18 <mdholloway> mobileapps rolled back deployment (again) due to occasional references endpoint timeouts [production]
22:17 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724, take 2 (duration: 07m 19s) [production]
22:15 <foks> reset user email and password for Nv8200pa [production]
22:09 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724, take 2 [production]
22:09 <mdholloway> mobileapps rolled back deployment due to endpoint check failure (not the same one as before); retrying momentarily [production]
22:08 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724 (duration: 03m 25s) [production]
22:08 <foks> reset user email and password for DarkKyoushu [production]
22:05 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724 [production]
21:51 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.5/includes/resourceloader/MessageBlobStore.php: T222539 / 734b3d84f7 (duration: 00m 56s) [production]
21:47 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.6/includes/resourceloader/MessageBlobStore.php: T222539 / 3cb01cc73ce9 (duration: 00m 56s) [production]
21:41 <urandom> decommissioning restbase1008-c -- T223976 [production]
20:46 <mdholloway> mobileapps rolled back deployment due to endpoint check failures [production]
20:43 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298, take 2 (duration: 04m 19s) [production]
20:39 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298, take 2 [production]
20:38 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298 (duration: 02m 41s) [production]
20:35 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298 [production]
19:26 <jforrester@deploy1001> Finished scap: Re-build i18n and re-scap everything for i18n issues for T224116 T224124 T220731 (duration: 32m 55s) [production]
18:53 <jforrester@deploy1001> Started scap: Re-build i18n and re-scap everything for i18n issues for T224116 T224124 T220731 [production]
18:35 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/FlaggedRevs: Hot-deploy reverting FlaggedRevs config for T224116 T224124 (duration: 00m 58s) [production]
18:17 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/UrlShortener/modules/ext.urlShortener.special.js: Fix i18n/command mix-up Ic99cf063a (duration: 01m 00s) [production]
17:38 <bblack> repool cp3046 as esams cache_upload ats-be node - T222937 [production]
17:06 <urandom> decommissioning restbase1008-b -- T223976 [production]
16:17 <hashar@deploy1001> rebuilt and synchronized wikiversions files: Revert group1 to 1.34.0-wmf.5 T224116 T224124 # T220731 [production]
15:11 <jbond@cumin1001> conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org [production]
15:08 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:08 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:08 <jbond@cumin1001> conftool action : set/pooled=no; selector: name=dns1002.wikimedia.org [production]
15:07 <jbond@cumin1001> conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org [production]
15:04 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:04 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:04 <jbond@cumin1001> conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org [production]
15:00 <jbond@cumin1001> conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org [production]
14:58 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:58 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:58 <jbond@cumin1001> conftool action : set/pooled=no; selector: name=dns2002.wikimedia.org [production]
14:57 <jbond@cumin1001> conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org [production]
14:54 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:54 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:54 <jbond@cumin1001> conftool action : set/pooled=no; selector: name=dns2001.wikimedia.org [production]