|
2019-05-23
§
|
| 02:35 |
<mutante> |
phabricator - going read-write again |
[production] |
| 02:24 |
<twentyafterfour> |
manually started aphlict on phab1003 |
[production] |
| 02:06 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet |
[production] |
| 02:04 |
<mutante> |
puppetmaster1001 - sudo -i conftool-merge |
[production] |
| 01:52 |
<twentyafterfour> |
phabricator is now served by phab1003 though still in read-only mode for a bit longer |
[production] |
| 01:52 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet |
[production] |
| 01:49 |
<mutante> |
puppetmaster1001 - conftool-merge |
[production] |
| 01:41 |
<eileen> |
civicrm revision changed from e6e846708f to 21afd001b6, config revision is 87e78d3eac |
[production] |
| 01:37 |
<mutante> |
depooled phab1001-vcs from git-ssh via conftool |
[production] |
| 01:36 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 01:33 |
<mutante> |
run puppet on mx1001/mx2001 - switch mail route for phab to phab1003 |
[production] |
| 01:30 |
<mutante> |
switched from phab1001 to phab1003 - applied on cp1008 varnish canary first |
[production] |
| 01:28 |
<twentyafterfour> |
stopping phd on phab1001 |
[production] |
| 01:18 |
<mutante> |
phabricator going readonly momentarily |
[production] |
| 01:09 |
<twentyafterfour> |
extended phab downtime in icinga, actual downtime hasn't started yet, prep work taking longer than expected |
[production] |
| 00:52 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@e040c6c]: Deploy GUI update (duration: 09m 54s) |
[production] |
| 00:45 |
<mutante> |
phab1003 - rsyncing /srv/repos from phab1001 |
[production] |
| 00:42 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@e040c6c]: Deploy GUI update |
[production] |
| 00:33 |
<ejegg> |
updated payments-wiki from fa005a0640 to 786d76e212 |
[production] |
|
2019-05-22
§
|
| 23:30 |
<twentyafterfour> |
scheduling downtime for phabricator from 0:00 to 1:00 utc |
[production] |
| 23:10 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511889/ (duration: 00m 55s) |
[production] |
| 22:18 |
<mdholloway> |
mobileapps rolled back deployment (again) due to occasional references endpoint timeouts |
[production] |
| 22:17 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724, take 2 (duration: 07m 19s) |
[production] |
| 22:15 |
<foks> |
reset user email and password for Nv8200pa |
[production] |
| 22:09 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724, take 2 |
[production] |
| 22:09 |
<mdholloway> |
mobileapps rolled back deployment due to endpoint check failure (not the same one as before); retrying momentarily |
[production] |
| 22:08 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724 (duration: 03m 25s) |
[production] |
| 22:08 |
<foks> |
reset user email and password for DarkKyoushu |
[production] |
| 22:05 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@39e0ef1]: Update mobileapps to fcf3724 |
[production] |
| 21:51 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.5/includes/resourceloader/MessageBlobStore.php: T222539 / 734b3d84f7 (duration: 00m 56s) |
[production] |
| 21:47 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.6/includes/resourceloader/MessageBlobStore.php: T222539 / 3cb01cc73ce9 (duration: 00m 56s) |
[production] |
| 21:41 |
<urandom> |
decommissioning restbase1008-c -- T223976 |
[production] |
| 20:46 |
<mdholloway> |
mobileapps rolled back deployment due to endpoint check failures |
[production] |
| 20:43 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298, take 2 (duration: 04m 19s) |
[production] |
| 20:39 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298, take 2 |
[production] |
| 20:38 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298 (duration: 02m 41s) |
[production] |
| 20:35 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@07632e1]: Update mobileapps to b058298 |
[production] |
| 19:26 |
<jforrester@deploy1001> |
Finished scap: Re-build i18n and re-scap everything for i18n issues for T224116 T224124 T220731 (duration: 32m 55s) |
[production] |
| 18:53 |
<jforrester@deploy1001> |
Started scap: Re-build i18n and re-scap everything for i18n issues for T224116 T224124 T220731 |
[production] |
| 18:35 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/FlaggedRevs: Hot-deploy reverting FlaggedRevs config for T224116 T224124 (duration: 00m 58s) |
[production] |
| 18:17 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/UrlShortener/modules/ext.urlShortener.special.js: Fix i18n/command mix-up Ic99cf063a (duration: 01m 00s) |
[production] |
| 17:38 |
<bblack> |
repool cp3046 as esams cache_upload ats-be node - T222937 |
[production] |
| 17:06 |
<urandom> |
decommissioning restbase1008-b -- T223976 |
[production] |
| 16:17 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: Revert group1 to 1.34.0-wmf.5 T224116 T224124 # T220731 |
[production] |
| 15:11 |
<jbond@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org |
[production] |
| 15:08 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:08 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:08 |
<jbond@cumin1001> |
conftool action : set/pooled=no; selector: name=dns1002.wikimedia.org |
[production] |
| 15:07 |
<jbond@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org |
[production] |
| 15:04 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |