|
2016-04-04
§
|
| 20:12 |
<subbu> |
finished deploying parsoid sha 579ec3e6 |
[production] |
| 20:07 |
<subbu> |
synced code; restarted parsoid on wtp1002 as a canary |
[production] |
| 20:04 |
<subbu> |
starting parsoid deploy |
[production] |
| 19:48 |
<ejegg> |
rolled payments-wiki back to a9659965d8b55b11518680a1170242f311c7f1d2 |
[production] |
| 19:40 |
<apergos> |
disabled puppet on snapshot1001,2,4 while new hosts come on line, til probably Apr 5-6 |
[production] |
| 19:38 |
<akosiaris> |
depool maps-test2004 |
[production] |
| 18:59 |
<ejegg> |
updated payments-wiki from a9659965d8b55b11518680a1170242f311c7f1d2 to eac5f979099056ef142500570555771dfb1992c1 |
[production] |
| 18:36 |
<tgr> |
ran MassMessages/sendMessages.php for T128056 |
[production] |
| 18:18 |
<cmjohnson1> |
stat1002 swapping failed disk slot 11 |
[production] |
| 18:04 |
<cmjohnson1> |
db1052 swapping failed disk slot 8 |
[production] |
| 17:37 |
<cmjohnson1> |
shutting down iridium to reapply thermal paste |
[production] |
| 17:30 |
<greg-g> |
Phabricator going down in about 10 minutes to hopefully address the overheating issue: T131742 |
[production] |
| 16:36 |
<urandom> |
Restarting bootstrap of restbase2004.codfw.wmnet : T95253 |
[production] |
| 15:53 |
<greg-g> |
15:45 < elukey> !log aqs1001 re-added to the aqs pool (nodejs NOT upgraded due to issues with Cassandra) |
[production] |
| 15:42 |
<Krenair> |
ran wikitech-static updates |
[production] |
| 15:18 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.19/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android production app: 50% [[gerrit:280957]] (duration: 00m 46s) |
[production] |
| 14:55 |
<urandom> |
Restarting restbase2004-a.codfw.wmnet (cancelling bootstrap of 2004-b) |
[production] |
| 14:46 |
<elukey> |
de-pooled aqs1001.eqiad from the confd pool for nodejs upgrade |
[production] |
| 13:40 |
<ema> |
nginx rolling restart for openssl upgrade on cache hosts |
[production] |
| 10:42 |
<elukey> |
re-pooled aqs1001.eqiad (no node upgrade, need more info about restbase) |
[production] |
| 10:23 |
<godog> |
reduce reserved blocks for /srv on restbase2004 |
[production] |
| 09:59 |
<moritzm> |
installing pcre3 updates |
[production] |
| 09:57 |
<godog> |
start expanding raid0 on restbase2003 |
[production] |
| 09:54 |
<ema> |
nginx rolling restart for openssl upgrade: cp1046, cp1052, cp1068, cp1071, cp1099 |
[production] |
| 09:53 |
<elukey> |
de-pooled aqs1001.eqiad.wmnet as pre-step for nodejs upgrade |
[production] |
| 09:41 |
<godog> |
depool restbase2003 before raid expansion |
[production] |
| 09:37 |
<godog> |
repool restbase2004 |
[production] |
| 09:33 |
<godog> |
deploy restbase ba39d2bcd2f5 to restbase2004 before repooling |
[production] |
| 09:26 |
<volans> |
Re-enabling Puppet on cluster mysql and parsercache to deploy change 279596, T111654 |
[production] |
| 09:10 |
<volans> |
Disabling Puppet on cluster mysql and parsercache to merge and test change 279596 on db2040, T111654 |
[production] |
| 08:50 |
<moritzm> |
installing gnupg updates |
[production] |
| 06:54 |
<moritzm> |
installing apt bugfix updates |
[production] |
| 04:18 |
<TimStarling> |
iridium came back up, but mcelog reports high CPU temperature prior to the shutdown |
[production] |
| 04:13 |
<TimStarling> |
attempting to turn iridium back on via drac. "getraclog" says it powered itself off after resetting four times |
[production] |
| 03:15 |
<greg-g> |
Phabricator is down |
[production] |
| 02:33 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Apr 4 02:33:39 UTC 2016 (duration 8m 31s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.19) (duration: 11m 45s) |
[production] |
| 00:49 |
<krinkle@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s) |
[production] |
| 00:30 |
<krinkle@tin> |
Synchronized w/static.php: (no message) (duration: 00m 35s) |
[production] |
| 00:21 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ruwiki (T45917) |
[production] |
| 00:06 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki metawiki --delete |
[production] |
|
2016-04-03
§
|
| 22:37 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki eswiki --lang-code ca (T45917) |
[production] |
| 22:37 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki eswiki (T45917) |
[production] |
| 22:34 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki eswiki |
[production] |
| 19:16 |
<Krenair> |
mw1146 began to respond with 503s to all requests, tried restarting apache2/hhvm and shortly afterwards it started working again |
[production] |
| 13:26 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: labs copy upload thing. noooooop for prod (duration: 00m 33s) |
[production] |
| 13:02 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: labs copy upload thing. noooooop for prod (duration: 00m 28s) |
[production] |
| 11:37 |
<ori@tin> |
Synchronized wmf-config/extension-list-labs: I0d081186: Also add Newsletter to extension-list-labs (duration: 00m 27s) |
[production] |
| 11:34 |
<ori@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (2/2) (duration: 00m 33s) |
[production] |
| 11:34 |
<ori@tin> |
Synchronized wmf-config/CommonSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (1/2) (duration: 00m 34s) |
[production] |