|
2014-10-30
§
|
| 17:18 |
<bd808> |
"Connection error: Unknown error (10.64.16.29)" 1052 in last 5m; 2877 in last 15m |
[production] |
| 17:16 |
<hashar> |
Upgrading Zuul to have the status page emit a Cache-Control header {{bug|72766}} wmf-deploy-20141030-1..wmf-deploy-20141030-2 |
[production] |
| 17:11 |
<bd808> |
Upgraded kibana to v3.1.1 again. Better testing now that logstash is working. |
[production] |
| 17:01 |
<bd808> |
Logs on logstash1003 showed "Failed to flush outgoing items <Errno::EBADF: Bad file descriptor - Bad file descriptor>" on shutdown. Maybe something not quite right about elasticsearch_http plugin? |
[production] |
| 17:00 |
<awight> |
Synchronized php-1.25wmf6/includes/specials/SpecialUpload.php: Parse 'upload_source_url' message on SpecialUpload (duration: 00m 10s) |
[production] |
| 16:59 |
<bd808> |
restarted logstash on logstash1003. No events logged since 00:00Z |
[production] |
| 16:58 |
<awight> |
Synchronized php-1.25wmf5/includes/specials/SpecialUpload.php: Parse 'upload_source_url' message on SpecialUpload (duration: 00m 11s) |
[production] |
| 16:58 |
<bd808> |
restarted logstash on logstash1002. No events logged since 00:00Z |
[production] |
| 16:58 |
<bd808> |
restarted logstash on logstash1001. No events logged since 00:00Z |
[production] |
| 16:55 |
<akosiaris> |
uploaded php5_5.3.10-1ubuntu3.15+wmf1 on apt.wikimedia.org |
[production] |
| 16:46 |
<bd808> |
Reverted kibana to e317bc6 |
[production] |
| 16:44 |
<oblivian> |
Synchronized wmf-config/CommonSettings.php: Serving 15% of anons with HHVM (ludicrous speed!) (duration: 00m 16s) |
[production] |
| 16:38 |
<bd808> |
Upgraded kibana to v3.1.1 via Trebuchet |
[production] |
| 16:38 |
<hashar> |
Zuul status page is freezing because the status.json is being cached :-/ |
[production] |
| 16:31 |
<awight> |
Synchronized php-1.25wmf6/extensions/CentralNotice: push CentralNotice updates (duration: 00m 09s) |
[production] |
| 16:28 |
<awight> |
Synchronized php-1.25wmf5/extensions/CentralNotice: push CentralNotice updates (duration: 00m 11s) |
[production] |
| 16:22 |
<manybubbles> |
moving shards off of elastic1003 and elastic1006 so they can be restarted. elastic1003 need hyperthreading and elastic1006 needs noatime. |
[production] |
| 16:17 |
<cmjohnson> |
powering off elastic1015-16 to replace ssds |
[production] |
| 16:04 |
<hashar> |
restarted Zuul with upgraded version ( wmf-deploy-20140924-1..wmf-deploy-20141030-1 ) |
[production] |
| 16:03 |
<hashar> |
Stopping zuul |
[production] |
| 16:00 |
<hoo> |
Synchronized wmf-config/CommonSettings.php: Fix oauthadmin (duration: 00m 09s) |
[production] |
| 15:43 |
<hashar> |
Going to upgrade Zuul and monitor the result over the next hour. |
[production] |
| 15:39 |
<ottomata> |
starting to reimage mw1032 |
[production] |
| 15:29 |
<oblivian> |
Synchronized wmf-config/CommonSettings.php: Serving 10% of anons with HHVM (duration: 00m 06s) |
[production] |
| 15:22 |
<reedy> |
Synchronized docroot and w: Fix dbtree caching (duration: 00m 15s) |
[production] |
| 15:13 |
<akosiaris> |
upgrading PHP on mw1113 to php5_5.3.10-1ubuntu3.15+wmf1 |
[production] |
| 15:07 |
<manybubbles> |
moving shards off of elastic1015 and elastic1016 so we can replace their hard drives/turn on hyper threading |
[production] |
| 15:07 |
<marktraceur> |
Synchronized php-1.25wmf6/extensions/Wikidata/: [SWAT] [wmf6] Fix edit link for aliases (duration: 00m 12s) |
[production] |
| 14:37 |
<cmjohnson> |
powering down elastic1003-1006 to replace ssds |
[production] |
| 14:33 |
<_joe_> |
pooling mw1031/2 in the hhvm appservers pool |
[production] |
| 12:51 |
<_joe_> |
rebooting mw1030 and mw1031 to use the updated kernel |
[production] |
| 12:48 |
<akosiaris> |
enabled puppet on uranium |
[production] |
| 11:38 |
<_joe_> |
depooling mw1030 and mw1031 for reimaging as hhvm appservers |
[production] |
| 10:15 |
<_joe_> |
load test ended |
[production] |
| 09:48 |
<_joe_> |
load testing the hhvm appserver pool as well |
[production] |
| 08:17 |
<_joe_> |
powercycling mw1189, enabling hyperthreading |
[production] |
| 08:04 |
<_joe_> |
doing the same with mw1189, to see how different appserver generations respond |
[production] |
| 07:25 |
<_joe_> |
raising the weight of mw1114 in the api pool to test the throughput it can withstand |
[production] |
| 04:47 |
<ori> |
enabled heap profiling on mw1189 |
[production] |
|
2014-10-29
§
|
| 23:42 |
<ejegg> |
updated tool from 19928683a8112e9aadd71ba47f199885ba517a02 to 419fb7aa32c6d0776056968378e358ee01985565 |
[production] |
| 23:38 |
<maxsem> |
Synchronized php-1.25wmf6/extensions/MobileFrontend/: (no message) (duration: 00m 07s) |
[production] |
| 23:35 |
<maxsem> |
Synchronized php-1.25wmf5/extensions/MobileFrontend/: (no message) (duration: 00m 04s) |
[production] |
| 23:13 |
<catrope> |
Synchronized php-1.25wmf6/extensions/VisualEditor: SWAT (duration: 00m 04s) |
[production] |
| 23:00 |
<mutante> |
restarting nginx on cp1044 |
[production] |
| 22:11 |
<AaronSchulz> |
Re-running setZoneAccess.php for swift |
[production] |
| 22:04 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts a6a23ac1ec |
[production] |
| 20:28 |
<subbu> |
reverted parsoid to version 617e9e61b625f25d79dfaab08830c396537be632 (due to stuck processes) |
[production] |
| 20:16 |
<reedy> |
Synchronized wmf-config/mc-labs.php: noop for prod (duration: 00m 17s) |
[production] |
| 20:07 |
<arlolra> |
updated Parsoid to version 4e21bdb6fccc377468fd3d1cbc656fb64464ea78 |
[production] |
| 19:45 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 16s) |
[production] |