|
2016-02-11
§
|
| 13:00 |
<paravoid> |
upgrading backup LVS servers to Linux 4.4.0 across all sites |
[production] |
| 12:42 |
<elukey> |
Add mc1008/mc1009 back into redis/memcached pools after maintenance |
[production] |
| 12:40 |
<moritzm> |
uploaded nodejs 4.3.0 for jessie-wikimedia to carbon |
[production] |
| 12:13 |
<elukey> |
re-enabled puppet on mc1009 |
[production] |
| 12:07 |
<elukey> |
enabled puppet on mc1008 |
[production] |
| 11:54 |
<_joe_> |
stopped scap runs for now (touched sync.flag) |
[production] |
| 11:52 |
<moritzm> |
updated xenon, praseodymium and cerium to nodejs 4.3.0 |
[production] |
| 11:45 |
<apergos> |
removed scap3 deb install from all hosts in prod |
[production] |
| 11:33 |
<elukey> |
disabled puppet, memcached, redis on mc1008/mc1009 for maintenance |
[production] |
| 11:12 |
<jynus> |
moving pending s2 slaves to the new master |
[production] |
| 10:52 |
<elukey> |
Correction about mc1008/mc1009 - puppet not disabled, will be done later on. |
[production] |
| 10:50 |
<elukey> |
removed mc1008/mc1009 from redis/memcached pools. Puppet disabled. |
[production] |
| 10:03 |
<godog> |
ms-be2018 / ms-be2019 swift weight to 3500 |
[production] |
| 09:52 |
<elukey> |
re-added mc1007.eqiad back into redis/memcached pools after maintenance |
[production] |
| 09:39 |
<moritzm> |
depooled elastic1023 for hardware problems (T126586) |
[production] |
| 09:31 |
<elukey> |
re-enabled puppet on mc1007.eqiad |
[production] |
| 09:04 |
<jynus> |
restarting HHVM on all running mediawiki job queue processors |
[production] |
| 08:46 |
<_joe_> |
enabled puppet on mw1017, was disabled with no reason |
[production] |
| 08:45 |
<elukey> |
Disabling puppet, redis, memcached on mc1007 for maintenance |
[production] |
| 08:33 |
<elukey> |
removed mc1007 from the redis/memcached pool for Jessie migration |
[production] |
| 06:56 |
<_joe_> |
removing content from sd{d,j}1 written on the root partition on ms-be1008 |
[production] |
| 06:39 |
<_joe_> |
stipped swift on ms-be1008, disk full because object were being written to the root partition |
[production] |
| 06:16 |
<bd808> |
Re-enabled puppet on logstash1003 and forced pupept run |
[production] |
| 06:02 |
<bd808> |
Raised logstash mem limit to -Xms512m on logstash1003 |
[production] |
| 05:57 |
<bd808> |
disabled puppet on logstash1003 to debug configuration |
[production] |
| 05:40 |
<bd808> |
logstash process on logstash1003 flapping; continuing to investigate |
[production] |
| 05:25 |
<bd808> |
Restarted logstash on logstash1003; killed by OOMKiler |
[production] |
| 02:16 |
<ori> |
restarted HHVM on mw1130; locked up (T89912) |
[production] |
| 02:15 |
<bd808> |
Fetched 1271d36 to tin:/srv/mediawiki-staging; should have been updated by sync-wikiversions |
[production] |
| 01:58 |
<tgr@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.13 again |
[production] |
| 01:47 |
<mutante> |
labservices1001 - out of disk |
[production] |
| 01:46 |
<mutante> |
ms-be1008 - delete/gzip large syslog, out of disk |
[production] |
| 01:41 |
<tgr@mira> |
Synchronized wmf-config/mobile.php: fix for T49647 (duration: 02m 15s) |
[production] |
| 01:05 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269883/1 (duration: 02m 11s) |
[production] |
| 00:53 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:48 |
<krenair@mira> |
Synchronized w/static/images/project-logos/wuuwiki.png: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:44 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264937/ (duration: 02m 15s) |
[production] |
| 00:35 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:32 |
<krenair@mira> |
Synchronized w/static/images/project-logos/hiwikiquote.png: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:12 |
<krenair@mira> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/259439/ (duration: 02m 20s) |
[production] |
|
2016-02-10
§
|
| 23:49 |
<tgr> |
switched mw1017 to wmf.13 (all groups) |
[production] |
| 23:38 |
<ori@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to php-1.27.0-wmf.12 |
[production] |
| 23:34 |
<ori> |
Restarted HHVM on mw1017 |
[production] |
| 23:27 |
<urandom> |
performing rolling restart of Cassandra in restbase staging (experimental gc settings) |
[production] |
| 23:15 |
<ori@mira> |
Synchronized wmf-config/mobile.php: I6946eccf9c: Better hack for T49647 (duration: 02m 19s) |
[production] |
| 23:10 |
<ori@tin> |
Synchronized php-1.27.0-wmf.12/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 30s) |
[production] |
| 23:08 |
<ori@tin> |
Synchronized php-1.27.0-wmf.13/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 37s) |
[production] |
| 22:49 |
<krinkle@mira> |
Synchronized w/static.php: (no message) (duration: 02m 18s) |
[production] |
| 22:40 |
<krenair@mira> |
Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269755/ (duration: 02m 23s) |
[production] |
| 22:36 |
<urandom> |
rolling restart of Cassandra staging complete (experimental gc settings) |
[production] |