|
2016-02-11
§
|
| 15:24 |
<elukey> |
puppet re-enabled on mc1010/mc1011 |
[production] |
| 14:41 |
<elukey> |
Stopped puppet, memcached, redis on mc1010/mc1011 for maintenance |
[production] |
| 14:07 |
<elukey> |
removed mc1010/mc1011 from the redis/memcached pools for maintenance |
[production] |
| 14:01 |
<urandom> |
`nodetool stop -- COMPACTION' on restbase1002.eqiad to free disk space (current state: https://phabricator.wikimedia.org/P2593) |
[production] |
| 13:47 |
<jynus> |
backuping, reimaging, restarting and defragmenting db1024 (old s2 master) |
[production] |
| 13:37 |
<paravoid> |
downgrading berkelium to Linux 3.19 & rebooting |
[production] |
| 13:34 |
<apergos> |
reenabled scap runs (removed flag) |
[production] |
| 13:21 |
<jynus> |
migrating dbstore1001 to the new s2-master |
[production] |
| 13:00 |
<paravoid> |
upgrading backup LVS servers to Linux 4.4.0 across all sites |
[production] |
| 12:42 |
<elukey> |
Add mc1008/mc1009 back into redis/memcached pools after maintenance |
[production] |
| 12:40 |
<moritzm> |
uploaded nodejs 4.3.0 for jessie-wikimedia to carbon |
[production] |
| 12:13 |
<elukey> |
re-enabled puppet on mc1009 |
[production] |
| 12:07 |
<elukey> |
enabled puppet on mc1008 |
[production] |
| 11:54 |
<_joe_> |
stopped scap runs for now (touched sync.flag) |
[production] |
| 11:52 |
<moritzm> |
updated xenon, praseodymium and cerium to nodejs 4.3.0 |
[production] |
| 11:45 |
<apergos> |
removed scap3 deb install from all hosts in prod |
[production] |
| 11:33 |
<elukey> |
disabled puppet, memcached, redis on mc1008/mc1009 for maintenance |
[production] |
| 11:12 |
<jynus> |
moving pending s2 slaves to the new master |
[production] |
| 10:52 |
<elukey> |
Correction about mc1008/mc1009 - puppet not disabled, will be done later on. |
[production] |
| 10:50 |
<elukey> |
removed mc1008/mc1009 from redis/memcached pools. Puppet disabled. |
[production] |
| 10:03 |
<godog> |
ms-be2018 / ms-be2019 swift weight to 3500 |
[production] |
| 09:52 |
<elukey> |
re-added mc1007.eqiad back into redis/memcached pools after maintenance |
[production] |
| 09:39 |
<moritzm> |
depooled elastic1023 for hardware problems (T126586) |
[production] |
| 09:31 |
<elukey> |
re-enabled puppet on mc1007.eqiad |
[production] |
| 09:04 |
<jynus> |
restarting HHVM on all running mediawiki job queue processors |
[production] |
| 08:46 |
<_joe_> |
enabled puppet on mw1017, was disabled with no reason |
[production] |
| 08:45 |
<elukey> |
Disabling puppet, redis, memcached on mc1007 for maintenance |
[production] |
| 08:33 |
<elukey> |
removed mc1007 from the redis/memcached pool for Jessie migration |
[production] |
| 06:56 |
<_joe_> |
removing content from sd{d,j}1 written on the root partition on ms-be1008 |
[production] |
| 06:39 |
<_joe_> |
stipped swift on ms-be1008, disk full because object were being written to the root partition |
[production] |
| 06:16 |
<bd808> |
Re-enabled puppet on logstash1003 and forced pupept run |
[production] |
| 06:02 |
<bd808> |
Raised logstash mem limit to -Xms512m on logstash1003 |
[production] |
| 05:57 |
<bd808> |
disabled puppet on logstash1003 to debug configuration |
[production] |
| 05:40 |
<bd808> |
logstash process on logstash1003 flapping; continuing to investigate |
[production] |
| 05:25 |
<bd808> |
Restarted logstash on logstash1003; killed by OOMKiler |
[production] |
| 02:16 |
<ori> |
restarted HHVM on mw1130; locked up (T89912) |
[production] |
| 02:15 |
<bd808> |
Fetched 1271d36 to tin:/srv/mediawiki-staging; should have been updated by sync-wikiversions |
[production] |
| 01:58 |
<tgr@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.13 again |
[production] |
| 01:47 |
<mutante> |
labservices1001 - out of disk |
[production] |
| 01:46 |
<mutante> |
ms-be1008 - delete/gzip large syslog, out of disk |
[production] |
| 01:41 |
<tgr@mira> |
Synchronized wmf-config/mobile.php: fix for T49647 (duration: 02m 15s) |
[production] |
| 01:05 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269883/1 (duration: 02m 11s) |
[production] |
| 00:53 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:48 |
<krenair@mira> |
Synchronized w/static/images/project-logos/wuuwiki.png: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:44 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264937/ (duration: 02m 15s) |
[production] |
| 00:35 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:32 |
<krenair@mira> |
Synchronized w/static/images/project-logos/hiwikiquote.png: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:12 |
<krenair@mira> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/259439/ (duration: 02m 20s) |
[production] |