|
2015-08-30
§
|
| 12:58 |
<godog> |
lvchange -ay labstore/others on labstore1002 |
[production] |
| 12:52 |
<godog> |
start-nfs on labstore1002 |
[production] |
| 12:31 |
<godog> |
lvchange -ay labstore/tools on labstore1002 |
[production] |
| 12:30 |
<godog> |
also disabled puppet on labstore1002 while investigating |
[production] |
| 12:15 |
<godog> |
trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose |
[production] |
| 11:18 |
<YuviPanda> |
powered labstore1002 back up |
[production] |
| 11:17 |
<YuviPanda> |
shut down labstore1002, going to powercycle from mgmt |
[production] |
| 10:34 |
<YuviPanda> |
disabled backups on labstore1002 to prevent overwriting of good backups on 2001 |
[production] |
| 10:07 |
<YuviPanda> |
run start-nfs in labstore1002 |
[production] |
| 09:54 |
<YuviPanda> |
rebooted labstore1002 |
[production] |
| 04:16 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s) |
[production] |
| 02:23 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00 |
[production] |
| 02:20 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s) |
[production] |
|
2015-08-29
§
|
| 15:26 |
<jynus> |
killing idle mysql connections from phabricator and setting wait and interactive timeout to 60 |
[production] |
| 09:30 |
<jynus> |
SCAP failed, cannot depool db1028 |
[production] |
| 09:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) |
[production] |
| 09:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) |
[production] |
| 09:05 |
<jynus> |
about to depool db1028 due to disk issue |
[production] |
| 04:17 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s) |
[production] |
| 02:24 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00 |
[production] |
| 02:21 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s) |
[production] |
|
2015-08-28
§
|
| 23:44 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s) |
[production] |
| 22:51 |
<bd808@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s) |
[production] |
| 22:05 |
<ori> |
disabling puppet on tin for a few minutes to test an ssh-agent-proxy change |
[production] |
| 20:04 |
<catrope@tin> |
Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s) |
[production] |
| 18:09 |
<robh> |
updating ldap-codfw cert |
[production] |
| 17:10 |
<catrope@tin> |
Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s) |
[production] |
| 17:08 |
<urandom> |
bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings |
[production] |
| 16:03 |
<chasemp> |
ferm for elasticsearch10(0[8-9|1[0-13]) |
[production] |
| 15:23 |
<chasemp> |
ferm for elasticsearch10[14-17] |
[production] |
| 11:09 |
<aude@tin> |
Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s) |
[production] |
| 10:29 |
<godog> |
reenable puppet on ms-fe1, ferm changes will go out on monday |
[production] |
| 09:48 |
<jynus> |
Cloning es1001 database into es1012 |
[production] |
| 09:45 |
<moritzm> |
enabled ferm for swift on esams |
[production] |
| 09:28 |
<moritzm> |
enabled ferm on strontium puppetmaster backend |
[production] |
| 09:00 |
<moritzm> |
enabled ferm on rhodium puppetmaster backend |
[production] |
| 08:29 |
<moritzm> |
uploaded debdeploy 0.0.3 to carbon |
[production] |
| 08:23 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s) |
[production] |
| 05:59 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s) |
[production] |
| 04:58 |
<ori@tin> |
Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s) |
[production] |
| 02:41 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00 |
[production] |
| 02:35 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s) |
[production] |
| 01:59 |
<TimStarling> |
on ruthenium: started parsoid_vd which was previously killed by oom-killer |
[production] |
| 01:58 |
<TimStarling> |
on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things |
[production] |
| 01:37 |
<TimStarling> |
on ruthenium restarted parsoid-rt-client and parsoid-vd-client |
[production] |
| 00:24 |
<mutante> |
powercycled mw2027 |
[production] |
| 00:19 |
<krenair@tin> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s) |
[production] |
| 00:05 |
<krenair@tin> |
Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s) |
[production] |
| 00:05 |
<Krenair> |
Another codfw host broke: mw2027 |
[production] |