|
2015-08-12
§
|
| 07:46 |
<_joe_> |
restarted apertium-apy on sca1001 and sca1002, too many open files, probably leaking |
[production] |
| 06:12 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Aug 12 06:12:31 UTC 2015 (duration 12m 30s) |
[production] |
| 05:54 |
<matt_flaschen> |
Killed support desk conversion. Will resume with profiling tomorrow. |
[production] |
| 04:45 |
<bblack> |
starting slow "apt-get -y upgrade" on cp* (mostly, nginx -> +wmf2), will execute over ~18-24h |
[production] |
| 03:34 |
<krenair@tin> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/229608 (duration: 00m 12s) |
[production] |
| 03:06 |
<ori> |
Installing xdebug on terbium so matt_flaschen can debug memory leak |
[production] |
| 03:04 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf18) at 2015-08-12 03:04:47+00:00 |
[production] |
| 02:58 |
<l10nupdate@tin> |
Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 13s) |
[production] |
| 02:30 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf17) at 2015-08-12 02:30:14+00:00 |
[production] |
| 02:26 |
<l10nupdate@tin> |
Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 52s) |
[production] |
| 01:22 |
<mutante> |
zirconium - shut down, i'm sure, mollyguard |
[production] |
| 00:22 |
<yurik> |
updated kartotherian |
[production] |
| 00:09 |
<mutante> |
restarted gitblit |
[production] |
| 00:06 |
<ori@tin> |
Synchronized php-1.26wmf17/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 32e5bcf90c702 (duration: 00m 13s) |
[production] |
| 00:06 |
<ori@tin> |
Synchronized php-1.26wmf18/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 3ab0b7e0f4948 (duration: 00m 12s) |
[production] |
| 00:02 |
<matt_flaschen> |
Resumed convertLqtPageOnLocalWiki.php run on MediaWiki.org's Project:Support_desk. |
[production] |
|
2015-08-11
§
|
| 23:19 |
<mattflaschen@tin> |
Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for memory leaks (duration: 00m 14s) |
[production] |
| 23:08 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230805/ (duration: 00m 12s) |
[production] |
| 23:05 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230804/ (duration: 00m 12s) |
[production] |
| 21:03 |
<twentyafterfour@tin> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf18 |
[production] |
| 21:02 |
<twentyafterfour> |
deployed scap fixes for my dumb mistakes |
[production] |
| 20:10 |
<ori@tin> |
Synchronized php-1.26wmf18/includes/objectcache/MultiWriteBagOStuff.php: 0acfe6a5bb: Fix argument handling in MultiWriteBagOStuff::get() (duration: 00m 12s) |
[production] |
| 19:28 |
<ori@tin> |
Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 13s) |
[production] |
| 19:28 |
<ori@tin> |
Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 11s) |
[production] |
| 19:26 |
<catrope@tin> |
Synchronized php-1.26wmf18/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: Fix missing editor switcher (duration: 00m 12s) |
[production] |
| 18:58 |
<twentyafterfour@tin> |
Finished scap: again: sync new branch 1.26wmf18 and update testwiki (duration: 04m 58s) |
[production] |
| 18:53 |
<twentyafterfour@tin> |
Started scap: again: sync new branch 1.26wmf18 and update testwiki |
[production] |
| 18:44 |
<twentyafterfour@tin> |
scap failed: OSError [Errno 1] Operation not permitted: '/srv/mediawiki-staging/wikiversions.php' (duration: 29m 27s) |
[production] |
| 18:37 |
<mutante> |
grafana switched to node krypton (jessie/VM) |
[production] |
| 18:21 |
<bd808> |
logstash log event volume back to normal levels following elasticsearch upgrade |
[production] |
| 18:15 |
<twentyafterfour@tin> |
Started scap: sync new branch 1.26wmf18 and update testwiki |
[production] |
| 18:06 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1006 |
[production] |
| 18:03 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1006; logstash-2015.08.11 shard recovering |
[production] |
| 18:02 |
<bd808> |
upgrading elasticsearch on logstash1006 |
[production] |
| 18:01 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1005 |
[production] |
| 17:43 |
<bd808> |
log event volume in logstash dropped dramatically again; seems to correlate with final recovery of logstash-2015.08.11 shard |
[production] |
| 17:29 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1005; logstash-2015.08.11 shard recovering |
[production] |
| 17:28 |
<bd808> |
upgrading elasticsearch on logstash1005 |
[production] |
| 17:27 |
<bd808> |
logstash event volume recovered after restarting all 3 logstash services |
[production] |
| 17:14 |
<bd808> |
log event volume in logstash dropped dramatically at 16:49; investigating |
[production] |
| 17:13 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1004 |
[production] |
| 16:42 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering |
[production] |
| 16:42 |
<mutante> |
restarted Apache on Etherpad |
[production] |
| 16:38 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1003 |
[production] |
| 16:37 |
<bd808> |
upgraded elaasticsearch to 1.7.1 on logstash1002 |
[production] |
| 16:36 |
<bd808> |
upgraded elaasticsearch to 1.7.1 on logstash1001 |
[production] |
| 16:23 |
<bd808> |
logstash upgrade on logstash1003 complete |
[production] |
| 16:20 |
<bd808> |
logstash upgrade on logstash1002 complete |
[production] |
| 16:16 |
<bd808> |
logstash upgrade on logstash1001 complete |
[production] |
| 15:50 |
<jynus> |
nuking db1002-db1007 on icinga |
[production] |