|
2015-08-11
§
|
| 18:53 |
<twentyafterfour@tin> |
Started scap: again: sync new branch 1.26wmf18 and update testwiki |
[production] |
| 18:44 |
<twentyafterfour@tin> |
scap failed: OSError [Errno 1] Operation not permitted: '/srv/mediawiki-staging/wikiversions.php' (duration: 29m 27s) |
[production] |
| 18:37 |
<mutante> |
grafana switched to node krypton (jessie/VM) |
[production] |
| 18:21 |
<bd808> |
logstash log event volume back to normal levels following elasticsearch upgrade |
[production] |
| 18:15 |
<twentyafterfour@tin> |
Started scap: sync new branch 1.26wmf18 and update testwiki |
[production] |
| 18:06 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1006 |
[production] |
| 18:03 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1006; logstash-2015.08.11 shard recovering |
[production] |
| 18:02 |
<bd808> |
upgrading elasticsearch on logstash1006 |
[production] |
| 18:01 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1005 |
[production] |
| 17:43 |
<bd808> |
log event volume in logstash dropped dramatically again; seems to correlate with final recovery of logstash-2015.08.11 shard |
[production] |
| 17:29 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1005; logstash-2015.08.11 shard recovering |
[production] |
| 17:28 |
<bd808> |
upgrading elasticsearch on logstash1005 |
[production] |
| 17:27 |
<bd808> |
logstash event volume recovered after restarting all 3 logstash services |
[production] |
| 17:14 |
<bd808> |
log event volume in logstash dropped dramatically at 16:49; investigating |
[production] |
| 17:13 |
<bd808> |
logstash cluster recovered after upgrade of elasticsearch on logstash1004 |
[production] |
| 16:42 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering |
[production] |
| 16:42 |
<mutante> |
restarted Apache on Etherpad |
[production] |
| 16:38 |
<bd808> |
upgraded elasticsearch to 1.7.1 on logstash1003 |
[production] |
| 16:37 |
<bd808> |
upgraded elaasticsearch to 1.7.1 on logstash1002 |
[production] |
| 16:36 |
<bd808> |
upgraded elaasticsearch to 1.7.1 on logstash1001 |
[production] |
| 16:23 |
<bd808> |
logstash upgrade on logstash1003 complete |
[production] |
| 16:20 |
<bd808> |
logstash upgrade on logstash1002 complete |
[production] |
| 16:16 |
<bd808> |
logstash upgrade on logstash1001 complete |
[production] |
| 15:50 |
<jynus> |
nuking db1002-db1007 on icinga |
[production] |
| 15:49 |
<bd808> |
upgrading logstash on logstash1001 |
[production] |
| 15:47 |
<bd808> |
Trebuchet deploy of logstash/plugins: Add logstash-filter-prune 0.1.5 (36144b2) |
[production] |
| 15:36 |
<bd808> |
Disabled puppet on logstash100[1-3] in preparation for upgrade to 1.5.3 |
[production] |
| 15:31 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (actually rebased this time) (duration: 00m 11s) |
[production] |
| 15:30 |
<bd808@tin> |
Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (actually rebased this time) (duration: 00m 11s) |
[production] |
| 15:17 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: Touched wmf-config/InitialiseSettings.php (duration: 00m 13s) |
[production] |
| 15:12 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (duration: 00m 12s) |
[production] |
| 15:11 |
<bd808@tin> |
Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (duration: 00m 12s) |
[production] |
| 10:45 |
<jynus> |
general maintenance on db1042 (restart, upgrade, db reconstruction) |
[production] |
| 10:38 |
<godog> |
upgrade cassandra on restbase1007 |
[production] |
| 10:31 |
<godog> |
upgrade cassandra on restbase1002 |
[production] |
| 10:25 |
<godog> |
upgrade cassandra on restbase1001 |
[production] |
| 09:56 |
<paravoid> |
switched routing-system autonomous-system to eqiad's subAS on cr1-eqiad/cr2--eqiad |
[production] |
| 09:09 |
<godog> |
reboot ms-be2009, cpu soft lockup |
[production] |
| 05:27 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Aug 11 05:27:33 UTC 2015 (duration 27m 32s) |
[production] |
| 02:26 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf17) at 2015-08-11 02:26:58+00:00 |
[production] |
| 02:23 |
<l10nupdate@tin> |
Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 48s) |
[production] |
| 01:20 |
<ori@tin> |
Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: Revert resourceloader: Add must-revalidate to Cache-Control (duration: 00m 12s) |
[production] |
| 00:10 |
<mutante> |
apache restart on krypton |
[production] |