|
2014-08-21
§
|
| 23:57 |
<ori> |
Started scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 |
[production] |
| 21:58 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.js: I8d27442d1: Workaround for bug introduced by Icf6ede09b (duration: 00m 03s) |
[production] |
| 21:57 |
<manybubbles> |
performing elasticsearch upgrade on elastic1015 |
[production] |
| 21:02 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.util.js: Touch resources/src/mediawiki/mediawiki.util.js (duration: 00m 06s) |
[production] |
| 20:44 |
<godog> |
rolling restart of swift-proxy on ms-fe1* |
[production] |
| 20:11 |
<godog> |
restarted swift-proxy on ms-fe1001 |
[production] |
| 19:55 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 13s) |
[production] |
| 19:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18 |
[production] |
| 19:46 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17 |
[production] |
| 19:44 |
<reedy> |
Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s) |
[production] |
| 19:31 |
<mutante> |
disabled mw1178 in pybal |
[production] |
| 19:27 |
<godog> |
restarted memcached on ms-fe1004 |
[production] |
| 19:23 |
<reedy|webirc> |
mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out |
[production] |
| 19:23 |
<reedy|webirc> |
mw1019 returned [127]: bash: sync-common: command not found |
[production] |
| 19:10 |
<reedy> |
Started scap: testwiki to 1.24wmf18 |
[production] |
| 18:28 |
<manybubbles> |
*victim* |
[production] |
| 18:28 |
<manybubbles> |
trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem. |
[production] |
| 17:29 |
<cmjohnson1> |
removing mw1130 from pybal |
[production] |
| 14:54 |
<hashar> |
Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves. |
[production] |
| 14:36 |
<hashar_> |
Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 |
[production] |
| 10:02 |
<hashar> |
Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. |
[production] |
| 03:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) |
[production] |
| 02:36 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 |
[production] |
| 00:08 |
<MatmaRex> |
(manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. |
[production] |
| 00:07 |
<manybubbles> |
bd808 needs to plan a logstash upgrade soon - let it be logged |
[production] |
| 00:00 |
<manybubbles> |
unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. |
[production] |
|
2014-08-20
§
|
| 23:07 |
<awight> |
stopping the Thank You job |
[production] |
| 22:51 |
<ori> |
disabled puppet on osmium to debug memory leak |
[production] |
| 21:46 |
<marktraceur> |
Synchronized php-1.24wmf17/extensions/MultimediaViewer/: Add disable-by-default option to MultimediaViewer (duration: 00m 07s) |
[production] |
| 21:10 |
<marktraceur> |
Synchronized wmf-config: Turn off Media Viewer for logged-in users at Commons. (duration: 00m 07s) |
[production] |
| 21:06 |
<marktraceur> |
updated /a/common to {{Gerrit|I226bd1468}}: Add item-redirect to OAuth permissions |
[production] |
| 19:50 |
<hashar> |
Restarting Zuul to prettify build results {{bug|66095}} |
[production] |
| 19:48 |
<awight> |
Synchronized php-1.24wmf17/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 05s) |
[production] |
| 19:47 |
<awight> |
Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 04s) |
[production] |
| 19:46 |
<awight> |
Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 07s) |
[production] |
| 16:11 |
<manybubbles> |
elastic1001 upgrade went well - upgrading elastic1002 now |
[production] |
| 15:48 |
<hashar> |
dns: Jenkins will now complain whenever you attempt to send tabs in any file of operations/dns.git {{bug|69478}} |
[production] |
| 15:17 |
<manybubbles> |
manually lowered elasticsearch recovery speeds to stem off high load caused by healing the restart of elastic1001 - we were slowing down enough that we were filling the pool counter |
[production] |
| 15:05 |
<anomie> |
Synchronized wmf-config/CommonSettings.php: SWAT: Add item-redirect to OAuth permissions [[gerrit:155257]] (duration: 00m 09s) |
[production] |
| 15:01 |
<anomie> |
Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/: SWAT: Touch files on advice of Wikidata folks (duration: 00m 09s) |
[production] |
| 15:01 |
<anomie> |
Synchronized wmf-config/Wikibase.php: SWAT: Fix config for specialSiteLinkGroups in Wikibase [[gerrit:155218]] (duration: 00m 09s) |
[production] |
| 14:49 |
<manybubbles> |
installing elasticsearch 1.3.2 on elasticsearch1001 only right now as a test |
[production] |
| 14:47 |
<manybubbles> |
upgrading elasticsearch plugins on all elasticsearch servers in preparation to upgrade to elasticsearch 1.3 - if we roll back we'll have to redeploy the plugins |
[production] |
| 14:11 |
<ottomata> |
changing group ownership and permissions on raw webrequest data in hdfs. Users now must be in the analytics-privatedata-users group to access. |
[production] |
| 13:47 |
<manybubbles> |
experimenting with lowering merge factor on enwiki's Cirrus index - should improve query performance at the cost of more background tasks in the Elasticserach cluster |
[production] |
| 13:37 |
<ottomata> |
disabling puppet on analytics1027 temporarily |
[production] |
| 13:11 |
<godog> |
reboot ms-be1003, xfs errors/panics |
[production] |