|
2014-10-29
§
|
| 18:58 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf6 |
[production] |
| 18:57 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf5 |
[production] |
| 18:47 |
<reedy> |
Finished scap: testwiki to 1.25wmf6 and build l10n cache (duration: 28m 30s) |
[production] |
| 18:18 |
<reedy> |
Started scap: testwiki to 1.25wmf6 and build l10n cache |
[production] |
| 17:24 |
<cmjohnson> |
shutting down to replace ssds in elastic1002,1007,1014 |
[production] |
| 17:07 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I8dd62e2cc: Re-enable hhvm beta feature on Wikidata (duration: 00m 06s) |
[production] |
| 16:20 |
<manybubbles> |
elastic101[7-9] look good to me - adding them to the cluster |
[production] |
| 16:17 |
<manybubbles> |
shutting down elasticsearch on elastic1002 - its empty and ready to have its disk upgraded/hyper threading enabled |
[production] |
| 16:05 |
<manybubbles> |
ignore my last log message about 1017 - typod |
[production] |
| 16:05 |
<manybubbles> |
shutting down elasticsearch on elastic1007 - its empty and ready to have its disk upgraded/hyper threading enabled |
[production] |
| 16:04 |
<manybubbles> |
shutting down elasticsearch on elastic1014 - its empty and ready to have its disk upgraded/hyper threading enabled |
[production] |
| 16:04 |
<manybubbles> |
shutting down elasticsearch on elastic1017 - its empty and ready to have its disk upgraded/hyper threading enabled |
[production] |
| 15:39 |
<manybubbles> |
start moving shards back to elastic1001 and elastic1008 now that they are up with hyperthreading on |
[production] |
| 15:37 |
<Reedy> |
deleted php-1.24wmf21 from mediawiki-installation |
[production] |
| 15:36 |
<Reedy> |
deleted php-1.24wmf20 from mediawiki-installation |
[production] |
| 15:35 |
<Reedy> |
deleted php-1.24wmf19 from mediawiki-installation |
[production] |
| 15:35 |
<akosiaris> |
uploaded apertium-apy_0.1+svn~57689-1 on apt.wikimedia.org |
[production] |
| 15:23 |
<manybubbles> |
unbanned elastic1013 now that it is back with hyper threading on |
[production] |
| 15:21 |
<reedy> |
Purged l10n cache for 1.25wmf3 |
[production] |
| 15:20 |
<reedy> |
Purged l10n cache for 1.25wmf2 |
[production] |
| 15:19 |
<bd808> |
Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded |
[production] |
| 15:15 |
<bd808> |
Restarted logstash on logstash1001. No MW events were being added to the index. |
[production] |
| 15:10 |
<anomie> |
Synchronized wmf-config/throttle.php: SWAT: Raise account creation throttle at cawiki temporarily [[gerrit:169708]] (duration: 00m 09s) |
[production] |
| 15:07 |
<anomie> |
Synchronized php-1.25wmf5/extensions/Wikidata: SWAT: Fix WikiData "add links" widget JS error [[gerrit:169700]] (duration: 00m 15s) |
[production] |
| 15:07 |
<Reedy> |
Killed old (pre 1.25) l10nupdate cache dirs from tin:/var/lib/l10nupdate |
[production] |
| 15:00 |
<manybubbles> |
started moving shard off of elastic1001, elastic1008, and elastic1013 so we can bounce them to enable hyper threading |
[production] |
| 14:55 |
<manybubbles> |
started rolling shards back to elastic1001, elastic1008, and elastic1013 after hard drive upgrade |
[production] |
| 14:21 |
<ottomata> |
set request.required.acks = 2 for all varnishkafkas |
[production] |
| 13:22 |
<manybubbles> |
lowered replication on logstash's template for new indexes from 3 way to 2 way |
[production] |
| 13:20 |
<demon> |
Synchronized wmf-config/lucene-production.php: unbreak lsearchd for commons, enwikitionary, etc (duration: 00m 04s) |
[production] |
| 13:11 |
<manybubbles> |
lowered redundancy on logstash from 3 way to 2 way |
[production] |
| 13:01 |
<cmjohnson> |
powering down/replacing elastic1017 and elastic1018 |
[production] |
| 12:59 |
<cmjohnson> |
disabling puppet on elastic1017 and 1018 |
[production] |
| 12:01 |
<cmjohnson> |
elastic1001, elastic1008 and elastic1013 powering down to replace ssds RT7779 |
[production] |
| 12:01 |
<springle> |
xtrabackup clone db1007 to db2029 |
[production] |
| 07:16 |
<ori> |
repooled mw1189 w/patched hhvm (<https://phabricator.wikimedia.org/T820#16428>) |
[production] |
| 03:39 |
<ori> |
upgraded mw1114 to custom package with patch from https://phabricator.wikimedia.org/T820#16428 applied |
[production] |
|
2014-10-28
§
|
| 23:12 |
<demon> |
Synchronized php-1.25wmf5/extensions/Wikidata: (no message) (duration: 00m 10s) |
[production] |
| 23:05 |
<_joe_> |
removed stale heap profile files from /run/hhvm on mw1114 |
[production] |
| 23:02 |
<demon> |
Synchronized wmf-config/CommonSettings.php: extension distributor stuffs (duration: 00m 05s) |
[production] |
| 22:09 |
<ejegg> |
updated crm from ffa543cab3eb508fa38b94c6de2643d168b0d507 to 1f0dc2ce0ab84765c085cc0ee369a7a047c0d005 |
[production] |
| 21:05 |
<awight> |
reverted payments, from 647d1eb7d8cccb73fabf5ffded9f713d24576c37 to e3d235f881282120409e1a6ed1a3908ce9a63c26 |
[production] |
| 21:02 |
<hashar> |
Zuul back in action. |
[production] |
| 20:54 |
<hashar> |
Zuul deadlocked again. Restarting Gearman plugin on Jenkins |
[production] |
| 20:53 |
<awight> |
updated payments from 525988487d6bbd08ddad50badd88e34e34104292 to 647d1eb7d8cccb73fabf5ffded9f713d24576c37 |
[production] |
| 20:29 |
<manybubbles> |
removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. |
[production] |
| 20:24 |
<manybubbles> |
disabling puppet on logstash1003 and trying to run elasticserach by hand to learn more about why its borked. |
[production] |
| 19:42 |
<andrewbogott> |
deleting unused labs projects: commons-dev, echo, farsi-wikitest |
[production] |
| 19:33 |
<bd808> |
Elasticsearch not recovering indices at all on logstash1003 and no logging output |
[production] |
| 19:11 |
<reedy> |
Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 15s) |
[production] |