1451-1500 of 10000 results (27ms)
2014-10-29 §
19:39 <reedy> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 16s) [production]
19:26 <reedy> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s) [production]
19:17 <ori> upgraded HHVM to 3.3.0+dfsg1-1+wm1 [production]
18:58 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf6 [production]
18:57 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf5 [production]
18:47 <reedy> Finished scap: testwiki to 1.25wmf6 and build l10n cache (duration: 28m 30s) [production]
18:18 <reedy> Started scap: testwiki to 1.25wmf6 and build l10n cache [production]
17:24 <cmjohnson> shutting down to replace ssds in elastic1002,1007,1014 [production]
17:07 <ori> Synchronized wmf-config/CommonSettings.php: I8dd62e2cc: Re-enable hhvm beta feature on Wikidata (duration: 00m 06s) [production]
16:20 <manybubbles> elastic101[7-9] look good to me - adding them to the cluster [production]
16:17 <manybubbles> shutting down elasticsearch on elastic1002 - its empty and ready to have its disk upgraded/hyper threading enabled [production]
16:05 <manybubbles> ignore my last log message about 1017 - typod [production]
16:05 <manybubbles> shutting down elasticsearch on elastic1007 - its empty and ready to have its disk upgraded/hyper threading enabled [production]
16:04 <manybubbles> shutting down elasticsearch on elastic1014 - its empty and ready to have its disk upgraded/hyper threading enabled [production]
16:04 <manybubbles> shutting down elasticsearch on elastic1017 - its empty and ready to have its disk upgraded/hyper threading enabled [production]
15:39 <manybubbles> start moving shards back to elastic1001 and elastic1008 now that they are up with hyperthreading on [production]
15:37 <Reedy> deleted php-1.24wmf21 from mediawiki-installation [production]
15:36 <Reedy> deleted php-1.24wmf20 from mediawiki-installation [production]
15:35 <Reedy> deleted php-1.24wmf19 from mediawiki-installation [production]
15:35 <akosiaris> uploaded apertium-apy_0.1+svn~57689-1 on apt.wikimedia.org [production]
15:23 <manybubbles> unbanned elastic1013 now that it is back with hyper threading on [production]
15:21 <reedy> Purged l10n cache for 1.25wmf3 [production]
15:20 <reedy> Purged l10n cache for 1.25wmf2 [production]
15:19 <bd808> Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded [production]
15:15 <bd808> Restarted logstash on logstash1001. No MW events were being added to the index. [production]
15:10 <anomie> Synchronized wmf-config/throttle.php: SWAT: Raise account creation throttle at cawiki temporarily [[gerrit:169708]] (duration: 00m 09s) [production]
15:07 <anomie> Synchronized php-1.25wmf5/extensions/Wikidata: SWAT: Fix WikiData "add links" widget JS error [[gerrit:169700]] (duration: 00m 15s) [production]
15:07 <Reedy> Killed old (pre 1.25) l10nupdate cache dirs from tin:/var/lib/l10nupdate [production]
15:00 <manybubbles> started moving shard off of elastic1001, elastic1008, and elastic1013 so we can bounce them to enable hyper threading [production]
14:55 <manybubbles> started rolling shards back to elastic1001, elastic1008, and elastic1013 after hard drive upgrade [production]
14:21 <ottomata> set request.required.acks = 2 for all varnishkafkas [production]
13:22 <manybubbles> lowered replication on logstash's template for new indexes from 3 way to 2 way [production]
13:20 <demon> Synchronized wmf-config/lucene-production.php: unbreak lsearchd for commons, enwikitionary, etc (duration: 00m 04s) [production]
13:11 <manybubbles> lowered redundancy on logstash from 3 way to 2 way [production]
13:01 <cmjohnson> powering down/replacing elastic1017 and elastic1018 [production]
12:59 <cmjohnson> disabling puppet on elastic1017 and 1018 [production]
12:01 <cmjohnson> elastic1001, elastic1008 and elastic1013 powering down to replace ssds RT7779 [production]
12:01 <springle> xtrabackup clone db1007 to db2029 [production]
07:16 <ori> repooled mw1189 w/patched hhvm (<https://phabricator.wikimedia.org/T820#16428>) [production]
03:39 <ori> upgraded mw1114 to custom package with patch from https://phabricator.wikimedia.org/T820#16428 applied [production]
2014-10-28 §
23:12 <demon> Synchronized php-1.25wmf5/extensions/Wikidata: (no message) (duration: 00m 10s) [production]
23:05 <_joe_> removed stale heap profile files from /run/hhvm on mw1114 [production]
23:02 <demon> Synchronized wmf-config/CommonSettings.php: extension distributor stuffs (duration: 00m 05s) [production]
22:09 <ejegg> updated crm from ffa543cab3eb508fa38b94c6de2643d168b0d507 to 1f0dc2ce0ab84765c085cc0ee369a7a047c0d005 [production]
21:05 <awight> reverted payments, from 647d1eb7d8cccb73fabf5ffded9f713d24576c37 to e3d235f881282120409e1a6ed1a3908ce9a63c26 [production]
21:02 <hashar> Zuul back in action. [production]
20:54 <hashar> Zuul deadlocked again. Restarting Gearman plugin on Jenkins [production]
20:53 <awight> updated payments from 525988487d6bbd08ddad50badd88e34e34104292 to 647d1eb7d8cccb73fabf5ffded9f713d24576c37 [production]
20:29 <manybubbles> removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. [production]
20:24 <manybubbles> disabling puppet on logstash1003 and trying to run elasticserach by hand to learn more about why its borked. [production]