|
2014-10-29
§
|
| 15:21 |
<reedy> |
Purged l10n cache for 1.25wmf3 |
[production] |
| 15:20 |
<reedy> |
Purged l10n cache for 1.25wmf2 |
[production] |
| 15:19 |
<bd808> |
Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded |
[production] |
| 15:15 |
<bd808> |
Restarted logstash on logstash1001. No MW events were being added to the index. |
[production] |
| 15:10 |
<anomie> |
Synchronized wmf-config/throttle.php: SWAT: Raise account creation throttle at cawiki temporarily [[gerrit:169708]] (duration: 00m 09s) |
[production] |
| 15:07 |
<anomie> |
Synchronized php-1.25wmf5/extensions/Wikidata: SWAT: Fix WikiData "add links" widget JS error [[gerrit:169700]] (duration: 00m 15s) |
[production] |
| 15:07 |
<Reedy> |
Killed old (pre 1.25) l10nupdate cache dirs from tin:/var/lib/l10nupdate |
[production] |
| 15:00 |
<manybubbles> |
started moving shard off of elastic1001, elastic1008, and elastic1013 so we can bounce them to enable hyper threading |
[production] |
| 14:55 |
<manybubbles> |
started rolling shards back to elastic1001, elastic1008, and elastic1013 after hard drive upgrade |
[production] |
| 14:21 |
<ottomata> |
set request.required.acks = 2 for all varnishkafkas |
[production] |
| 13:22 |
<manybubbles> |
lowered replication on logstash's template for new indexes from 3 way to 2 way |
[production] |
| 13:20 |
<demon> |
Synchronized wmf-config/lucene-production.php: unbreak lsearchd for commons, enwikitionary, etc (duration: 00m 04s) |
[production] |
| 13:11 |
<manybubbles> |
lowered redundancy on logstash from 3 way to 2 way |
[production] |
| 13:01 |
<cmjohnson> |
powering down/replacing elastic1017 and elastic1018 |
[production] |
| 12:59 |
<cmjohnson> |
disabling puppet on elastic1017 and 1018 |
[production] |
| 12:01 |
<cmjohnson> |
elastic1001, elastic1008 and elastic1013 powering down to replace ssds RT7779 |
[production] |
| 12:01 |
<springle> |
xtrabackup clone db1007 to db2029 |
[production] |
| 07:16 |
<ori> |
repooled mw1189 w/patched hhvm (<https://phabricator.wikimedia.org/T820#16428>) |
[production] |
| 03:39 |
<ori> |
upgraded mw1114 to custom package with patch from https://phabricator.wikimedia.org/T820#16428 applied |
[production] |
|
2014-10-28
§
|
| 23:12 |
<demon> |
Synchronized php-1.25wmf5/extensions/Wikidata: (no message) (duration: 00m 10s) |
[production] |
| 23:05 |
<_joe_> |
removed stale heap profile files from /run/hhvm on mw1114 |
[production] |
| 23:02 |
<demon> |
Synchronized wmf-config/CommonSettings.php: extension distributor stuffs (duration: 00m 05s) |
[production] |
| 22:09 |
<ejegg> |
updated crm from ffa543cab3eb508fa38b94c6de2643d168b0d507 to 1f0dc2ce0ab84765c085cc0ee369a7a047c0d005 |
[production] |
| 21:05 |
<awight> |
reverted payments, from 647d1eb7d8cccb73fabf5ffded9f713d24576c37 to e3d235f881282120409e1a6ed1a3908ce9a63c26 |
[production] |
| 21:02 |
<hashar> |
Zuul back in action. |
[production] |
| 20:54 |
<hashar> |
Zuul deadlocked again. Restarting Gearman plugin on Jenkins |
[production] |
| 20:53 |
<awight> |
updated payments from 525988487d6bbd08ddad50badd88e34e34104292 to 647d1eb7d8cccb73fabf5ffded9f713d24576c37 |
[production] |
| 20:29 |
<manybubbles> |
removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. |
[production] |
| 20:24 |
<manybubbles> |
disabling puppet on logstash1003 and trying to run elasticserach by hand to learn more about why its borked. |
[production] |
| 19:42 |
<andrewbogott> |
deleting unused labs projects: commons-dev, echo, farsi-wikitest |
[production] |
| 19:33 |
<bd808> |
Elasticsearch not recovering indices at all on logstash1003 and no logging output |
[production] |
| 19:11 |
<reedy> |
Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 15s) |
[production] |
| 19:07 |
<Reedy> |
frwikibooks collation updated |
[production] |
| 19:06 |
<Reedy> |
Running mwscript updateCollation.php --wiki=frwikibooks --previous-collation=uppercase |
[production] |
| 19:05 |
<reedy> |
Synchronized wmf-config/: All of the config changes! (duration: 00m 14s) |
[production] |
| 18:53 |
<reedy> |
Synchronized wmf-config/: Bump cache epoch for Wikidata (duration: 00m 14s) |
[production] |
| 18:52 |
<reedy> |
Finished scap: Split Cite extension, scap to build l10n cache for CiteThisPage (duration: 33m 52s) |
[production] |
| 18:29 |
<bd808|LUNCH> |
restarted elasticsearch node on logstash1003 |
[production] |
| 18:18 |
<reedy> |
Started scap: Split Cite extension, scap to build l10n cache for CiteThisPage |
[production] |
| 18:11 |
<reedy> |
Synchronized wmf-config/: Set = true for Italian Wikipedia in November-December 2014 (duration: 00m 14s) |
[production] |
| 18:05 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf5 |
[production] |
| 17:27 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I56082795: Modify $wmgAddWikiNotify for use by notifyNewProjects (duration: 00m 05s) |
[production] |
| 17:00 |
<bd808> |
restarted logstash on logstash1002 to try and get gelf input events into kibana again |
[production] |
| 16:58 |
<bd808> |
disk utilization on logstash100[123] greater than 80% |
[production] |
| 16:57 |
<cscott> |
no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) |
[production] |
| 16:57 |
<andrew> |
Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 03s) |
[production] |
| 16:56 |
<bd808> |
No new logs in /var/log/elasticsearch for logstash100[123] since Sep 30 06:25 |
[production] |
| 16:55 |
<andrew> |
Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 02s) |
[production] |
| 16:44 |
<reedy> |
Synchronized php-1.25wmf4/extensions/OpenStackManager: (no message) (duration: 00m 14s) |
[production] |
| 15:47 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |