6351-6400 of 10000 results (54ms)
2017-01-07 §
19:07 <ejegg> disabled dedupe civicrm contacts [production]
16:18 <cwd|afk> re-enabled dedupe civicrm contacts [production]
16:00 <cwd|afk> disabled dedupe civicrm contacts [production]
13:31 <dcausse> elastic@codfw removing/readding replicas for viwiki_general and zhwiki_content (affected by something similar to https://github.com/elastic/elasticsearch/issues/12661) - T154765 [production]
11:35 <_joe_> from medelevium [production]
11:35 <_joe_> restarted apache/otrs, removed a 8 gb error.log [production]
05:11 <Dereckson> Update statistics count on so.wikipedia (T154833) [production]
02:35 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Jan 7 02:35:56 UTC 2017 (duration 5m 20s) [production]
02:30 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.7) (duration: 11m 08s) [production]
2017-01-06 §
23:08 <godog> force puppet run on cache_upload in eqiad to switch thumbs back from codfw [production]
22:38 <demon@tin> Synchronized multiversion: updateBranchPointers consolidation (duration: 00m 56s) [production]
20:14 <demon@tin> Synchronized w: Dropping old entry point (duration: 00m 41s) [production]
19:35 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 40s) [production]
19:29 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/mw.UploadWizard.js: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 59s) [production]
19:17 <ebernhardson> restarting elasticsearch on relforge100[12] to test new search-ltr plugin [production]
19:07 <ostriches> gerrit: Started full reindex of all changes, should be background but will be watching [production]
18:59 <mutante> gerrit restarting for config change 308753 - will be back in seconds [production]
18:13 <mutante> mw1205 - restarted hhvm [production]
18:08 <godog> force puppet run on eqiad cache_upload to switch thumbs to codfw [production]
16:17 <godog> bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation [production]
16:14 <hashar> Restarting Nodepool [production]
16:05 <ema> wiping codfw caches T154758 [production]
15:29 <cmjohnson1> powering off mw1239 to reseat DIMM [production]
15:22 <papaul> elastic2025-elastic2036 - signing puppet certs, salt-key, initial run [production]
14:53 <mark> papaul powercycled asw-a7-codfw 14:50 [production]
14:16 <reedy@tin> Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s) [production]
13:51 <reedy@tin> Started scap: Rebuild message cache for Echo api messages being missing T154110 [production]
10:00 <ariel@tin> Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s) [production]
09:24 <paravoid> asw-a7-codfw is down, serial console unresponsive [production]
09:12 <ariel@tin> Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s) [production]
07:08 <moritzm> installing crypto++ security updates on trusty hosts [production]
04:37 <matt_flaschen> Finished FlowFixInconsistentBoards.php (production mode) on all wikis [production]
04:27 <matt_flaschen> Started FlowFixInconsistentBoards.php (production mode) on all wikis [production]
04:10 <mutante> Icinga now using Letsencrypt cert and all good [production]
03:42 <mutante> icinga - debugging issue with cert change [production]
03:05 <papaul> OS instalaltion on elastic2025-elastic2036 [production]
02:34 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s) [production]
01:27 <bd808> Restarted logstash on logstash1002 (T154732) [production]
01:06 <Krinkle> stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests [production]
01:01 <ostriches> gerrit: back up from upgrade [production]
01:00 <ostriches> gerrit: down for upgrade [production]
00:13 <mutante> analytics1036, ms-fe1003 - ran puppet to fix Icinga [production]
00:11 <mutante> carbon - stopping ganglia-monitor-aggregator for good [production]
2017-01-05 §
23:49 <mutante> rolling out exim4 upgrades (DSA 3747-1) on all remaining eqiad (all-eqiad) [production]
23:46 <godog> fix root-owned files on puppetmaster1001:/var/lib/git/operations/private/ causing /srv/private post-commit hook to fail [production]
23:18 <mutante> switching eqiad ganglia aggregator - running puppet on install1001 - disabling on carbon, re-enabling puppet across eqiad [production]
23:05 <mutante> temp disabling puppet on all eqiad hosts via salt - during ganglia aggregator switch [production]
22:51 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/Echo/includes/api: silence api warnings (duration: 02m 46s) [production]
22:40 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/VisualEditor: silence some api warnings (duration: 02m 48s) [production]
22:22 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/CodeReview/api/ApiQueryCodeComments.php: silence some warnings (duration: 02m 46s) [production]