6251-6300 of 10000 results (52ms)
2017-02-21 §
15:17 <hashar> European SWAT complete [production]
15:17 <hashar@tin> Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.hooks.php: Fix site picks: missing from globals (duration: 01m 00s) [production]
15:12 <gehel> restarting kartotherian / tilerator(ui) on maps1* [production]
15:09 <gehel> restarting kartotherian / tilerator(ui) on maps2* [production]
15:06 <gehel> restarting kartotherian / tilerator(ui) on maps-test* [production]
15:06 <godog> roll-restart restbase after statsd move to graphite1001 - T157022 [production]
15:06 <elukey> Increased manually maximum httpd keep alive requests and timeout on bohrium (piwik) - T154558 [production]
14:56 <gehel@puppetmaster1001> conftool action : set/pooled=no; selector: name=elastic10(33|34|38|42).eqiad.wmnet [production]
14:43 <hashar@tin> Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: Add a maintenance script for opt-in T133031 (duration: 00m 41s) [production]
14:35 <moritzm> upgrading openssl on logstash cluster / various base service restarts [production]
14:29 <dcausse> truncated main log file on elastic1030 [production]
14:29 <hashar@tin> Synchronized portals: (no justification provided) (duration: 00m 40s) [production]
14:29 <moritzm> restarting NTP servers on dns_recursors to pick up openssl update (one by one) [production]
14:28 <hashar@tin> Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 40s) [production]
14:25 <moritzm> upgrading openssl on memcached clusters / various base service restarts [production]
14:22 <hashar@tin> Synchronized wmf-config/InitialiseSettings.php: Enable TwoColConflict extension on arwiki - T158493 (duration: 00m 40s) [production]
14:13 <hashar@tin> Synchronized portals: (no justification provided) (duration: 00m 41s) [production]
14:12 <hashar@tin> Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 40s) [production]
14:10 <hashar@tin> Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/: Fix broken site picks feature for compact language links (duration: 01m 04s) [production]
14:08 <hashar@tin> Synchronized wmf-config/InitialiseSettings.php: Enable ReadingDepth logging on Wikipedias - T148262 T155639 (duration: 00m 45s) [production]
14:00 <moritzm> upgrading openssl on maps clusters / various base service restarts [production]
13:41 <elukey> restarting nodejs on aqs1* to pick up openssl security upgrades [production]
13:21 <moritzm> upgrading openssl on aqs cluster / various base service restarts [production]
13:06 <moritzm> upgrading openssl on parsoid clusters / various base service restarts [production]
12:55 <moritzm> upgrading openssl on database servers / various base service restarts [production]
12:53 <volans> re-enabled puppet on neodymium and puppetmaster1001 after Gerrit 330436 was merged T154588 [production]
12:51 <volans> re-enabled puppet on planet2001, was disabled since a week without reason [production]
12:39 <volans> reenabled ircecho aftrer fixing ferm issue and run puppet on affected hosts [production]
12:08 <volans> stopped ircecho temporarily while fixing ferm [production]
12:01 <volans> temporarily disabled puppet on neodymium and puppetmaster1001 to merge Gerrit 330436 T154588 [production]
11:32 <moritzm> upgrading openssl on kafka clusters / various base service restarts [production]
11:15 <moritzm> upgrading openssl on restbase clusters / various base service restarts [production]
11:05 <moritzm> upgrading openssl on hadoop cluster / various base service restarts [production]
11:02 <elukey> rolling restart of cassandra-metrics-collector on aqs1* for T157022 [production]
10:55 <elukey> rolling restart of the analyics jmxtrans daemons for T157022 [production]
10:29 <moritzm> restarting base services on mw2* after openssl update [production]
10:14 <godog> downgrade carbon-c-relay on graphite1001 to trusty's version and bounce daemons [production]
09:58 <moritzm> upgrading mira/tin to HHVM 3.12.14 [production]
09:46 <godog> upgrade graphite on graphite1001 and bounce carbon daemons [production]
09:26 <ema> cp3030: libssl1.1 upgraded to 1.1.0e-1+wmf1, libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1 [production]
08:53 <godog> switch statsd/graphite DNS to graphite1001 - T157022 [production]
08:32 <moritzm> upgrading mw1170-mw1208 to HHVM 3.12.14 [production]
08:30 <gehel> increasing concurrent recoveries / relocations to 8 on elasticsearch eqiad [production]
08:24 <gehel@puppetmaster1001> conftool action : set/pooled=no; selector: name=elastic10(27|32|37|41).eqiad.wmnet [production]
07:31 <marostegui> Deploy alter table enwiki.revision db2055 - T132416 [production]
07:29 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2048 and depool db2055 - T132416 (duration: 00m 51s) [production]
02:24 <l10nupdate@tin> ResourceLoader cache refresh completed at Tue Feb 21 02:24:37 UTC 2017 (duration 5m 20s) [production]
02:19 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.12) (duration: 07m 20s) [production]
01:17 <tstarling@tin> Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 42s) [production]
2017-02-20 §
20:31 <gehel> taking threaddumps and restarting elastic1017 (high load) [production]