4451-4500 of 10000 results (45ms)
2016-09-21 §
02:45 <mutante> thumbor1002 moved nginx access logs to /srv for more space on / [production]
02:39 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 28s) [production]
02:11 <mutante> thumbor1001/1002 - moved logs from /var/log/thumbor to /srv/thumborlogs to free some space, the actual issue is in /tmp though. lots of systemd-private-* dirs with large sizes. like https://bugzilla.redhat.com/show_bug.cgi?id=1183684 ? [production]
01:58 <awight> update SmashPig from 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c to f1f55096ac20899fa9700aa0f80dece5b47a7fd3 [production]
01:51 <mutante> thumbor servers ran out of disk space [production]
01:15 <awight> updated SmashPig to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c [production]
00:45 <eileen> jobs enabled again - update failed to run due to trigger - which didn't affect staging maybe not on - will edit & try again [production]
2016-09-20 §
23:49 <eileen> disabled Dedupe CiviCRM contacts [production]
23:47 <eileen> disabled Project Dedupe Major gifts contacts [production]
23:47 <eileen> disabled CiviCRM contacts (high numbers) [production]
23:46 <eileen> disabled Thank you mail send [production]
23:46 <eileen> disabled Donations queue consume [production]
23:44 <eileen> CiviCRM update from de1df9ec4fe58487eeb61cc69160f228e009f2cf to 5393a13727e8dbad05ffef9ddd44e965eb9282d1 [production]
23:35 <catrope@tin> Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: Always use requestIdleCallback polyfill for batchEval (T146099) (duration: 00m 46s) [production]
23:22 <catrope@tin> Synchronized php-1.28.0-wmf.19/extensions/Echo/: SWAT (duration: 00m 55s) [production]
23:20 <catrope@tin> Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) [production]
23:17 <catrope@tin> Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) [production]
22:26 <mobrovac> change-prop deploying 4417255 [production]
21:41 <mutante> powercycled mw1294 (down, frozen console) [production]
21:26 <thcipriani> starting branch cut for wmf.20 [production]
21:17 <chasemp> rsync initial transfer of others on labstore1001 to labstore1004 [production]
21:08 <awight> update SmashPig from db68be988194c960aebca691d0fd8e6a6d24246a to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c [production]
20:26 <Pchelolo> restbase deploy ca41acd3f [production]
20:20 <Pchelolo> restbase deploy ca41acd3f canary on restbase1007 [production]
20:13 <Pchelolo> restbase deploy ca41acd3f to staging [production]
20:07 <legoktm@tin> Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Initializes the zoom widget after page loading (duration: 00m 47s) [production]
19:50 <legoktm@tin> Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Makes sure that the zoom widget is initialized before zooming in/out - https://gerrit.wikimedia.org/r/#/c/311765/ (duration: 00m 48s) [production]
17:47 <Pchelolo> update RESTBase to 4829630f [production]
17:27 <Pchelolo> update RESTBase to 4829630f canary on restbase1007 [production]
17:01 <elukey> adding aqs1006 to live traffic - aqs.svc.eqiad.wmnet - T144497 [production]
16:58 <elukey> adding aqs1005 to live traffic - aqs.svc.eqiad.wmnet - T144497 [production]
16:48 <gehel> increase recovery bandwidth on elasticsearch eqiad to match codfw - T145404 [production]
16:32 <elukey> restarting cassandra on aqs100[56] (started the work earlier on today, stopped due to T146130) [production]
14:31 <dcausse> restarting relforge100[12].eqiad.wmnet servers for kernel upgrade and java settings change [production]
14:23 <moritzm> installing tomcat security updates on Ubuntu servers [production]
13:41 <jynus> disabling puppet on labtestweb2001 [production]
13:40 <ottomata> merged --until flag change in check_graphite script (this could affect all graphite based alerts) [production]
13:37 <zeljkof> executed script: mwscript maintenance/updateArticleCount.php --wiki=wikidatawiki --update [production]
13:34 <zeljkof> EU SWAT finished [production]
13:28 <gehel> restarting for elasticsearch and kernel upgrade - eqiad cluster - T145404 / T146123 [production]
13:05 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:308430|Change $wgArticleCountMethod in Wikidata from default (link) to any (T144687)]] (duration: 00m 47s) [production]
11:41 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s) [production]
11:38 <moritzm> upgrading ganeti2001 to Linux 4.4 (ganeti2006 has been promoted to new master node) [production]
11:05 <moritzm> upgrading ganeti2006 to Linux 4.4 [production]
10:49 <moritzm> upgrading ganeti2005 to Linux 4.4 [production]
10:44 <moritzm> reimaging app servers mw1240-mw1242 and API servers mw1194/mw1195 to jessie [production]
10:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s) [production]
10:36 <moritzm> upgrading ganeti2004 to Linux 4.4 [production]
10:28 <akosiaris> force mw2232 to use palladium for report handler testing [production]
10:25 <jynus> deploying schema change on s1 hosts T139090 [production]