|
2016-09-21
§
|
| 03:13 |
<awight> |
update SmashPig from f1f55096ac20899fa9700aa0f80dece5b47a7fd3 to 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda |
[production] |
| 02:46 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Sep 21 02:46:22 UTC 2016 (duration 7m 11s) |
[production] |
| 02:45 |
<mutante> |
thumbor1002 moved nginx access logs to /srv for more space on / |
[production] |
| 02:39 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 28s) |
[production] |
| 02:11 |
<mutante> |
thumbor1001/1002 - moved logs from /var/log/thumbor to /srv/thumborlogs to free some space, the actual issue is in /tmp though. lots of systemd-private-* dirs with large sizes. like https://bugzilla.redhat.com/show_bug.cgi?id=1183684 ? |
[production] |
| 01:58 |
<awight> |
update SmashPig from 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c to f1f55096ac20899fa9700aa0f80dece5b47a7fd3 |
[production] |
| 01:51 |
<mutante> |
thumbor servers ran out of disk space |
[production] |
| 01:15 |
<awight> |
updated SmashPig to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c |
[production] |
| 00:45 |
<eileen> |
jobs enabled again - update failed to run due to trigger - which didn't affect staging maybe not on - will edit & try again |
[production] |
|
2016-09-20
§
|
| 23:49 |
<eileen> |
disabled Dedupe CiviCRM contacts |
[production] |
| 23:47 |
<eileen> |
disabled Project Dedupe Major gifts contacts |
[production] |
| 23:47 |
<eileen> |
disabled CiviCRM contacts (high numbers) |
[production] |
| 23:46 |
<eileen> |
disabled Thank you mail send |
[production] |
| 23:46 |
<eileen> |
disabled Donations queue consume |
[production] |
| 23:44 |
<eileen> |
CiviCRM update from de1df9ec4fe58487eeb61cc69160f228e009f2cf to 5393a13727e8dbad05ffef9ddd44e965eb9282d1 |
[production] |
| 23:35 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: Always use requestIdleCallback polyfill for batchEval (T146099) (duration: 00m 46s) |
[production] |
| 23:22 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.19/extensions/Echo/: SWAT (duration: 00m 55s) |
[production] |
| 23:20 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) |
[production] |
| 23:17 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) |
[production] |
| 22:26 |
<mobrovac> |
change-prop deploying 4417255 |
[production] |
| 21:41 |
<mutante> |
powercycled mw1294 (down, frozen console) |
[production] |
| 21:26 |
<thcipriani> |
starting branch cut for wmf.20 |
[production] |
| 21:17 |
<chasemp> |
rsync initial transfer of others on labstore1001 to labstore1004 |
[production] |
| 21:08 |
<awight> |
update SmashPig from db68be988194c960aebca691d0fd8e6a6d24246a to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c |
[production] |
| 20:26 |
<Pchelolo> |
restbase deploy ca41acd3f |
[production] |
| 20:20 |
<Pchelolo> |
restbase deploy ca41acd3f canary on restbase1007 |
[production] |
| 20:13 |
<Pchelolo> |
restbase deploy ca41acd3f to staging |
[production] |
| 20:07 |
<legoktm@tin> |
Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Initializes the zoom widget after page loading (duration: 00m 47s) |
[production] |
| 19:50 |
<legoktm@tin> |
Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Makes sure that the zoom widget is initialized before zooming in/out - https://gerrit.wikimedia.org/r/#/c/311765/ (duration: 00m 48s) |
[production] |
| 17:47 |
<Pchelolo> |
update RESTBase to 4829630f |
[production] |
| 17:27 |
<Pchelolo> |
update RESTBase to 4829630f canary on restbase1007 |
[production] |
| 17:01 |
<elukey> |
adding aqs1006 to live traffic - aqs.svc.eqiad.wmnet - T144497 |
[production] |
| 16:58 |
<elukey> |
adding aqs1005 to live traffic - aqs.svc.eqiad.wmnet - T144497 |
[production] |
| 16:48 |
<gehel> |
increase recovery bandwidth on elasticsearch eqiad to match codfw - T145404 |
[production] |
| 16:32 |
<elukey> |
restarting cassandra on aqs100[56] (started the work earlier on today, stopped due to T146130) |
[production] |
| 14:31 |
<dcausse> |
restarting relforge100[12].eqiad.wmnet servers for kernel upgrade and java settings change |
[production] |
| 14:23 |
<moritzm> |
installing tomcat security updates on Ubuntu servers |
[production] |
| 13:41 |
<jynus> |
disabling puppet on labtestweb2001 |
[production] |
| 13:40 |
<ottomata> |
merged --until flag change in check_graphite script (this could affect all graphite based alerts) |
[production] |
| 13:37 |
<zeljkof> |
executed script: mwscript maintenance/updateArticleCount.php --wiki=wikidatawiki --update |
[production] |
| 13:34 |
<zeljkof> |
EU SWAT finished |
[production] |
| 13:28 |
<gehel> |
restarting for elasticsearch and kernel upgrade - eqiad cluster - T145404 / T146123 |
[production] |
| 13:05 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:308430|Change $wgArticleCountMethod in Wikidata from default (link) to any (T144687)]] (duration: 00m 47s) |
[production] |
| 11:41 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s) |
[production] |
| 11:38 |
<moritzm> |
upgrading ganeti2001 to Linux 4.4 (ganeti2006 has been promoted to new master node) |
[production] |
| 11:05 |
<moritzm> |
upgrading ganeti2006 to Linux 4.4 |
[production] |
| 10:49 |
<moritzm> |
upgrading ganeti2005 to Linux 4.4 |
[production] |
| 10:44 |
<moritzm> |
reimaging app servers mw1240-mw1242 and API servers mw1194/mw1195 to jessie |
[production] |
| 10:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s) |
[production] |
| 10:36 |
<moritzm> |
upgrading ganeti2004 to Linux 4.4 |
[production] |