|
2010-02-08
§
|
| 12:51 |
<Tim> |
locke went into swap half an hour ago and is now dead. Trying reboot. |
[production] |
| 07:38 |
<Tim> |
stopped recompressTracked.php for now due to bug 20757 |
[production] |
| 07:04 |
<tstarling> |
synchronized php-1.5/includes/Revision.php 'r62120' |
[production] |
| 06:18 |
<Tim> |
cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk |
[production] |
| 06:01 |
<tstarling> |
synchronized php-1.5/wmf-config/db.php 're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months' |
[production] |
| 05:55 |
<Tim> |
added grant for root@fenari to ms3/2 |
[production] |
| 05:31 |
<Tim> |
started apache on srv213, srv161 and srv240, was stopped for no apparent reason |
[production] |
| 04:52 |
<andrew> |
synchronized php-1.5/includes/ChangesList.php 'Merge r62117' |
[production] |
| 04:37 |
<Andrew> |
Deploying LiquidThreads alpha updates with scap |
[production] |
| 04:23 |
<Andrew> |
Planning to deploy LiquidThreads alpha (liquidthreads.labs and test) in the next few minutes |
[production] |
| 01:15 |
<Tim> |
added mysql and gmetric users on srv155 temporarily, to get those services back up |
[production] |
| 00:42 |
<Tim> |
ran apt-get upgrade on srv155. Doing reboot test for new kernel. |
[production] |
|
2010-02-07
§
|
| 23:36 |
<Andrew> |
Clarification: Somebody should fix it, not I think it's fixed now. |
[production] |
| 23:35 |
<Andrew> |
The issue of centralnotice pulling notices from the wrong place should be fixed now, though |
[production] |
| 23:34 |
<Andrew> |
Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing. |
[production] |
| 23:33 |
<Andrew> |
Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6 |
[production] |
| 22:11 |
<Andrew> |
Reports that fundraising banners are appearing on random pages for some users. |
[production] |
| 21:32 |
<domas> |
we're running on single ES copy. |
[production] |
| 21:32 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'disabling broken storage1' |
[production] |
| 20:05 |
<domas> |
removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then. |
[production] |
| 19:01 |
<domas> |
blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action) |
[production] |
|
2010-02-05
§
|
| 06:00 |
<andrew> |
synchronized php-1.5/wmf-config/liquidthreads.php 'Re-activate LiquidThreads email notifications, DoS has passed' |
[production] |
| 05:22 |
<Tim> |
freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs |
[production] |
| 02:45 |
<apergos> |
cleared out /tmp on srv169... |
[production] |
| 00:23 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for jQuery UI stylesheet' |
[production] |
| 00:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for combined.min.css' |
[production] |
| 00:03 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'Resync this, doesn't seem to have been picked up right' |
[production] |
|
2010-02-04
§
|
| 23:55 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 23:55 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62002' |
[production] |
| 23:50 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 23:37 |
<RoanKattouw> |
Running scap to deploy new UsabilityInitiative code |
[production] |
| 22:55 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix' |
[production] |
| 22:54 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 'r61995' |
[production] |
| 22:28 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UsabilityInitiative_alpha on usabilitywiki' |
[production] |
| 22:20 |
<tstarling> |
synchronized php-1.5/includes/api/ApiUpload.php 'disabled chunk upload' |
[production] |
| 22:09 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix' |
[production] |
| 22:07 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php |
[production] |
| 22:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php |
[production] |
| 22:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css |
[production] |
| 22:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js |
[production] |
| 22:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js |
[production] |
| 22:05 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png |
[production] |
| 22:04 |
<RoanKattouw> |
Deploying r61993 with individual sync-files |
[production] |
| 21:15 |
<atglenn> |
turned off hourly snaps on ms7, we'll have daily plus the standard replication |
[production] |
| 18:49 |
<atglenn> |
cleared /tmp on srv175 blah blah |
[production] |
| 18:36 |
<atglenn> |
cleared out php* files from /tmp on srv163 (need cron job) |
[production] |