|
2011-02-18
§
|
| 21:08 |
<RoanKattouw> |
Restarted import of Geograph images. Running in a screen on hume (as import.sh) with 30-min sleeps between batches |
[production] |
| 17:19 |
<mark> |
Created mgmt.eqiad.wmnet and 65.10.in-addr.arpa DNS zones/ORIGINs |
[production] |
| 14:29 |
<catrope> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Bump filter cache version' |
[production] |
| 14:29 |
<catrope> |
synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82402' |
[production] |
| 05:55 |
<tstarling> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php |
[production] |
| 03:48 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27367' |
[production] |
| 02:12 |
<LocalisationUpdate> |
failed |
[production] |
| 00:35 |
<Tim> |
fixed various cron jobs on hume |
[production] |
| 00:30 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh 'fix directory' |
[production] |
| 00:01 |
<tstarling> |
synchronized php-1.17/wmf-config/secure.php 'wgStylePath fix' |
[production] |
|
2011-02-17
§
|
| 23:57 |
<tstarling> |
synchronized php-1.17/includes/DefaultSettings.php |
[production] |
| 23:54 |
<tstarling> |
synchronized php-1.5/includes/DefaultSettings.php 'commenting out the $wgStylePath reference assignment, to try to fix the canary mismatch errors' |
[production] |
| 23:42 |
<Tim> |
fixed logrotate on nfs2 and ran it with -f |
[production] |
| 23:16 |
<catrope> |
synchronized php-1.17/includes/api/ApiQueryPageProps.php 'r82363' |
[production] |
| 23:16 |
<catrope> |
synchronized php-1.17/includes/specials/SpecialContributions.php 'r82363' |
[production] |
| 23:16 |
<catrope> |
synchronized php-1.17/includes/specials/SpecialNewpages.php 'r82363' |
[production] |
| 23:08 |
<catrope> |
synchronized php-1.17/includes/MessageCache.php 'r82362' |
[production] |
| 22:45 |
<catrope> |
synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'r82360' |
[production] |
| 22:38 |
<catrope> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Increment RL minifier cache version again' |
[production] |
| 22:37 |
<catrope> |
synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82358' |
[production] |
| 22:25 |
<catrope> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r82355. Will expire the entire ResourceLoader minifier cache' |
[production] |
| 22:08 |
<catrope> |
synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'Deploy minfier fixes' |
[production] |
| 21:37 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 21:36 |
<catrope> |
synchronized php-1.17/extensions/LiquidThreads_alpha/jquery/plugins.js 'r82349' |
[production] |
| 21:36 |
<catrope> |
synchronized php-1.17/extensions/LiquidThreads/jquery/plugins.js 'r82349' |
[production] |
| 16:06 |
<mark> |
synchronized php-1.5/../php-1.17/includes/media/SVGMetadataExtractor.php |
[production] |
| 15:55 |
<mark> |
Disabled apparmor profile for apache2 on srv178, which was spamming the log |
[production] |
| 15:09 |
<mark> |
Temporarily disabled wikidiff on srv151 (puppet will reenable soon) |
[production] |
| 14:53 |
<mark> |
Cleaned up srv178 |
[production] |
| 04:39 |
<demon> |
synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'Sync wfErrorLog removals, was filling up /home' |
[production] |
| 04:15 |
<apergos1> |
killed udp2log on nfs2 and restarted it via init script after removing preprocessorfatal.log again, got back 160 gb, we had gone down to 6.5 gb free on /home on fenari |
[production] |
| 02:47 |
<Tim> |
re-enabled snaprotate, bug 27471 does not involve data loss |
[production] |
| 02:22 |
<LocalisationUpdate> |
failed |
[production] |
| 01:59 |
<Tim> |
redirected /home/wikipedia/common/php symlink to php-1.17 |
[production] |
| 01:37 |
<Tim> |
disabled snaprotate.pl temporarily while I investigate bug 27471. This has to be reverted within a few days or else the snapshots will fill |
[production] |
| 00:04 |
<Tim> |
rebooting spence, its load is too high to let me get a terminal |
[production] |
|
2011-02-16
§
|
| 21:17 |
<catrope> |
synchronized php-1.17/wmf-config/db.php 'Depool srv178 from ES' |
[production] |
| 21:09 |
<Ryan_Lane|dc> |
on nfs2 |
[production] |
| 21:09 |
<Ryan_Lane|dc> |
killing udp2log and starting it via init script |
[production] |
| 21:08 |
<Ryan_Lane|dc> |
restarting udp2log on nfs2 |
[production] |
| 21:07 |
<Ryan_Lane|dc> |
restarting rsyslog on nfs2 |
[production] |
| 21:05 |
<Ryan_Lane|dc> |
restarting rsyslog on fenari |
[production] |
| 21:00 |
<RoanKattouw> |
Removing /h/w/logs/preprocessorfatal.log , was 146GB |
[production] |
| 20:59 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Remove debug logging, 146 GB of the same error message over and over again is enough' |
[production] |
| 20:39 |
<RobH> |
srv178 powered down, full disk, preventing syncs, needs to be powered backup and fixed later |
[production] |