|
2011-04-02
§
|
| 11:36 |
<mark> |
Created btrfs filesystem on ms6, striped (raid10 style) over 46 devices - very experimental |
[production] |
| 09:50 |
<mark> |
Reinstalling ms6 with Ubuntu 10.04 |
[production] |
| 09:50 |
<mark> |
Fixed torrus again |
[production] |
| 06:02 |
<mark> |
!wikipedia The image thumbnail servers appear stable now |
[production] |
| 04:59 |
<mark> |
Increased nginx worker processes from 1 to 4, set file limit to 30k |
[production] |
| 04:40 |
<mark> |
!wikipedia Image Thumbnail server outage, it's being worked on |
[production] |
| 04:34 |
<mark> |
Power cycling ms4 again |
[production] |
| 04:06 |
<mark> |
Power cycled ms4 again |
[production] |
| 04:02 |
<mark> |
Removed ms4 from pmtpa.upload config, sending all thumbs to ms5 |
[production] |
| 03:47 |
<mark> |
Restarted rsyncs ms4->ms5 |
[production] |
| 03:25 |
<Ryan_Lane> |
powercycling ms4 again |
[production] |
| 02:59 |
<Ryan_Lane> |
rebooting ms4 |
[production] |
| 02:46 |
<Ryan_Lane> |
seems ms4 is totally dead, powercycling it |
[production] |
| 01:09 |
<Ryan_Lane> |
installing python-pyinotify on spence for an updated ircecho |
[production] |
|
2011-04-01
§
|
| 21:35 |
<Ryan_Lane> |
purging some binlogs on db9 to free up space |
[production] |
| 21:35 |
<RobH> |
bugzilla now version 4 |
[production] |
| 21:31 |
<RobH> |
taking down bugzilla for a quick upgrade |
[production] |
| 18:48 |
<Ryan_Lane> |
added ctwoo, brion, py, and reedy to the engineering alias |
[production] |
| 18:36 |
<mark> |
Deployed ms5.pmtpa.wmnet as a special 'apache' for pmtpa squid uploads... now serving a small portion of commons thumbs |
[production] |
| 18:11 |
<RobH> |
bugzilla back online, CRproxy was affected, and repaired |
[production] |
| 17:30 |
<RobH> |
bugzilla.wikimedia.org going offline for database backup and upgrade |
[production] |
| 17:13 |
<RobH> |
beginning upgrade process for bugzilla, it's availability will be in question during this time |
[production] |
| 16:59 |
<mark> |
Turned off Etag in the webserver7 configuration (/opt/webserver7/https-ms4/config/obj.conf) on ms4 |
[production] |
| 16:50 |
<notpeter> |
rm-ing old binlogs on db9 after confirming that there is no slave lag on db10 |
[production] |
| 15:53 |
<mark> |
Puppetised nginx and htcp purger setup on ms5 |
[production] |
| 11:36 |
<apergos> |
restarted lighty on dataset2 (but why did it die?) |
[production] |
| 00:06 |
<tstarling> |
synchronized php-1.17/includes/specials/SpecialImport.php 'r85099' |
[production] |
|
2011-03-31
§
|
| 20:52 |
<notpeter> |
also, restarting t3h nagioz |
[production] |
| 20:51 |
<notpeter> |
added self to /etc/nagios/contacts.cfg |
[production] |
| 20:32 |
<awjrichards> |
synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'r85088 updates to UserDailyContribs API to retrieve past year edit count' |
[production] |
| 18:58 |
<catrope> |
synchronized php-1.17/includes/specials/SpecialImport.php 'r85078' |
[production] |
| 14:58 |
<aaron> |
synchronized php-1.17/wmf-config/flaggedrevs.php |
[production] |
| 14:58 |
<aaron> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'same cleanup; ptwikinews (just did sqwiki)' |
[production] |
| 14:49 |
<aaron> |
synchronized php-1.17/wmf-config/flaggedrevs.php |
[production] |
| 14:48 |
<aaron> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'moved add/removgroups settings from flaggedrevs.php to here' |
[production] |
| 14:44 |
<mark> |
power cycled singer |
[production] |
| 14:40 |
<aaron> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'added auteditor group on bewiki' |
[production] |
| 14:36 |
<mark> |
Temporarily overriding reload caching request headers on upload squids to ease ms4 load |
[production] |
| 14:35 |
<aaron> |
synchronized php-1.17/wmf-config/flaggedrevs.php 'added auteditor group on bewiki; removed redundant config settings' |
[production] |
| 14:24 |
<mark> |
Remounted /export/thumbs with noatime on ms4 |
[production] |
| 14:21 |
<aaron> |
synchronized php-1.17/wmf-config/flaggedrevs.php 'added stablesettings right to sysop on bewiki' |
[production] |
| 14:01 |
<mark> |
Outage resolved by disabling ehcache |
[production] |
| 13:57 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 13:51 |
<root> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 13:37 |
<mark> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ehcache' |
[production] |
| 11:43 |
<mark> |
set max_stale 1 month on pmtpa.upload squids |
[production] |
| 11:21 |
<mark> |
Set refresh_pattern on pmtpa upload squids |
[production] |
| 10:47 |
<mark> |
Set refresh_pattern on sq85 |
[production] |
| 00:21 |
<Ryan_Lane> |
added drac management script and dependencies to fenari, via puppet |
[production] |
| 00:06 |
<Ryan_Lane> |
make that python-paramiko |
[production] |