|
2011-03-08
§
|
| 10:48 |
<Tim> |
switching master for s1: db32 -> db36 |
[production] |
| 10:08 |
<midom> |
synchronized php-1.17/wmf-config/db.php 'adding db33 back with snapshotting disabled - checking if that plus temptables were causing the melt' |
[production] |
| 08:53 |
<jeluf> |
synchronized php-1.17/wmf-config/db.php 'removed db33 from s4' |
[production] |
| 08:51 |
<JeLuF> |
db33 down, commons unavailable |
[production] |
| 07:31 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 're-enabled squid purging' |
[production] |
| 07:24 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh 'temp hack: no priority for htmlCacheUpdate' |
[production] |
| 07:22 |
<Tim> |
it appears more likely that the overload was due to htmlCacheUpdate jobs updating page_touched, in a flood after jobs-loop.sh fixes |
[production] |
| 07:18 |
<Tim> |
apaches overloaded after I fixed job queue squid purging. Disabled all squid purging and stopped job queue. |
[production] |
| 07:15 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php |
[production] |
| 06:47 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh |
[production] |
| 06:40 |
<tstarling> |
synchronized php-1.17/includes/SquidUpdate.php |
[production] |
| 06:40 |
<tstarling> |
synchronized php-1.17/includes/Title.php |
[production] |
| 06:39 |
<tstarling> |
synchronized php-1.17/includes/DefaultSettings.php |
[production] |
| 06:06 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh |
[production] |
| 05:51 |
<Tim> |
deployed new jobs-loop.sh script which prioritises fast important jobs like email notification. Thousands of emails are currently being sent. |
[production] |
| 05:23 |
<Tim> |
apt-get upgrade on prototype.wikimedia.org |
[production] |
| 05:19 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'filemover $wgRemoveGroups' |
[production] |
| 05:05 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'added filemover group to enwiki, as on commons, per request on WP:VPR' |
[production] |
| 04:33 |
<Tim> |
re-running upgrade-1.17wmf1-final.php to do db34, db21 and db18 |
[production] |
| 04:31 |
<Tim> |
new master positions: db39-bin.000001:106, db29-bin.001:79, db37-bin.000001:106 |
[production] |
| 04:31 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:30 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:30 |
<Tim> |
doing master switch s7: db18 -> db37 |
[production] |
| 04:29 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:29 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:29 |
<Tim> |
doing master switch s6: db21 -> db29 |
[production] |
| 04:28 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:25 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:25 |
<Tim> |
doing master switch: s3 db34 -> db39 |
[production] |
| 02:05 |
<Tim> |
apt-get upgrade on formey |
[production] |
| 01:47 |
<Tim> |
apt-get upgrade on grosely and project2 |
[production] |
| 01:35 |
<Tim> |
apt-get upgrade on mchenry |
[production] |
| 01:22 |
<Tim> |
apt-get upgrade on singer |
[production] |
| 00:53 |
<Tim> |
manually upgraded python-memcache on several servers from 1.44-1wm1 to 1.44-1ubuntu1wm1~hardy1, which apt thought was a downgrade |
[production] |
| 00:22 |
<midom> |
synchronized php-1.17/wmf-config/db.php 'sending db33 back to duty' |
[production] |
| 00:20 |
<RobH> |
filterxsshtml re-enabled on limesurvey, initial testing done |
[production] |
|
2011-03-07
§
|
| 23:57 |
<Tim> |
running apt-get upgrade on mediawiki-installation servers, for USN-1085-1 |
[production] |
| 23:43 |
<Tim> |
restarted upgrade-1.17wmf1-final.php, which had stopped due to user_last_timestamp existing on some wikis |
[production] |
| 23:42 |
<Tim> |
updating schema change scripts to r83491 and restarting |
[production] |
| 23:30 |
<RobH> |
disabled filterxsshtml in limesurvey install, per awjr |
[production] |
| 23:30 |
<RobH> |
enabled filterxsshtml in limesurvey install |
[production] |
| 23:13 |
<catrope> |
synchronized php-1.17/includes/api/ApiQueryBase.php 'r83490' |
[production] |
| 21:45 |
<catrope> |
ran sync-common-all |
[production] |
| 21:44 |
<RoanKattouw> |
Running sync-common-all to deploy r83487 |
[production] |
| 21:41 |
<hashar> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '27875 - Create a new usergroup Patroller with 'autopatrol' and 'patrol' right on simple.wikipedia' |
[production] |
| 21:24 |
<hashar:> |
subversion working copy is now clean :) |
[production] |
| 21:22 |
<hashar:> |
committing stuff in wmf-commit since the svn repository is available again |
[production] |
| 19:48 |
<midom> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 12:34 |
<hcatlin> |
Deploying mobile fix |
[production] |
| 11:02 |
<Tim> |
removing some duplicate jobs from enwiki.job |
[production] |