|
2011-03-08
§
|
| 10:55 |
<Tim> |
running upgrade-1.17wmf1-final.php to upgrade old masters |
[production] |
| 10:53 |
<Tim> |
master positions: db36-bin.000001:106, db13-bin.000001:106, db31-bin.000001:106, db23-bin.001:79 |
[production] |
| 10:53 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:52 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:52 |
<Tim> |
switching master for s5: db14 -> db23 |
[production] |
| 10:52 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:52 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:51 |
<Tim> |
switching master for s4: db22 -> db31 |
[production] |
| 10:51 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:49 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:48 |
<Tim> |
switching master for s2: db30 -> db13 |
[production] |
| 10:48 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:48 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 10:48 |
<Tim> |
switching master for s1: db32 -> db36 |
[production] |
| 10:08 |
<midom> |
synchronized php-1.17/wmf-config/db.php 'adding db33 back with snapshotting disabled - checking if that plus temptables were causing the melt' |
[production] |
| 08:53 |
<jeluf> |
synchronized php-1.17/wmf-config/db.php 'removed db33 from s4' |
[production] |
| 08:51 |
<JeLuF> |
db33 down, commons unavailable |
[production] |
| 07:31 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 're-enabled squid purging' |
[production] |
| 07:24 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh 'temp hack: no priority for htmlCacheUpdate' |
[production] |
| 07:22 |
<Tim> |
it appears more likely that the overload was due to htmlCacheUpdate jobs updating page_touched, in a flood after jobs-loop.sh fixes |
[production] |
| 07:18 |
<Tim> |
apaches overloaded after I fixed job queue squid purging. Disabled all squid purging and stopped job queue. |
[production] |
| 07:15 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php |
[production] |
| 06:47 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh |
[production] |
| 06:40 |
<tstarling> |
synchronized php-1.17/includes/SquidUpdate.php |
[production] |
| 06:40 |
<tstarling> |
synchronized php-1.17/includes/Title.php |
[production] |
| 06:39 |
<tstarling> |
synchronized php-1.17/includes/DefaultSettings.php |
[production] |
| 06:06 |
<tstarling> |
synchronized php-1.17/maintenance/jobs-loop.sh |
[production] |
| 05:51 |
<Tim> |
deployed new jobs-loop.sh script which prioritises fast important jobs like email notification. Thousands of emails are currently being sent. |
[production] |
| 05:23 |
<Tim> |
apt-get upgrade on prototype.wikimedia.org |
[production] |
| 05:19 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'filemover $wgRemoveGroups' |
[production] |
| 05:05 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'added filemover group to enwiki, as on commons, per request on WP:VPR' |
[production] |
| 04:33 |
<Tim> |
re-running upgrade-1.17wmf1-final.php to do db34, db21 and db18 |
[production] |
| 04:31 |
<Tim> |
new master positions: db39-bin.000001:106, db29-bin.001:79, db37-bin.000001:106 |
[production] |
| 04:31 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:30 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:30 |
<Tim> |
doing master switch s7: db18 -> db37 |
[production] |
| 04:29 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:29 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:29 |
<Tim> |
doing master switch s6: db21 -> db29 |
[production] |
| 04:28 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:25 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php |
[production] |
| 04:25 |
<Tim> |
doing master switch: s3 db34 -> db39 |
[production] |
| 02:05 |
<Tim> |
apt-get upgrade on formey |
[production] |
| 01:47 |
<Tim> |
apt-get upgrade on grosely and project2 |
[production] |
| 01:35 |
<Tim> |
apt-get upgrade on mchenry |
[production] |
| 01:22 |
<Tim> |
apt-get upgrade on singer |
[production] |
| 00:53 |
<Tim> |
manually upgraded python-memcache on several servers from 1.44-1wm1 to 1.44-1ubuntu1wm1~hardy1, which apt thought was a downgrade |
[production] |
| 00:22 |
<midom> |
synchronized php-1.17/wmf-config/db.php 'sending db33 back to duty' |
[production] |
| 00:20 |
<RobH> |
filterxsshtml re-enabled on limesurvey, initial testing done |
[production] |