|
2011-01-14
§
|
| 06:42 |
<Tim> |
backend squid on sq41-48 all crashed at once, because their root partitions filled up some time ago, and then something HUP'd them all at once. Deleted cache.log and restarted squid. |
[production] |
| 03:45 |
<Tim> |
commented out the $wgCentralAuthCookieDomain = '' in CentralNotice.php |
[production] |
| 03:45 |
<tstarling> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php |
[production] |
| 03:33 |
<mark_> |
db37 binlog: master_log_file='db37-bin.000028', master_log_pos=927356414; |
[production] |
| 03:29 |
<mark_> |
db37 start pos was: master_log_pos=927356414; |
[production] |
| 03:29 |
<mark_> |
domas: change master to master_host='db39', master_user='repl', master_log_file='db39-bin.000050', master_log_pos=720582706; |
[production] |
| 03:26 |
<mark_> |
domas switched s3 master to db39 (position?), and s7 master to db37 (pos?) |
[production] |
| 03:24 |
<mark_> |
domas depooled db25 from s3 and s7 |
[production] |
| 03:18 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 03:18 |
<midom> |
synchronized php-1.5/wmf-config/db.php 's3 is now two-server 5.1 cluster' |
[production] |
| 03:08 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 02:08 |
<tstarling> |
synchronized php-1.5/wmf-config/db.php 'read-only on s3+s7' |
[production] |
| 01:55 |
<Ryan_Lane> |
stopping mysql on db17 |
[production] |
|
2011-01-13
§
|
| 22:35 |
<RobH> |
sq50 done with testing, bringing it back into service |
[production] |
| 22:26 |
<richcole> |
shutting down dataset1 for recabling |
[production] |
| 20:31 |
<RobH> |
pulling sq50 out of rotations, killing puppet and squid processes, and using as my testing for new udev rules on squids |
[production] |
| 20:05 |
<catrope> |
synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r80187' |
[production] |
| 20:05 |
<catrope> |
synchronized php-1.5/extensions/CodeReview/CodeReview.php 'r80187' |
[production] |
| 16:41 |
<catrope> |
synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Live hack that tries to fix login issues' |
[production] |
| 12:57 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26697 - Set $wgForceUIMsgAsContentMsg for tenwiki' |
[production] |
| 05:36 |
<Ryan_Lane> |
repooling sq49 and sq50 |
[production] |
| 04:25 |
<Ryan_Lane> |
depooling sq50 as well |
[production] |
| 04:09 |
<Ryan_Lane> |
starting squid on sq49 |
[production] |
| 04:08 |
<Ryan_Lane> |
stopping squid on sq49 |
[production] |
| 04:06 |
<Ryan_Lane> |
stopping rsyslog, cleaning logs, and reenabling syslog in sq49 |
[production] |
| 04:02 |
<Ryan_Lane> |
depooling sq49 |
[production] |
| 01:47 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 01:31 |
<RobH> |
sq51-sq54 back online |
[production] |
| 01:29 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for test wiki' |
[production] |
| 01:24 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki' |
[production] |
| 01:23 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki' |
[production] |
| 01:20 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 01:20 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php |
[production] |
| 01:19 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'swapping out down servers' |
[production] |
| 01:13 |
<Ryan_Lane> |
fixing torrus |
[production] |
| 01:10 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php |
[production] |
| 01:07 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php |
[production] |
| 01:06 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/1x1.png |
[production] |
| 01:06 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 00:47 |
<RobH> |
sq55-sq58 reinstalled lucid, back online, sq51-54 coming back down |
[production] |
| 00:09 |
<RobH> |
sq55-sq58 coming down for reinstallation to lucid |
[production] |
| 00:05 |
<RobH> |
sq51-sq54 reinstalled lucid, pushing back online |
[production] |
|
2011-01-12
§
|
| 23:32 |
<RobH> |
sq51-sq54 coming down for reinstallation to lucid |
[production] |
| 23:31 |
<RobH> |
sq49, sq50 reinstalled lucid, online |
[production] |
| 23:27 |
<RobH> |
sq46-sq48 reinstalled lucid and back online |
[production] |
| 22:53 |
<RobH> |
sq46-sq50 coming down for reinstallation to lucid |
[production] |
| 22:51 |
<RobH> |
sq42-sq45 reinstalled lucid, online |
[production] |
| 22:13 |
<RobH> |
sq42-sq45 down for reinstallation to lucid |
[production] |
| 20:43 |
<RobH> |
sq41 coming down for reinstallation to lucid |
[production] |
| 18:42 |
<apergos> |
that file needs to be left there for awhile apparently (say the google instructions) so please don't just toss it yet |
[production] |