|
2011-01-13
§
|
| 04:08 |
<Ryan_Lane> |
stopping squid on sq49 |
[production] |
| 04:06 |
<Ryan_Lane> |
stopping rsyslog, cleaning logs, and reenabling syslog in sq49 |
[production] |
| 04:02 |
<Ryan_Lane> |
depooling sq49 |
[production] |
| 01:47 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 01:31 |
<RobH> |
sq51-sq54 back online |
[production] |
| 01:29 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for test wiki' |
[production] |
| 01:24 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki' |
[production] |
| 01:23 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki' |
[production] |
| 01:20 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 01:20 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php |
[production] |
| 01:19 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'swapping out down servers' |
[production] |
| 01:13 |
<Ryan_Lane> |
fixing torrus |
[production] |
| 01:10 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php |
[production] |
| 01:07 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php |
[production] |
| 01:06 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/1x1.png |
[production] |
| 01:06 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php |
[production] |
| 00:47 |
<RobH> |
sq55-sq58 reinstalled lucid, back online, sq51-54 coming back down |
[production] |
| 00:09 |
<RobH> |
sq55-sq58 coming down for reinstallation to lucid |
[production] |
| 00:05 |
<RobH> |
sq51-sq54 reinstalled lucid, pushing back online |
[production] |
|
2011-01-12
§
|
| 23:32 |
<RobH> |
sq51-sq54 coming down for reinstallation to lucid |
[production] |
| 23:31 |
<RobH> |
sq49, sq50 reinstalled lucid, online |
[production] |
| 23:27 |
<RobH> |
sq46-sq48 reinstalled lucid and back online |
[production] |
| 22:53 |
<RobH> |
sq46-sq50 coming down for reinstallation to lucid |
[production] |
| 22:51 |
<RobH> |
sq42-sq45 reinstalled lucid, online |
[production] |
| 22:13 |
<RobH> |
sq42-sq45 down for reinstallation to lucid |
[production] |
| 20:43 |
<RobH> |
sq41 coming down for reinstallation to lucid |
[production] |
| 18:42 |
<apergos> |
that file needs to be left there for awhile apparently (say the google instructions) so please don't just toss it yet |
[production] |
| 18:40 |
<apergos> |
this is about domain verification for google storage. |
[production] |
| 18:39 |
<ariel> |
synchronized docroot/www.wikimedia.org/google126853a33948578b.html |
[production] |
| 17:48 |
<RobH> |
all text squids in pmtpa/sdtpa have been upgraded to lucid |
[production] |
| 17:48 |
<RobH> |
sq75-sq78 back in service, lucid |
[production] |
| 17:45 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Remove db15 from rotation' |
[production] |
| 17:22 |
<RobH> |
sq75-sq78 coming down for reinstall to lucid |
[production] |
| 17:19 |
<RobH> |
sq71-sq74 reinstalled to lucid and pushed back into service |
[production] |
| 16:39 |
<mark_> |
Increased CARP weight of amssq31 to 30 |
[production] |
| 16:33 |
<mark_> |
Powercycled amssq38 |
[production] |
| 16:30 |
<RobH> |
sq71-sq74 coming down for reinstallation to lucid |
[production] |
| 16:27 |
<RobH> |
sq65-sq66 reinstalled to lucid and online |
[production] |
| 16:20 |
<mark_> |
Lowered CARP weight of amssq* non-SSD text squids from 10 to 8 to relieve disk and memory pressure |
[production] |
| 16:20 |
<apergos> |
running cleanup on index.html and md5sums for the XML dump jobs that recorded bogus "recombine" jobs that weren't actually run |
[production] |
| 16:18 |
<mark_> |
Stopped old puppet instance and restarted squid-frontend on amssq37 |
[production] |
| 16:16 |
<RobH> |
amsq31 upgraded to lucid and back in service |
[production] |
| 16:05 |
<RobH> |
both srv182 and srv183 were not responsive to serial console, rebooted both. |
[production] |
| 15:42 |
<RobH> |
sq65 & sq66 coming down for reinstallation to lucid |
[production] |
| 15:40 |
<RobH> |
sq62-sq64 installed and online lucid |
[production] |
| 15:31 |
<RobH> |
reinstalling amssq31 |
[production] |
| 15:11 |
<RobH> |
sq62-sq64 down for reinstall |
[production] |
| 15:06 |
<RobH> |
sq59-sq61 online as lucid hosts |
[production] |
| 14:59 |
<RobH> |
updated wmf repo, copying package for squid from karmic to lucid as well (seems to have been lost from other changes, as frontend was still there) |
[production] |
| 14:02 |
<RobH> |
sq59-sq61 offline for reinstall |
[production] |