|
2009-07-31
§
|
| 20:48 |
<Fred> |
and srv207 |
[production] |
| 20:47 |
<Fred> |
bounced apache on srv129 |
[production] |
| 20:46 |
<Fred> |
added missing srv130 to apaches nodelist and synched common on it. |
[production] |
| 18:46 |
<Fred> |
re-synched nagios to get rid of 'false' mysql alerts |
[production] |
| 18:42 |
<Fred> |
deployed wikimedia-base (0.20) everywhere. |
[production] |
| 17:40 |
<Fred> |
updated wikimedia-base to include acct and enable SAR in order to do post-crash analysis when necessary. New version: 0.20. (Also changed dependency from ntp-simple to ntp as the ntp-simple package does not exist anymore) |
[production] |
| 15:17 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reenabled CentralNotice' |
[production] |
| 15:15 |
<kate> |
synchronized php-1.5/wmf-config/db-pmtpa.php 'remove db26 to dump s1 for TS' |
[production] |
| 14:15 |
<brion> |
removing stray readonly message from db.php on s3 (but not on s3frja... god our config's weird) |
[production] |
| 14:15 |
<brion> |
synchronized php-1.5/wmf-config/db-pmtpa.php |
[production] |
| 14:14 |
<brion> |
noting for toolserver repl fix: old s3 pos was db18-bin.090 454738665 |
[production] |
| 14:10 |
<brion> |
disabling general readonly |
[production] |
| 14:10 |
<brion> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 14:09 |
<brion> |
s3 reset to master_host='db11', master_log_file='db11-bin.001', master_log_pos=79 |
[production] |
| 14:06 |
<brion> |
setting db11 up as temp master on s3 |
[production] |
| 14:04 |
<mark> |
Restarted MySQL on db18, running recovery |
[production] |
| 14:01 |
<brion> |
prepping a manual master switch |
[production] |
| 13:50 |
<brion> |
synchronized php-1.5/wmf-config/db-pmtpa.php 'disabled down db18' |
[production] |
| 13:49 |
<brion> |
removing down db18 |
[production] |
| 13:47 |
<brion> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 13:28 |
<mark> |
synchronized php-1.5/wmf-config/db-pmtpa.php |
[production] |
| 13:27 |
<Andrew> |
Note that sync-file wmf-config/db-pmtpa.php does not work, you use sync-file db-pmtpa.php |
[production] |
| 13:27 |
<Andrew> |
Updated SwitchSettings.php for new location of db.php (wmf-config/db-pmtpa.php) |
[production] |
| 13:26 |
<andrew> |
synchronized php-1.5/wmf-config/db-pmtpa.php 'test' |
[production] |
| 12:40 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'temporarily disabled CentralNotice' |
[production] |
| 12:32 |
<Rob> |
scheduled reboot of csw5-pmtpa took place at 8am, traffic between esams and pmtpa is very high since as traffic and squid cache misses normalize out. |
[production] |
|
2009-07-30
§
|
| 20:26 |
<mark> |
Pooled eximenis as frontend upload squid |
[production] |
| 19:10 |
<brion> |
ran sync-common-all |
[production] |
| 19:10 |
<brion> |
fred fixed login perms on 101, 122. running a sync-common-all to resync deployment |
[production] |
| 19:02 |
<brion> |
srv101, srv122 serving HTTP but not taking updates via ssh |
[production] |
| 18:29 |
<brion> |
installing texvc build & run deps on wikitech box for parser testing |
[production] |
| 16:07 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20015' |
[production] |
| 00:12 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php 'disable temp req logging on en' |
[production] |
| 00:11 |
<brion> |
synchronized wmf-deployment/api.php 'reenable api to test' |
[production] |
| 00:08 |
<brion> |
synchronized wmf-deployment/api.php 'temp disable api to test' |
[production] |
| 00:08 |
<brion> |
connections still fill up with api disabled |
[production] |
| 00:07 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 00:07 |
<brion> |
synchronized wmf-deployment/wmf-config/InitialiseSettings.php |
[production] |
| 00:04 |
<brion> |
setting up some experimental req url logging on backend for enwiki issues |
[production] |
|
2009-07-29
§
|
| 23:43 |
<brion> |
temporarily bumping max connections to 6k on db16 |
[production] |
| 23:39 |
<brion> |
new boxes need to be fixed so apache-restart family of scripts work, or else the scripts replaced |
[production] |
| 18:17 |
<Rob> |
srv131 back online |
[production] |
| 18:17 |
<Rob> |
srv118 and srv130 back online |
[production] |
| 18:15 |
<Rob> |
rebooted srv131 for lockup |
[production] |
| 18:13 |
<Rob> |
rebooting srv118 & srv130, locked up |
[production] |
| 18:11 |
<Rob> |
srv110 and srv113 back online |
[production] |
| 18:08 |
<Rob> |
rebooted srv110, srv113, both locked up |
[production] |
| 16:18 |
<river> |
removed stale fingerprint for ns1.wikimedia.org on bayle |
[production] |
| 14:01 |
<Rob> |
shutting down mysql on db12, when done will shut down system and replace bad battery on controller |
[production] |
| 13:30 |
<Rob> |
upgraded dobson memory for squid ssh testing |
[production] |