401-450 of 3457 results (4ms)
2009-07-31 §
18:46 <Fred> re-synched nagios to get rid of 'false' mysql alerts [production]
18:42 <Fred> deployed wikimedia-base (0.20) everywhere. [production]
17:40 <Fred> updated wikimedia-base to include acct and enable SAR in order to do post-crash analysis when necessary. New version: 0.20. (Also changed dependency from ntp-simple to ntp as the ntp-simple package does not exist anymore) [production]
15:17 <mark> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reenabled CentralNotice' [production]
15:15 <kate> synchronized php-1.5/wmf-config/db-pmtpa.php 'remove db26 to dump s1 for TS' [production]
14:15 <brion> removing stray readonly message from db.php on s3 (but not on s3frja... god our config's weird) [production]
14:15 <brion> synchronized php-1.5/wmf-config/db-pmtpa.php [production]
14:14 <brion> noting for toolserver repl fix: old s3 pos was db18-bin.090 454738665 [production]
14:10 <brion> disabling general readonly [production]
14:10 <brion> synchronized php-1.5/wmf-config/CommonSettings.php [production]
14:09 <brion> s3 reset to master_host='db11', master_log_file='db11-bin.001', master_log_pos=79 [production]
14:06 <brion> setting db11 up as temp master on s3 [production]
14:04 <mark> Restarted MySQL on db18, running recovery [production]
14:01 <brion> prepping a manual master switch [production]
13:50 <brion> synchronized php-1.5/wmf-config/db-pmtpa.php 'disabled down db18' [production]
13:49 <brion> removing down db18 [production]
13:47 <brion> synchronized php-1.5/wmf-config/CommonSettings.php [production]
13:28 <mark> synchronized php-1.5/wmf-config/db-pmtpa.php [production]
13:27 <Andrew> Note that sync-file wmf-config/db-pmtpa.php does not work, you use sync-file db-pmtpa.php [production]
13:27 <Andrew> Updated SwitchSettings.php for new location of db.php (wmf-config/db-pmtpa.php) [production]
13:26 <andrew> synchronized php-1.5/wmf-config/db-pmtpa.php 'test' [production]
12:40 <mark> synchronized php-1.5/wmf-config/InitialiseSettings.php 'temporarily disabled CentralNotice' [production]
12:32 <Rob> scheduled reboot of csw5-pmtpa took place at 8am, traffic between esams and pmtpa is very high since as traffic and squid cache misses normalize out. [production]
2009-07-30 §
20:26 <mark> Pooled eximenis as frontend upload squid [production]
19:10 <brion> ran sync-common-all [production]
19:10 <brion> fred fixed login perms on 101, 122. running a sync-common-all to resync deployment [production]
19:02 <brion> srv101, srv122 serving HTTP but not taking updates via ssh [production]
18:29 <brion> installing texvc build & run deps on wikitech box for parser testing [production]
16:07 <robh> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20015' [production]
00:12 <brion> synchronized wmf-deployment/wmf-config/CommonSettings.php 'disable temp req logging on en' [production]
00:11 <brion> synchronized wmf-deployment/api.php 'reenable api to test' [production]
00:08 <brion> synchronized wmf-deployment/api.php 'temp disable api to test' [production]
00:08 <brion> connections still fill up with api disabled [production]
00:07 <brion> synchronized wmf-deployment/wmf-config/CommonSettings.php [production]
00:07 <brion> synchronized wmf-deployment/wmf-config/InitialiseSettings.php [production]
00:04 <brion> setting up some experimental req url logging on backend for enwiki issues [production]
2009-07-29 §
23:43 <brion> temporarily bumping max connections to 6k on db16 [production]
23:39 <brion> new boxes need to be fixed so apache-restart family of scripts work, or else the scripts replaced [production]
18:17 <Rob> srv131 back online [production]
18:17 <Rob> srv118 and srv130 back online [production]
18:15 <Rob> rebooted srv131 for lockup [production]
18:13 <Rob> rebooting srv118 & srv130, locked up [production]
18:11 <Rob> srv110 and srv113 back online [production]
18:08 <Rob> rebooted srv110, srv113, both locked up [production]
16:18 <river> removed stale fingerprint for ns1.wikimedia.org on bayle [production]
14:01 <Rob> shutting down mysql on db12, when done will shut down system and replace bad battery on controller [production]
13:30 <Rob> upgraded dobson memory for squid ssh testing [production]
13:24 <Rob> shutdown eiximenis, upgraded ram [production]
04:36 <Tim> running svn cleanup on locked directories as indicated by find -name lock [production]
04:19 <Tim> resharing ms1:/export/upload with root allowed, so that the above command actually works [production]