501-550 of 3108 results (6ms)
2009-06-29 §
21:35 <Andrew> updated Vector skin to r52581 [production]
21:18 <Andrew> Lost connection to zwinger mid-scap. Running scap again to make sure everything's fine. [production]
21:10 <Andrew> Scapping to deploy UsabilityInitiative CSS/JS changes, and license updates done by Fred. [production]
21:09 <Andrew> updated WikimediaMessages to push latest changes pushed to svn by siebrand. [production]
20:58 <Andrew> not scapping yet, deploying license change stuff too, waiting for Fred to do the rest of the updates for that [production]
20:52 <Andrew> scapping [production]
20:49 <Andrew> Activating UsabilityInitiative extension and removing vector from $wgSkipSkins on testwiki [production]
20:38 <RobH_A90> racked and installed mobile1 [production]
20:38 <RobH_A90> pulled nehlam test server [production]
19:01 <Andrew> pdated UsabilityInitiative extension on test as prep for deployment on testwiki. [production]
2009-06-28 §
19:39 <RobH_away> ran authdns-update [production]
19:39 <RobH_away> changing bayes mgmt IP (to make it sane and in line with the rest of the mgmt IP ranges) and adding the new IP to dns and reverse template files [production]
18:58 <brion> bayes back up [production]
18:52 <midom> synchronized php-1.5/db.php 'db28 live' [production]
18:49 <brion> poking LOM on bayes; box is down, EZ requested reboot. [production]
08:54 <domas> manually reset db28 position, based on innodb internal info, got some filesystem relaylog corruption [production]
08:52 <midom> synchronized php-1.5/includes/GlobalFunctions.php 'removing messages profiling hook' [production]
06:43 <Tim> rebooted db28 with /proc/sysrq-trigger [production]
06:29 <Tim> depooled db28, locked up in kswapd, needs reboot [production]
2009-06-27 §
07:42 <domas> changed log expiration on db9 to 100 days [production]
2009-06-26 §
20:33 <Rob> blog.wikimedia.org back online, very restrictive settings and improved security (i hope) [production]
19:09 <Rob> singer https services back online for survey. ocs, wm09schols [production]
18:20 <Rob> Singer Restore: survey.wikimedia.org - UP, ocs.wikimania2009.wikimedia.org - UP [production]
17:11 <Rob> pulled blog.wikimedia.org out of squid via changing its dns to point directly at singer. just to make it easier to secure and fix on the fly later today [production]
15:03 <Rob> firing singer back up for reinstall and restoration from the haxor [production]
14:42 <midom> synchronized php-1.5/includes/parser/ParserCache.php 'removing the pcache hack for now' [production]
11:44 <domas> halted singer, don't start it without contacting me. [production]
07:27 <brion> synchronized php-1.5/includes/parser/ParserCache.php 'Putting live hack for MJ article back. Total traffic spike down but article traffic still spiking.' [production]
05:53 <Tim> rebooting srv182, swap death since ~00:10 UTC [production]
05:50 <Tim> killed broken mysqld_safe instances on srv157, srv161, srv167, srv173, srv174, srv175, srv181, srv185, srv186 [production]
05:17 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv113 by srv142' [production]
05:14 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv182 by srv92' [production]
2009-06-25 §
23:36 <brion> synchronized php-1.5/includes/parser/ParserCache.php 'live hack to extend caching of Michael Jackson' [production]
23:17 <brion> synchronized php-1.5/includes/db/Database.php 'tweak the db err msg' [production]
22:46 <brion> synchronized php-1.5/mc-pmtpa.php 'putting 156 mc back' [production]
22:44 <brion> synchronized php-1.5/mc-pmtpa.php 'swapping in 145 in place of temp down 156' [production]
22:43 <tomaszf> rebooting srv156 due to hard down [production]
22:42 <midom> synchronized php-1.5/db.php '156 down' [production]
18:26 <RobH_A90> srv92 back up with replaced fans [production]
18:16 <RobH_A90> srv92 down due to bad fan, going to replace it with another dead server fan, yay for frankenstien servers [production]
18:09 <RobH_A90> db17 has had a cold reset, service processors are now responsive, as well as system [production]
09:21 <tstarling> synchronized php-1.5/languages/Language.php [production]
2009-06-24 §
16:27 <robh> synchronized php-1.5/InitialiseSettings.php 'bug 19364' [production]
15:56 <Fred> modified /home/wikipedia/conf/nagios/sync to include new nagios server in the sync/restart process. [production]
15:55 <Fred> cleaned up /home/wikipedia/conf/nagios/conf.php to remove unused server / conf parameters. [production]
15:54 <Fred> Cleaned up dsh node_group and resynched Nagios. [production]
10:26 <midom> synchronized php-1.5/includes/specials/SpecialRecentchanges.php 'livehack upper limit, as mediawiki devs are lazy.. oh wait.' [production]
08:29 <domas> compressed databases on db3-5 using lzip, freed up the space. :) [production]
08:16 <midom> synchronized php-1.5/db.php 'db24 back to life' [production]
06:28 <Tim> updated GeSHi to 1.0.8.4 [production]