351-400 of 2969 results (6ms)
2009-06-30 §
03:34 <Rob> seeing good results with the firewall plugin on the corporate blog, copying it over to the techblog and setting it to active. [production]
02:57 <Rob> It was net gremlins! [production]
02:56 <brion> appears to have been temporary networking issue (external?) affecting access to some clients [production]
02:48 <brion> some sort of downtime reported for a few minutes. no apparent current problems persisting, all looks well [production]
02:11 <hcatlin> Mobile site is having some encoding issues with UTF-8 characters, therefore the redirects have been disabled from common.js [production]
2009-06-29 §
23:03 <andrew> synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'Updated l10n' [production]
23:00 <Fred> updated DNS for *.m.wikipedia.org to point to mobile1 instead of eiximenis [production]
22:05 <andrew> synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'Trevor told me to!' [production]
22:05 <andrew> synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.css 'Trevor told me to!' [production]
21:47 <Fred> apache stuck on srv[210-214] - restarting. [production]
21:36 <Andrew> scapping for vector updates [production]
21:35 <Andrew> updated Vector skin to r52581 [production]
21:18 <Andrew> Lost connection to zwinger mid-scap. Running scap again to make sure everything's fine. [production]
21:10 <Andrew> Scapping to deploy UsabilityInitiative CSS/JS changes, and license updates done by Fred. [production]
21:09 <Andrew> updated WikimediaMessages to push latest changes pushed to svn by siebrand. [production]
20:58 <Andrew> not scapping yet, deploying license change stuff too, waiting for Fred to do the rest of the updates for that [production]
20:52 <Andrew> scapping [production]
20:49 <Andrew> Activating UsabilityInitiative extension and removing vector from $wgSkipSkins on testwiki [production]
20:38 <RobH_A90> racked and installed mobile1 [production]
20:38 <RobH_A90> pulled nehlam test server [production]
19:01 <Andrew> pdated UsabilityInitiative extension on test as prep for deployment on testwiki. [production]
2009-06-28 §
19:39 <RobH_away> ran authdns-update [production]
19:39 <RobH_away> changing bayes mgmt IP (to make it sane and in line with the rest of the mgmt IP ranges) and adding the new IP to dns and reverse template files [production]
18:58 <brion> bayes back up [production]
18:52 <midom> synchronized php-1.5/db.php 'db28 live' [production]
18:49 <brion> poking LOM on bayes; box is down, EZ requested reboot. [production]
08:54 <domas> manually reset db28 position, based on innodb internal info, got some filesystem relaylog corruption [production]
08:52 <midom> synchronized php-1.5/includes/GlobalFunctions.php 'removing messages profiling hook' [production]
06:43 <Tim> rebooted db28 with /proc/sysrq-trigger [production]
06:29 <Tim> depooled db28, locked up in kswapd, needs reboot [production]
2009-06-27 §
07:42 <domas> changed log expiration on db9 to 100 days [production]
2009-06-26 §
20:33 <Rob> blog.wikimedia.org back online, very restrictive settings and improved security (i hope) [production]
19:09 <Rob> singer https services back online for survey. ocs, wm09schols [production]
18:20 <Rob> Singer Restore: survey.wikimedia.org - UP, ocs.wikimania2009.wikimedia.org - UP [production]
17:11 <Rob> pulled blog.wikimedia.org out of squid via changing its dns to point directly at singer. just to make it easier to secure and fix on the fly later today [production]
15:03 <Rob> firing singer back up for reinstall and restoration from the haxor [production]
14:42 <midom> synchronized php-1.5/includes/parser/ParserCache.php 'removing the pcache hack for now' [production]
11:44 <domas> halted singer, don't start it without contacting me. [production]
07:27 <brion> synchronized php-1.5/includes/parser/ParserCache.php 'Putting live hack for MJ article back. Total traffic spike down but article traffic still spiking.' [production]
05:53 <Tim> rebooting srv182, swap death since ~00:10 UTC [production]
05:50 <Tim> killed broken mysqld_safe instances on srv157, srv161, srv167, srv173, srv174, srv175, srv181, srv185, srv186 [production]
05:17 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv113 by srv142' [production]
05:14 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv182 by srv92' [production]
2009-06-25 §
23:36 <brion> synchronized php-1.5/includes/parser/ParserCache.php 'live hack to extend caching of Michael Jackson' [production]
23:17 <brion> synchronized php-1.5/includes/db/Database.php 'tweak the db err msg' [production]
22:46 <brion> synchronized php-1.5/mc-pmtpa.php 'putting 156 mc back' [production]
22:44 <brion> synchronized php-1.5/mc-pmtpa.php 'swapping in 145 in place of temp down 156' [production]
22:43 <tomaszf> rebooting srv156 due to hard down [production]
22:42 <midom> synchronized php-1.5/db.php '156 down' [production]
18:26 <RobH_A90> srv92 back up with replaced fans [production]