251-300 of 10000 results (8ms)
2012-01-13 §
16:33 <mutante> syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 [production]
16:33 <dzahn> synchronized ./wmf-config/InitialiseSettings.php [production]
16:07 <mutante> updated blog theme and installed a plugin per RT 2271 [production]
14:34 <mutante> srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring [production]
12:59 <Guest45399> PXE booting srv191, installing OS [production]
07:45 <Ryan_Lane> disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. [production]
07:43 <Ryan_Lane> added a grant for mediawiki in the database to fix labsconsole mediawiki outage [production]
07:42 <Ryan_Lane> fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue [production]
03:29 <Ryan_Lane> switching labsconsole.wikimedia.org address to point to virt0 [production]
02:58 <Ryan_Lane> dns server is up on virt0 [production]
02:57 <Ryan_Lane> switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed [production]
02:56 <Ryan_Lane> switched rabbitmq server in labs to virt0 [production]
02:56 <Ryan_Lane> switched mysql masters for labs to virt0 [production]
02:05 <LocalisationUpdate> completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 [production]
01:47 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' [production]
01:39 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' [production]
01:09 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' [production]
01:02 <reedy> synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' [production]
01:02 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' [production]
00:33 <Reedy> That was only touch [production]
00:33 <reedy> synchronized wmf-config/InitialiseSettings.php [production]
00:32 <Reedy> srv223 is also out of diskspace [production]
00:29 <Reedy> srv219 is out of diskspace [production]
00:28 <reedy> synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' [production]
00:17 <Ryan_Lane> stopping puppet on all virt nodes [production]
2012-01-12 §
21:54 <Ryan_Lane> relabeled port at virt0 [production]
21:54 <Ryan_Lane> moved new virt0 from squid vlan to public-services2 [production]
21:43 <Ryan_Lane> rebuilding mobile2 as virt0 [production]
21:43 <Ryan_Lane> Adding back mgmt info for mobile1, changing mobile2 to virt0 [production]
21:11 <Ryan_Lane> rebuilding mobile1 as virt0 [production]
21:08 <Ryan_Lane> renaming mobile1 to virt0 [production]
20:54 <binasher> installing percona-toolkit on few remaining hardy dbs [production]
20:26 <cmjohnson1> shutting down srv178-189 for decommissioning [production]
20:14 <binasher> granted the "process" priv to nagios@localhost on all production db clusters [production]
20:07 <reedy> synchronized php-1.18/includes/specials/SpecialSearch.php '[[rev:108751|r108751]]' [production]
20:07 <LeslieCarr> reassigning ports on asw-b-sdtpa [production]
17:00 <notpeter> stop sodium to do manual reinstall [production]
16:33 <RobH> adjusting all power strip humidity sensor 2 (floor level) to 12% humidity, as the center rack has the proper levels, floor levels always are low in humidity. [production]
16:04 <mutante> starting nagios-nrpe-server on ALL via dsh to speed up nagios recovery [production]
15:33 <mutante> starting nagios-nrpe-server on srv's via dsh [production]
02:04 <LocalisationUpdate> completed (1.18) at Thu Jan 12 02:04:31 UTC 2012 [production]
00:48 <preilly> pushing quick fix for special random [production]
00:48 <preilly> synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to fix random link' [production]
00:41 <LeslieCarr> added ganglia1002 and ganglia1001 to dns [production]
2012-01-11 §
23:18 <RobH> searchidx1001 offline and powered down until replacement memory arrives (2012-01-13) rt 2208 [production]
22:56 <RobH> poking searchidx1001 for memory error [production]
22:45 <RobH> mw1108 online and ready for install per rt2253 [production]
22:42 <RobH> mw1099 repaired, ready for os install per rt2252 [production]
22:39 <RobH> mw1081 ready for install rt2251 [production]
22:32 <RobH> no its not ;] [production]