1001-1050 of 10000 results (10ms)
2012-01-14 §
00:19 <Ryan_Lane> powercycling formey [production]
2012-01-13 §
22:40 <Ryan_Lane> force running puppet on all instances [production]
22:40 <Ryan_Lane> re-generated certificates for all instances [production]
22:40 <Ryan_Lane> deleted all puppet certificates on all instances [production]
22:31 <LeslieCarr> restarting ganglia1001 [production]
21:41 <Ryan_Lane> changing virt1 to be a cname of virt0 [production]
21:17 <Ryan_Lane> killed pdns on virt1 [production]
20:40 <Ryan_Lane> stopped pdns on virt1 [production]
20:32 <reedy> synchronized php-1.18/extensions/Contest/specials/ '[[rev:108843|r108843]]' [production]
20:28 <Ryan_Lane> changed NS records for wmflabs.org and wmflabs to point to virt0 [production]
20:28 <Ryan_Lane> changed recursor to point wmflabs domain to virt0 [production]
20:16 <K4-713> synchronized payments cluster to [[rev:108833|r108833]] [production]
19:30 <Ryan_Lane> shutting down virt1 to ensure migration was completed [production]
19:12 <LeslieCarr> restarting gmond on cp1043 [production]
18:55 <LeslieCarr> hard powercycling ms1002 [production]
18:48 <LeslieCarr> rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896 [production]
16:44 <mutante> added alswiktionary & alswikibooks to closed.dblist [production]
16:43 <dzahn> synchronized closed.dblist [production]
16:33 <mutante> syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 [production]
16:33 <dzahn> synchronized ./wmf-config/InitialiseSettings.php [production]
16:07 <mutante> updated blog theme and installed a plugin per RT 2271 [production]
14:34 <mutante> srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring [production]
12:59 <Guest45399> PXE booting srv191, installing OS [production]
07:45 <Ryan_Lane> disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. [production]
07:43 <Ryan_Lane> added a grant for mediawiki in the database to fix labsconsole mediawiki outage [production]
07:42 <Ryan_Lane> fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue [production]
03:29 <Ryan_Lane> switching labsconsole.wikimedia.org address to point to virt0 [production]
02:58 <Ryan_Lane> dns server is up on virt0 [production]
02:57 <Ryan_Lane> switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed [production]
02:56 <Ryan_Lane> switched rabbitmq server in labs to virt0 [production]
02:56 <Ryan_Lane> switched mysql masters for labs to virt0 [production]
02:05 <LocalisationUpdate> completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 [production]
01:47 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' [production]
01:39 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' [production]
01:09 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' [production]
01:02 <reedy> synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' [production]
01:02 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' [production]
00:33 <Reedy> That was only touch [production]
00:33 <reedy> synchronized wmf-config/InitialiseSettings.php [production]
00:32 <Reedy> srv223 is also out of diskspace [production]
00:29 <Reedy> srv219 is out of diskspace [production]
00:28 <reedy> synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' [production]
00:17 <Ryan_Lane> stopping puppet on all virt nodes [production]
2012-01-12 §
21:54 <Ryan_Lane> relabeled port at virt0 [production]
21:54 <Ryan_Lane> moved new virt0 from squid vlan to public-services2 [production]
21:43 <Ryan_Lane> rebuilding mobile2 as virt0 [production]
21:43 <Ryan_Lane> Adding back mgmt info for mobile1, changing mobile2 to virt0 [production]
21:11 <Ryan_Lane> rebuilding mobile1 as virt0 [production]
21:08 <Ryan_Lane> renaming mobile1 to virt0 [production]
20:54 <binasher> installing percona-toolkit on few remaining hardy dbs [production]