401-450 of 10000 results (12ms)
2012-01-16 §
16:06 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33758 - Arabic numerals on Arabic Wiktionary' [production]
15:55 <cmjohnson1> shutting down srv199 for main board replacement [production]
15:09 <mutante> torrus was broken ([[RT:2279]]) and did not start due to corrupted berkeleydb, used db4.8_recover, service started again [production]
12:38 <mark> Running puppet on freshly installed sodium [production]
09:15 <apergos> cleaned up /tmp on srv223... seems like cleanup once an hour by cron isn't often enough any more, scalers are doing too much work [production]
02:05 <LocalisationUpdate> completed (1.18) at Mon Jan 16 02:05:05 UTC 2012 [production]
2012-01-15 §
02:45 <reedy> synchronized php-1.18/extensions/CentralNotice/CentralNotice.db.php '[[rev:108949|r108949]]' [production]
02:05 <LocalisationUpdate> completed (1.18) at Sun Jan 15 02:05:26 UTC 2012 [production]
2012-01-14 §
23:37 <Ryan_Lane> shutting down virt1 [production]
20:29 <Ryan_Lane> stopping opendj on virt1 [production]
20:07 <Ryan_Lane> stopping pdns on virt1 [production]
02:15 <Reedy> That was me testing something [production]
02:14 <LocalisationUpdate> failed: SVN update of extensions failed [production]
02:04 <LocalisationUpdate> completed (1.18) at Sat Jan 14 02:04:46 UTC 2012 [production]
01:00 <Ryan_Lane> brought pdns on virt1 back up [production]
00:53 <Ryan_Lane> stopping pdns on virt1 again to test dns [production]
00:19 <Ryan_Lane> powercycling formey [production]
2012-01-13 §
22:40 <Ryan_Lane> force running puppet on all instances [production]
22:40 <Ryan_Lane> re-generated certificates for all instances [production]
22:40 <Ryan_Lane> deleted all puppet certificates on all instances [production]
22:31 <LeslieCarr> restarting ganglia1001 [production]
21:41 <Ryan_Lane> changing virt1 to be a cname of virt0 [production]
21:17 <Ryan_Lane> killed pdns on virt1 [production]
20:40 <Ryan_Lane> stopped pdns on virt1 [production]
20:32 <reedy> synchronized php-1.18/extensions/Contest/specials/ '[[rev:108843|r108843]]' [production]
20:28 <Ryan_Lane> changed NS records for wmflabs.org and wmflabs to point to virt0 [production]
20:28 <Ryan_Lane> changed recursor to point wmflabs domain to virt0 [production]
20:16 <K4-713> synchronized payments cluster to [[rev:108833|r108833]] [production]
19:30 <Ryan_Lane> shutting down virt1 to ensure migration was completed [production]
19:12 <LeslieCarr> restarting gmond on cp1043 [production]
18:55 <LeslieCarr> hard powercycling ms1002 [production]
18:48 <LeslieCarr> rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896 [production]
16:44 <mutante> added alswiktionary & alswikibooks to closed.dblist [production]
16:43 <dzahn> synchronized closed.dblist [production]
16:33 <mutante> syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 [production]
16:33 <dzahn> synchronized ./wmf-config/InitialiseSettings.php [production]
16:07 <mutante> updated blog theme and installed a plugin per RT 2271 [production]
14:34 <mutante> srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring [production]
12:59 <Guest45399> PXE booting srv191, installing OS [production]
07:45 <Ryan_Lane> disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. [production]
07:43 <Ryan_Lane> added a grant for mediawiki in the database to fix labsconsole mediawiki outage [production]
07:42 <Ryan_Lane> fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue [production]
03:29 <Ryan_Lane> switching labsconsole.wikimedia.org address to point to virt0 [production]
02:58 <Ryan_Lane> dns server is up on virt0 [production]
02:57 <Ryan_Lane> switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed [production]
02:56 <Ryan_Lane> switched rabbitmq server in labs to virt0 [production]
02:56 <Ryan_Lane> switched mysql masters for labs to virt0 [production]
02:05 <LocalisationUpdate> completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 [production]
01:47 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' [production]
01:39 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' [production]