151-200 of 8686 results (4ms)
2010-09-21 §
02:43 <laner> synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info' [production]
02:39 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info' [production]
02:13 <tfinc> synchronizing Wikimedia installation... Revision: 72920 [production]
02:12 <Tim> rebooting locke and sanger for kernel upgrade [production]
02:09 <Tim> rebooting hume for kernel upgrade [production]
01:33 <tfinc> ran sync-common-all [production]
01:12 <Tim-away> fixed syntax error in InitialiseSettings.php, which caused the site to go down [production]
2010-09-20 §
22:23 <Ryan_Lane> finished kernel upgrades on tesla VMs [production]
20:54 <Ryan_Lane> moved mysql data from /var/lib/mysql to /a/sqldata on prototype [production]
18:33 <Ryan_Lane> Correction: previous ufw logs were for eiximenis [production]
18:33 <Ryan_Lane> added ufw application configuration for etherpad. Added allow rules for OpenSSH and Etherpad in ufw config. Enabled ufw. [production]
18:32 <Ryan_Lane> added ufw prerouting rules for 80->9000 for etherpad. [production]
18:32 <Ryan_Lane> restarted etherpad on eiximenis [production]
17:46 <Ryan_Lane> upgrading kernel on all tesla VMs [production]
16:30 <Ryan_Lane> running database recovery on torrus on streber, and restarting the service [production]
14:29 <Tim> rebooting mchenry for kernel upgrade [production]
13:39 <Tim> rebooting ersch, erzurumi, formey and kaulen for new kernel [production]
12:34 <Tim> rebooting brewster and eiximenis for new kernel [production]
12:25 <Tim> updated browne /etc/motd [production]
08:49 <Tim> rebooting alsted for new kernel [production]
07:35 <Tim> rebooting fenari to get new kernel [production]
07:03 <Tim> upgrading everything on fenari [production]
2010-09-17 §
20:56 <RobH> db7 & db8 no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding on both [production]
20:35 <RobH> locke no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding, closing rt#114 [production]
19:36 <RobH> db28 coming down for fanboard replacement (not hot swappable since its the controller) [production]
19:30 <RobH> all SP back online for search1-search12 reporting error free. [production]
19:25 <RobH> resetting the SP on search1-search12 to make them forget the spare powersupply they really do not have. [production]
19:16 <RobH> full power reset cleared drac errors, noted for followup, system back online. [production]
19:07 <RobH> srv206 has all kinds of errors, working on it, ignore nagios flaps. [production]
18:49 <RobH> db34 online and ready to be setup into database deployments. [production]
00:15 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Fixing placement of VariablePage extension' [production]
2010-09-16 §
21:39 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding enwiki to variable page extension' [production]
21:38 <Ryan_Lane> fixed puppet issue with ganglia on memcache servers (a bad puppet file had previously been pushed) [production]
21:35 <tfinc> ran sync-common-all [production]
21:27 <tfinc> synchronized php-1.5/extensions/VariablePage/VariablePage.i18n.php 'Adding variable page extension for donate link' [production]
17:36 <Ryan_Lane> took sq33 out of lvs rotation [production]
16:02 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25142: Redisable anon page creation, there was no consensus' [production]
14:23 <RobH> sq60,sq73,sq75 coming back into service [production]
14:13 <RobH> fixed sq33, working on backend on sq60,sq73,sq75, all having cache cleaned [production]
13:54 <RobH> taking another look at sq33, pulled from service [production]
13:49 <RobH> working on srv206 [production]
13:46 <RobH> sq73 & sq75 up [production]
13:35 <RobH> sq38 & sq60 back in service [production]
13:29 <RobH> rebooted sq38 & sq60 to bring them back online [production]
12:45 <mark> Powercycled amssq52 [production]
12:37 <mark> knsq3 declared dead [production]
11:43 <mark> Inserted extra switch fabric and spare 4x 10G line card modules into csw1-esams [production]
2010-09-15 §
21:16 <Ryan_Lane> doing initial repo copies from mayflower to formey [production]
21:16 <Ryan_Lane> added svn users to formey.wikimedia.org [production]
18:43 <mark> Moving back bits.esams traffic [production]