7751-7800 of 10000 results (28ms)
2010-09-16 §
13:29 <RobH> rebooted sq38 & sq60 to bring them back online [production]
12:45 <mark> Powercycled amssq52 [production]
12:37 <mark> knsq3 declared dead [production]
11:43 <mark> Inserted extra switch fabric and spare 4x 10G line card modules into csw1-esams [production]
2010-09-15 §
21:16 <Ryan_Lane> doing initial repo copies from mayflower to formey [production]
21:16 <Ryan_Lane> added svn users to formey.wikimedia.org [production]
18:43 <mark> Moving back bits.esams traffic [production]
18:21 <mark> Moving traffic for bits.esams to bits.pmtpa [production]
18:08 <mark> Restarted varnish on knsq5 [production]
18:07 <mark> Changed LVS scheduler from 'wlc' to 'wrr' for bits on amslvs1 [production]
18:02 <mark> Restarting pybal on amslvs1, with proxyfetch disabled for varnish/bits [production]
17:48 <mark> Restarted varnish servers on knsq2, 4 and 5 [production]
17:37 <mark> temporarily disabled the proxyfetch monitor on amslvs1 to stabilize [production]
17:06 <mark> Restarted pdns instances on dobson [production]
16:59 <mark> Shutdown powerdns (auth) and pdns-recursor on dobson, preparing for reinstall [production]
16:19 <mark> Temporarily making ns1 the DNS master, to rebuild dobson [production]
15:52 <mark> Fixed regular expressions in puppet site.pp [production]
14:48 <mark> ms2 upgrade to Lucid is complete [production]
14:47 <mark> Replaced apparmor profile on ms2 by newer one from ms1, restarted replication [production]
13:25 <mark> Starting ms2 upgrade to Lucid [production]
01:09 <Ryan_Lane> finished rebuilding ersch [production]
00:58 <Ryan_Lane> rebuilding ersch [production]
00:57 <Ryan_Lane> finished rebuilding alsted [production]
00:39 <Ryan_Lane> rebuilding alsted [production]
2010-09-14 §
22:55 <awjr> Rolled back PHP 5.3.2 to PHP 5.2.10 on payments.wikimedia.org due to outstanding PHP bug 50394 [production]
20:58 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'disabled $wgFlaggedRevsComments on huwiki (unused)' [production]
20:06 <awjr> restarting payments.wikimedia.org post ubuntu 10.04 upgrade [production]
19:41 <awjr> performing upgrade from ubuntu 8.04->10.04 on payments.wikimedia.org [production]
19:06 <mark> Restarted pdns on dobson/ns0 [production]
19:05 <mark> Fixing wikimedia.org MX records in DNS [production]
18:59 <RobH> mail server issues are being investigated. [production]
18:48 <Ryan_Lane> removed public IP address to test-payments [production]
18:45 <awjr> Restarting test-payments post upgrade [production]
18:40 <Ryan_Lane> added public IP address to test-payments [production]
18:37 <awjr> Performing Ubuntu upgrade from 8.04 -> 10.04 on test-payments on tesla [production]
18:37 <RobH> db34 is running memtest86, relocating the memory moved it, who knows if its due to reseating or relocation? hence memtest is in progress. [production]
18:00 <RobH> db34 system board replaced, and also bad cpu replaced, back in rack and installing os. [production]
17:55 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25128 - Books namespace for fiwiki' [production]
17:53 <Ryan_Lane> delegated corp.wikimedia.org to ns1.corp.wikimedia.org and ns2.corp.wikimedia.org [production]
17:49 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25160 - Please create an "autopatrolled" usergroup at metawiki' [production]
17:46 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25142 - Set $wgGroupPermissions['*']['createpage'] = true; zhwiki (Allowed IP createpage)' [production]
17:43 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25166 - patroller on Fawiki' [production]
17:39 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php [production]
17:36 <JeLuF> closed swwikibooks (bug 25170) [production]
17:36 <jeluf> ran sync-common-all [production]
17:10 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25125 - Please update logo for Kurdish Wikipedia' [production]
11:43 <tstarling> synchronized php-1.5/wmf-config/InitialiseSettings.php 'PagedTiffHandler' [production]
11:42 <tstarling> synchronized php-1.5/wmf-config/CommonSettings.php 'enabling PagedTiffHandler on all wikis instead of just commons' [production]
11:41 <fixed> logmsgbot UDP listener [production]
09:58 <rainman-sr> search10-19 show down in ganglia and nagios with message "Backend Squid HTTP", although they are alive and well and not squids. Could someone look into this? [production]