3351-3400 of 10000 results (21ms)
2011-10-25 §
10:39 <mutante> mw64 - Apache HTTP flapping OK-CRIT-OK.. - high load avg.,swapping, swap Size/Used 975864 [production]
09:08 <mutante> erzurumi - installed libpam security updates [production]
05:59 <Tim> on hume: running refreshImageMetadata.php on all wikis [production]
02:18 <LocalisationUpdate> completed (1.18) at Tue Oct 25 02:20:43 UTC 2011 [production]
2011-10-24 §
22:36 <aaron> synchronized php-1.18/includes/User.php [production]
22:31 <aaron> synchronized php-1.18/includes 'deployed [[rev:100657|r100657]]' [production]
22:26 <aaron> synchronized php-1.18/includes 'deployed [[rev:100655|r100655]]' [production]
21:57 <LeslieCarr> reformatting all of the wm****.eqiad machines [production]
21:54 <Ryan_Lane> turning off ci2 [production]
19:41 <cmjohnson1> resetting storage1 to access bios and check setup [production]
19:20 <aaron> synchronized wmf-config/flaggedrevs.php 'pre-emptive wg(Add/Remove)Groups settings to account for [[rev:100636|r100636]]; should have no visible effect' [production]
19:06 <reedy> synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php '[[rev:100635|r100635]]' [production]
18:59 <mutante> sodium - initial mailman setup, created site list, will now notify ops@ via mail, storing pass in private repo ./files/misc [production]
18:36 <LeslieCarr> restarting dead nagios on spence [production]
17:40 <nikerabbit> synchronizing Wikimedia installation... : i18ndeploy [[rev:100623|r100623]] [production]
16:05 <mark> restarted pdns on nescio [production]
14:59 <notpeter> pdns_recursor hung on dobson. restarting [production]
14:30 <Jeff_Green> installed nagios/nsca on spence [production]
14:27 <mutante> snapshot4 - dist-upgrade/kernel, reboot [production]
14:19 <mark> Manually fixed package situation on formey [production]
14:10 <mutante> ekrem - gzipped access.log.1, can move/delete access.logs older than X months? [production]
14:08 <mutante> ekrem at 99% disk space ..again [production]
13:14 <mutante> something fails btw with etherpad using openoffice ([soffice] <defunct>), (was before and after reboot) [production]
13:10 <mutante> hooper - fsck was forced and OK, back up, blog and etherpad reachable again [production]
13:06 <mutante> hooper - dist-upgrade/kernel, reboot [production]
13:03 <mutante> hooper - temp. shutting down blog and etherpad for kernel update/reboot [production]
11:34 <mutante> added TERM=dumb to sudo line in /home/w/bin/scap to prevent "unknown terminal type" errors (RT 1767) [production]
10:52 <mutante> gurvin - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) [production]
10:52 <mutante> fixing date on yvon [production]
10:43 <mutante> yvon - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) [production]
04:22 <tstarling> synchronizing Wikimedia installation... : [[rev:100578|r100578]] [production]
03:48 <Tim> gave the whole Engineering group write access to the ops-requests queue [production]
03:44 <Tim> added myself to the operations group in RT so that I could resolve RT #1789 [production]
02:53 <Tim> re-enabled cron job [production]
02:26 <tstarling> synchronized php-1.18/extensions/CheckUser/maintenance/purgeOldData.php '[[rev:100574|r100574]]' [production]
02:21 <LocalisationUpdate> completed (1.18) at Mon Oct 24 02:23:42 UTC 2011 [production]
02:09 <aaron> synchronized /php-1.18/resources/jquery.ui/themes/vector/images 'deployed [[rev:100571|r100571]]' [production]
02:00 <Tim> disabled CheckUser purge cron job due to excessive replication lag [production]
01:40 <Tim> on hume: installed CheckUser purging cron job [production]
01:36 <Tim> made a shell script to run extensions/CheckUser/maintenance/purgeOldData.php on all wikis, for use in a cron job. Running it on hume to test it. [production]
2011-10-23 §
02:24 <demon> synchronized php-1.18/extensions/Contest/resources/contest.special.welcome.css '[[rev:100525|r100525]]' [production]
02:21 <LocalisationUpdate> completed (1.18) at Sun Oct 23 02:24:36 UTC 2011 [production]
2011-10-22 §
17:45 <mark> storage1 rebooted, but didn't come back (probably iptables issue) [production]
17:21 <mark> Moved storage1 to private subnet in SVN, and then reverted that change again [production]
17:10 <mark> Running dist-upgrade on storage1 [production]
17:03 <ben> synchronized wmf-config/db.php 'putting ms2 back in rotation after its reboot and RAID recovery' [production]
16:41 <RobH> removed myself from nagios contacts and restarted nagios service, feels awesome. [production]
16:34 <maplebed> copied mysql-at-facebook to ms2; it's now running that instead of the packaged version [production]
16:19 <mark> ms2 back up [production]
16:18 <mark> disk ata28 broken on ms2 [production]