5851-5900 of 10000 results (22ms)
2011-04-20 §
14:34 <mark> Downgrading squid on sq66 [production]
14:15 <mark> Stopping squid on sq72 [production]
14:14 <RobH> reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes [production]
14:09 <RobH> singer died, rebooting. [production]
14:07 <RobH> secure, blogs on singer, bouncing due to singer cpu max [production]
13:53 <RobH> singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing [production]
11:34 <catrope> ran sync-common-all [production]
11:33 <RoanKattouw> Running sync-common-all to deploy r86464 [production]
11:11 <catrope> synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)' [production]
00:41 <awjr> deployment of r86441 to payments servers complete [production]
00:37 <awjr> deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers [production]
00:31 <awjr> re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing) [production]
00:14 <midom> synchronized php-1.17/wmf-config/db.php [production]
2011-04-19 §
23:24 <richcole1> rebooting db11 to check raid [production]
22:57 <awjr> reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm [production]
22:53 <awjr> deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org [production]
22:20 <Ryan_Lane> virt1-4 installed [production]
20:54 <Ryan_Lane> powercycling singer [production]
20:53 <Ryan_Lane> restarting apache on spence [production]
20:52 <mark> powercycled singer [production]
20:35 <Ryan_Lane> powercycling srv196 [production]
17:31 <Ryan_Lane> finished ams and kn text squids [production]
17:17 <RobH> change live, etherpad still online and functional [production]
17:16 <RobH> pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist [production]
17:16 <RobH> updated puppet entry for etherpad vhost [production]
16:54 <Ryan_Lane> upgrading squid on all squid systems, slowly, over the course of the day [production]
15:09 <mark> Killed all etherpad-user processes and started etherpad [production]
15:09 <RoanKattouw> Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table [production]
15:05 <RoanKattouw> Etherpad is broken, serving 500s. Have e-mailed Peter [production]
14:06 <demon> synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395' [production]
12:50 <mark> NICEd bayes processing cron job on mchenry [production]
10:14 <RoanKattouw> <logmsgbot>\t!log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1' [production]
10:14 <RoanKattouw> <logmsgbot>\t!log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384' [production]
10:14 <RoanKattouw> <logmsgbot>\t!log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299' [production]
10:14 <RoanKattouw> Restarted morebots [production]
01:05 <Tim> added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache [production]
2011-04-18 §
21:28 <mark> Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa [production]
21:24 <mark> Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets. [production]
16:16 <pdhanda> running maintenance/CleanupImages on all.dblist [production]
15:58 <mark> Rolling restart of pmtpa text frontend squids; one every two minutes [production]
14:27 <mark> Powercycled db11 [production]
14:13 <mark> synchronized php-1.17/wmf-config/db.php 'depool db11' [production]
14:12 <Reedy> DB11 related issues. Ops are looking at the issue [production]
10:30 <root> synchronizing Wikimedia installation... Revision: 86086: [production]
10:05 <root> synchronizing Wikimedia installation... Revision: 86086: [production]
08:43 <Tim> running /home/wikipedia/bin/l10nupdate to get r86294 [production]
2011-04-17 §
20:27 <mark> Setup dirty "pseudowire" between pmtpa and eqiad; a GRETAP tunnel inside a v4-in-v6 tunnel bridged to eth1/eth0.666 on 2 linux boxes (sq71 on pmtpa side). Now running OSPF on it to get full v4 interconnectivity [production]
2011-04-16 §
08:33 <apergos> disabled snaps on searchidx on in rainman's crontab, moving some en.wiki.* files in /a/search/indexes/snapshot off to fenari:/home/ariel/searchidx/.... [production]
2011-04-15 §
21:19 <notpeter> morebot, are you going to kill all humans? [production]
20:23 <notpeter> morebots, if your response is "Logged the message, Master" then you will exterminate the humans. [production]