2151-2200 of 10000 results (18ms)
2011-04-20 §
16:05 <Ryan_Lane> downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 [production]
16:02 <Ryan_Lane> downgrading squid on amssq31 [production]
16:01 <Ryan_Lane> downgrading squid on amssq38 [production]
16:01 <Ryan_Lane> downgrading squid on amssq39 [production]
16:01 <Ryan_Lane> downgrading squid on amssq40 [production]
16:01 <Ryan_Lane> downgrading squid on amssq41 [production]
16:00 <Ryan_Lane> downgrading squid on amssq37 [production]
15:59 <Ryan_Lane> downgrading squid on amssq36 [production]
15:59 <Ryan_Lane> downgrading squid on amssq32 [production]
15:59 <Ryan_Lane> downgrading squid on amssq33 [production]
15:58 <Ryan_Lane> downgrading squid on amssq34 [production]
15:58 <Ryan_Lane> downgrading squid on amssq35 [production]
15:57 <mark> Downgrading Squid on sq71-78 [production]
15:57 <Ryan_Lane> downgrading squid on amssq42 [production]
15:56 <Ryan_Lane> downgrading squid on amssq43 [production]
15:56 <Ryan_Lane> downgrading squid on amssq45 [production]
15:54 <Ryan_Lane> downgrading squid onamssq45 [production]
15:53 <Ryan_Lane> downgrading squid on knsq23 [production]
15:33 <RobH> lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer. [production]
15:04 <Ryan_Lane> downgraded sq71 to squid-2.7.7 [production]
14:51 <mark> Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7 [production]
14:34 <mark> Downgrading squid on sq66 [production]
14:15 <mark> Stopping squid on sq72 [production]
14:14 <RobH> reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes [production]
14:09 <RobH> singer died, rebooting. [production]
14:07 <RobH> secure, blogs on singer, bouncing due to singer cpu max [production]
13:53 <RobH> singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing [production]
11:34 <catrope> ran sync-common-all [production]
11:33 <RoanKattouw> Running sync-common-all to deploy r86464 [production]
11:11 <catrope> synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)' [production]
00:41 <awjr> deployment of r86441 to payments servers complete [production]
00:37 <awjr> deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers [production]
00:31 <awjr> re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing) [production]
00:14 <midom> synchronized php-1.17/wmf-config/db.php [production]
2011-04-19 §
23:24 <richcole1> rebooting db11 to check raid [production]
22:57 <awjr> reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm [production]
22:53 <awjr> deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org [production]
22:20 <Ryan_Lane> virt1-4 installed [production]
20:54 <Ryan_Lane> powercycling singer [production]
20:53 <Ryan_Lane> restarting apache on spence [production]
20:52 <mark> powercycled singer [production]
20:35 <Ryan_Lane> powercycling srv196 [production]
17:31 <Ryan_Lane> finished ams and kn text squids [production]
17:17 <RobH> change live, etherpad still online and functional [production]
17:16 <RobH> pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist [production]
17:16 <RobH> updated puppet entry for etherpad vhost [production]
16:54 <Ryan_Lane> upgrading squid on all squid systems, slowly, over the course of the day [production]
15:09 <mark> Killed all etherpad-user processes and started etherpad [production]
15:09 <RoanKattouw> Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table [production]
15:05 <RoanKattouw> Etherpad is broken, serving 500s. Have e-mailed Peter [production]