|
2011-04-20
§
|
| 15:53 |
<Ryan_Lane> |
downgrading squid on knsq23 |
[production] |
| 15:33 |
<RobH> |
lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer. |
[production] |
| 15:04 |
<Ryan_Lane> |
downgraded sq71 to squid-2.7.7 |
[production] |
| 14:51 |
<mark> |
Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7 |
[production] |
| 14:34 |
<mark> |
Downgrading squid on sq66 |
[production] |
| 14:15 |
<mark> |
Stopping squid on sq72 |
[production] |
| 14:14 |
<RobH> |
reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes |
[production] |
| 14:09 |
<RobH> |
singer died, rebooting. |
[production] |
| 14:07 |
<RobH> |
secure, blogs on singer, bouncing due to singer cpu max |
[production] |
| 13:53 |
<RobH> |
singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing |
[production] |
| 11:34 |
<catrope> |
ran sync-common-all |
[production] |
| 11:33 |
<RoanKattouw> |
Running sync-common-all to deploy r86464 |
[production] |
| 11:11 |
<catrope> |
synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)' |
[production] |
| 00:41 |
<awjr> |
deployment of r86441 to payments servers complete |
[production] |
| 00:37 |
<awjr> |
deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers |
[production] |
| 00:31 |
<awjr> |
re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing) |
[production] |
| 00:14 |
<midom> |
synchronized php-1.17/wmf-config/db.php |
[production] |
|
2011-04-19
§
|
| 23:24 |
<richcole1> |
rebooting db11 to check raid |
[production] |
| 22:57 |
<awjr> |
reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm |
[production] |
| 22:53 |
<awjr> |
deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org |
[production] |
| 22:20 |
<Ryan_Lane> |
virt1-4 installed |
[production] |
| 20:54 |
<Ryan_Lane> |
powercycling singer |
[production] |
| 20:53 |
<Ryan_Lane> |
restarting apache on spence |
[production] |
| 20:52 |
<mark> |
powercycled singer |
[production] |
| 20:35 |
<Ryan_Lane> |
powercycling srv196 |
[production] |
| 17:31 |
<Ryan_Lane> |
finished ams and kn text squids |
[production] |
| 17:17 |
<RobH> |
change live, etherpad still online and functional |
[production] |
| 17:16 |
<RobH> |
pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist |
[production] |
| 17:16 |
<RobH> |
updated puppet entry for etherpad vhost |
[production] |
| 16:54 |
<Ryan_Lane> |
upgrading squid on all squid systems, slowly, over the course of the day |
[production] |
| 15:09 |
<mark> |
Killed all etherpad-user processes and started etherpad |
[production] |
| 15:09 |
<RoanKattouw> |
Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table |
[production] |
| 15:05 |
<RoanKattouw> |
Etherpad is broken, serving 500s. Have e-mailed Peter |
[production] |
| 14:06 |
<demon> |
synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395' |
[production] |
| 12:50 |
<mark> |
NICEd bayes processing cron job on mchenry |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1' |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384' |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299' |
[production] |
| 10:14 |
<RoanKattouw> |
Restarted morebots |
[production] |
| 01:05 |
<Tim> |
added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache |
[production] |
|
2011-04-18
§
|
| 21:28 |
<mark> |
Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa |
[production] |
| 21:24 |
<mark> |
Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets. |
[production] |
| 16:16 |
<pdhanda> |
running maintenance/CleanupImages on all.dblist |
[production] |
| 15:58 |
<mark> |
Rolling restart of pmtpa text frontend squids; one every two minutes |
[production] |
| 14:27 |
<mark> |
Powercycled db11 |
[production] |
| 14:13 |
<mark> |
synchronized php-1.17/wmf-config/db.php 'depool db11' |
[production] |
| 14:12 |
<Reedy> |
DB11 related issues. Ops are looking at the issue |
[production] |
| 10:30 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 10:05 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 08:43 |
<Tim> |
running /home/wikipedia/bin/l10nupdate to get r86294 |
[production] |