|
2011-04-19
§
|
| 23:24 |
<richcole1> |
rebooting db11 to check raid |
[production] |
| 22:57 |
<awjr> |
reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm |
[production] |
| 22:53 |
<awjr> |
deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org |
[production] |
| 22:20 |
<Ryan_Lane> |
virt1-4 installed |
[production] |
| 20:54 |
<Ryan_Lane> |
powercycling singer |
[production] |
| 20:53 |
<Ryan_Lane> |
restarting apache on spence |
[production] |
| 20:52 |
<mark> |
powercycled singer |
[production] |
| 20:35 |
<Ryan_Lane> |
powercycling srv196 |
[production] |
| 17:31 |
<Ryan_Lane> |
finished ams and kn text squids |
[production] |
| 17:17 |
<RobH> |
change live, etherpad still online and functional |
[production] |
| 17:16 |
<RobH> |
pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist |
[production] |
| 17:16 |
<RobH> |
updated puppet entry for etherpad vhost |
[production] |
| 16:54 |
<Ryan_Lane> |
upgrading squid on all squid systems, slowly, over the course of the day |
[production] |
| 15:09 |
<mark> |
Killed all etherpad-user processes and started etherpad |
[production] |
| 15:09 |
<RoanKattouw> |
Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table |
[production] |
| 15:05 |
<RoanKattouw> |
Etherpad is broken, serving 500s. Have e-mailed Peter |
[production] |
| 14:06 |
<demon> |
synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395' |
[production] |
| 12:50 |
<mark> |
NICEd bayes processing cron job on mchenry |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1' |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384' |
[production] |
| 10:14 |
<RoanKattouw> |
<logmsgbot>\t!log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299' |
[production] |
| 10:14 |
<RoanKattouw> |
Restarted morebots |
[production] |
| 01:05 |
<Tim> |
added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache |
[production] |
|
2011-04-18
§
|
| 21:28 |
<mark> |
Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa |
[production] |
| 21:24 |
<mark> |
Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets. |
[production] |
| 16:16 |
<pdhanda> |
running maintenance/CleanupImages on all.dblist |
[production] |
| 15:58 |
<mark> |
Rolling restart of pmtpa text frontend squids; one every two minutes |
[production] |
| 14:27 |
<mark> |
Powercycled db11 |
[production] |
| 14:13 |
<mark> |
synchronized php-1.17/wmf-config/db.php 'depool db11' |
[production] |
| 14:12 |
<Reedy> |
DB11 related issues. Ops are looking at the issue |
[production] |
| 10:30 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 10:05 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 08:43 |
<Tim> |
running /home/wikipedia/bin/l10nupdate to get r86294 |
[production] |
|
2011-04-15
§
|
| 21:19 |
<notpeter> |
morebot, are you going to kill all humans? |
[production] |
| 20:23 |
<notpeter> |
morebots, if your response is "Logged the message, Master" then you will exterminate the humans. |
[production] |
| 18:41 |
<mark> |
Rsynced private thumbs dir from ms4 to ms5 |
[production] |
| 15:45 |
<mark> |
Upgraded re0.cr1-eqiad to JUNOS 10.4R3.4 |
[production] |
| 15:27 |
<mark> |
Switching over routing engines from re0 to re1 on cr1-eqiad |
[production] |
| 15:23 |
<mark> |
Upgraded re1.cr1-eqiad to JUNOS 10.4R3.4 |
[production] |
| 14:44 |
<mark> |
Upgraded re0.cr2-eqiad to JUNOS 10.4R3.4 |
[production] |
| 14:34 |
<mark> |
Upgraded re1.cr2-eqiad to JUNOS 10.4R3.4 |
[production] |
| 09:57 |
<apergos> |
moving some old (from 2009) dirs from /a/search/indexes/import on searchidx1 to /home/ariel/searchidx1/... on fenari, if I'm told we can toss em outright I'll do that later. |
[production] |
| 06:01 |
<apergos> |
cleared out space on searchidx1 again. we only have 4.5gb free and it's still at 100%. out of tricks... |
[production] |
| 00:54 |
<notpeter> |
py making some changes to nagios. subscribing the service to lots of files. also, lots more files being pushed out by puppet. |
[production] |