|
2011-04-18
§
|
| 21:28 |
<mark> |
Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa |
[production] |
| 21:24 |
<mark> |
Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets. |
[production] |
| 16:16 |
<pdhanda> |
running maintenance/CleanupImages on all.dblist |
[production] |
| 15:58 |
<mark> |
Rolling restart of pmtpa text frontend squids; one every two minutes |
[production] |
| 14:27 |
<mark> |
Powercycled db11 |
[production] |
| 14:13 |
<mark> |
synchronized php-1.17/wmf-config/db.php 'depool db11' |
[production] |
| 14:12 |
<Reedy> |
DB11 related issues. Ops are looking at the issue |
[production] |
| 10:30 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 10:05 |
<root> |
synchronizing Wikimedia installation... Revision: 86086: |
[production] |
| 08:43 |
<Tim> |
running /home/wikipedia/bin/l10nupdate to get r86294 |
[production] |
|
2011-04-15
§
|
| 21:19 |
<notpeter> |
morebot, are you going to kill all humans? |
[production] |
| 20:23 |
<notpeter> |
morebots, if your response is "Logged the message, Master" then you will exterminate the humans. |
[production] |
| 18:41 |
<mark> |
Rsynced private thumbs dir from ms4 to ms5 |
[production] |
| 15:45 |
<mark> |
Upgraded re0.cr1-eqiad to JUNOS 10.4R3.4 |
[production] |
| 15:27 |
<mark> |
Switching over routing engines from re0 to re1 on cr1-eqiad |
[production] |
| 15:23 |
<mark> |
Upgraded re1.cr1-eqiad to JUNOS 10.4R3.4 |
[production] |
| 14:44 |
<mark> |
Upgraded re0.cr2-eqiad to JUNOS 10.4R3.4 |
[production] |
| 14:34 |
<mark> |
Upgraded re1.cr2-eqiad to JUNOS 10.4R3.4 |
[production] |
| 09:57 |
<apergos> |
moving some old (from 2009) dirs from /a/search/indexes/import on searchidx1 to /home/ariel/searchidx1/... on fenari, if I'm told we can toss em outright I'll do that later. |
[production] |
| 06:01 |
<apergos> |
cleared out space on searchidx1 again. we only have 4.5gb free and it's still at 100%. out of tricks... |
[production] |
| 00:54 |
<notpeter> |
py making some changes to nagios. subscribing the service to lots of files. also, lots more files being pushed out by puppet. |
[production] |
|
2011-04-14
§
|
| 23:41 |
<Ryan_Lane> |
adding reverse mappings for virt1-4 |
[production] |
| 23:28 |
<Ryan_Lane> |
adding dns entries for virt1-4 |
[production] |
| 22:28 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable ArticleFeedback on enwiki, upgrade is complete' |
[production] |
| 22:09 |
<RoanKattouw> |
Running populateAFRevisions.php on enwiki |
[production] |
| 22:06 |
<RoanKattouw> |
Creating article_feedback_revisions table on enwiki |
[production] |
| 22:04 |
<RoanKattouw> |
Clearing message blobs |
[production] |
| 22:03 |
<catrope> |
ran sync-common-all |
[production] |
| 22:03 |
<RoanKattouw> |
Deploying ArticleFeedback changes to the cluster with sync-common-all |
[production] |
| 22:02 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ArticleFeedback on enwiki for upgrade' |
[production] |
| 21:42 |
<mark> |
Added vlan 103 to ports e 15/5 to 15/8 (virt1-4 eth1) (tagged) on csw1-sdtpa |
[production] |
| 21:42 |
<mark> |
Put ports e 15/1 to 15/4 (virt1-4 eth0) in vlan 105 (untagged) on csw1-sdtpa |
[production] |
| 21:40 |
<mark> |
Created VLAN 105 ("virt-hosts") on csw1-sdtpa, assigned virtual interface ve 9, assigned ip 10.4.16.0/24 |
[production] |
| 21:39 |
<mark> |
Added virtual interface ve 7 ("virt") on csw1-sdtpa, assigned ip 10.4.0.1/24 |
[production] |
| 21:39 |
<mark> |
Removed virtual interface ve 7 ("virt") on csw5-pmtpa |
[production] |
| 21:24 |
<Ryan_Lane> |
cleaning up searchidx1 |
[production] |
| 21:11 |
<RoanKattouw> |
Running extensions/ArticleFeedback/populateAFRevisions.php on testwiki |
[production] |
| 21:11 |
<RoanKattouw> |
Creating article_feedback_revisions table on testwiki |
[production] |
| 21:08 |
<RoanKattouw> |
Updating ArticleFeedback code on test |
[production] |
| 16:18 |
<RobH> |
singer back online, blogs, survey, and secure should all be back in service |
[production] |
| 16:15 |
<RobH> |
blogs offline due to singer outage, as well as survey, working on resolution |
[production] |
| 16:14 |
<RobH> |
powercycling singer, unresponsive to ssh and to serial console |
[production] |
| 16:13 |
<RobH> |
secure issue is being investigated |
[production] |
| 16:13 |
<pdhanda> |
running cleanupTitles.php for all wikis in all.dblist |
[production] |
| 16:13 |
<RobH> |
investigating singer cpu spike |
[production] |
| 15:36 |
<mark> |
Implemented LDAP lookups on mail relay mchenry |
[production] |